Indicators as a Technology of Global Governance Kevin E. Davis, Benedict Kingsbury, Sally Engle Merry* July, 2011 Abstract The use of indicators is a prominent feature of contemporary global governance. Indicators are produced by organizations ranging from public actors such as the World Bank or the US State Department, to NGOs such as Freedom House, to hybrid entities such as the Global Fund, to private sector political risk rating agencies. They are used to compare and rank states for purposes as varied as deciding how to allocate foreign aid or investment and whether states have complied with their treaty obligations. This paper defines the concept of an “indicator”, analyzes distinctive features of indicators as technologies of governance, and identifies various ways in which the use of indicators has the potential to alter the topology and dynamics of global governance. Particular attention is paid to how indicators can affect processes of standard- setting, decision-making, and contestation in global governance. The World Bank Doing Business indicators and the United Nations Human Development Index are analyzed as case studies. I. Introduction The production and use of indicators in global governance is increasing rapidly. Users include public international development agencies such as the World Bank and the United Nations, national governmental aid agencies such as the US government’s Millennium Challenge * New York University. The authors acknowledge with thanks the National Science Foundation Program in Law and Social Sciences grants #SES-0921368 and #SES-023717, the latter to the research project on indicators and governance by information at the Institute for International Law and Justice at NYU Law School (iilj.org) to support a multi-country collaborative team of scholars for forthcoming work focused on indicators in developing and transitional countries. Many helpful suggestions were made by Angelina Fisher and the other members of that group, Amanda Perry-Kessaris, and participants in sessions at Annual Meetings of the American Society of International Law and the Law and Society Association and in workshops at the NYU, Emory, Toronto, and ASU Law Schools. The authors are grateful also for research assistance by Maxwell Kardon and Jessica Shimmin, and for generous financial support for related work from Carnegie Corporation of New York, the Straus Institute for the Advanced Study of Law and Justice at NYU, NYU Law School’s D’Agostino and Greenberg Faculty Research Fund, and the Rockefeller Foundation. 1 Corporation (“MCC”), global businesses and investors, bodies concerned with assessing or enforcing compliance with existing legal standards such as human rights treaty monitoring bodies, advocacy groups including many NGOs, and various scientific or expert communities, especially in the field of political science. Examples of prominent indicators and their producers or promulgators include: Doing Business Indicators produced by the International Finance Corporation (a member of the World Bank Group); Governance Indicators, including The Control of Corruption and Rule of Law, under the imprimatur of the World Bank; the Millennium Development Goals indicators under UN auspices; the Corruption Perceptions Index created by Transparency International; the Human Development Index produced by the United Nations Development Program (UNDP); the Trafficking in Persons indicators produced by the US State Department; and various indicators produced by consultancies specialized in advising investors on political risks. The Office of the United Nations High Commissioner for Human Rights has developed indicators for several core human rights. The burgeoning production and use of indicators in global governance has the potential to alter the forms, the exercise, and perhaps even the distributions of power in certain spheres of global governance. Yet the increasing use of indicators has not been accompanied by systematic study of and reflection on the implications, possibilities and pitfalls of this practice. As a result, little attention has been paid to questions such as: What social processes surround the creation and use of indicators? How do the conditions of production influence the kinds of knowledge that indicators provide? How does the use of indicators in global governance change the nature of decision-making and the use of law? How does it affect the distribution of power between and among those who govern and those who are governed? What is the nature of responses to the exercises of power through indicators, including forms of contestation and attempts to 2 regulate the production or use of indicators? The answers to these questions all have significant normative, theoretical and practical implications. Investigation of the significance of indicators as a social technology affecting power and legal relations in global governance can build usefully on several existing bodies of scholarship. In this paper we draw particularly on work in three areas. First, we use portions of the substantial body of work on connections of law and power in global governance, some but not all of which are socio-legal in methodology and in the research questions addressed (Koh 1997; Chayes and Chayes 1998; Dezalay and Garth 2002; Rajagopal 2003; Braithwaite 2004; Slaughter 2004; Santos and Rodriguez-Garavito 2005; Kingsbury et al 2005; Halliday and Oskinsky 2006; Merry 2006; Goodale and Merry 2007; Halliday 2009; Halliday and Carruthers 2009; Kingsbury et al. 2009; Simmons 2009; Symposium 2010). This scholarship seeks to theorize modes of governance operating outside or across states, including works addressing shifts from command and control approaches to more managerial and participatory processes, or to private governance and non-centralized networks. It also examines shifts connected in varying ways with neoliberalism, market hegemony, economic globalization, and new forms of information, communication and social relations. Other important areas of governance scholarship deal with “new governance” and experimentalist learning models (de Burca 2010; Symposium 2010; Sabel and Zeitlin 2010), with theories of governmentality (Miller and Rose 2008), and with networks, materiality, and other drivers of changes in relations between individuals and society or parts and the whole (Latour 2008, 2011) . A second starting point is theoretical writings on quantification and indicators as social phenomena, both general works (Hacking 1990; Porter 1995; Desrosières 1999; Espeland and Stevens 1998, 2008; Espeland and Sauder 2007; Saetnan, Lomell and Hammer 2011) and a small 3 but growing body of studies relating to specific uses of indicators and quantification in global governance contexts (Arndt and Oman 2006; Davis and Kruse 2007; Hood, Dixon and Beeston 2008; Arndt 2008; Bogdandy and Goldmann 2008; Fougner 2008; Rosga and Satterthwaite 2008; Ravallion 2010b; Satterthwaite 2011; Merry 2011; Davis, Fisher, Kingsbury and Merry 2012) Third, important insights and perspectives on indicators come from science and technology studies (STS) (Bowker and Star 1999; Latour 1987; Lampland and Star 2009; Saetnan, Lomell and Hammer 2011), including actor network theory (Latour 2005, 2011), Indicators are an increasingly important technology within many forms of global governance. This paper proposes an approach to the investigation of the role of indicators starting from the following combination of conceptual claims and hypotheses: Indicators represent a distinctive method of producing knowledge about societies. A particular feature of global governance indicators is the way they tacitly embody theories about both the appropriate standards against which to measure societies (or institutions) and the appropriate ways in which to measure compliance with those standards. Many also embody theories about social change. Those theories are generated through dynamic collective processes that differ in significant ways from other political processes. Shrouding these sorts of theoretical claims in an indicator will, depending on various circumstances discussed at greater length below, make them either more or less influential and either more or less open to various forms of contestation and regulation. Consequently, using any given indicator in global governance involves tacitly accepting both a very particular set of claims about the standards against which societies or institutions ought to be evaluated and a particular process for generating those claims. The use of indicators affects 4 not only the kinds of power and influence commanded by global decision-makers but also the most effective ways of contesting and regulating their decisions. Part II below sets out our conceptual claims regarding the defining characteristics of indicators. Part III identifies defining features of governance and global governance and sets out several hypotheses concerning the reasons for, and the implications of, the turn to indicators in global governance. Part IV presents case studies of the World Bank’s Doing Business indicators and the United Nations’ Human Development Index which provide some preliminary confirming evidence for several of the hypotheses presented in Part III concerning how the production, use, contestation and review of indicators can alter the nature of global governance. Part V concludes. II. What is an Indicator? A. Indicators defined There is no agreed meaning of ‘indicator’, but for the purposes of our inquiry into indicators as an important emerging technology in the practice of global governance the concept can be delimited in the following way. An indicator is a named collection of rank-ordered data that purports to represent the past or projected performance of different units. The data are generated through a process that simplifies raw data about a complex social phenomenon. The data, in this simplified and processed form, are capable of being used to compare particular units of analysis (such as countries or institutions or corporations), synchronically or over time, and to evaluate their performance by reference to one or more standards. 5 This working definition subsumes indexes, rankings, and composites which aggregate different indicators. Many of the best-known indicators are aggregations or ‘mash-up’ compilations (Ravallion 2010a), with substantial discretion available to the compiler in choosing what specific indicators to include, with what weightings and what devices to limit double- counting or to smooth over data unavailability. Examples include the HDI and the World Governance Indicators. While the processes and uses of aggregation raise many special issues, for the purposes of this paper the term “indicators” also includes these aggregations. We focus on the subset of indicators that are used for evaluation or judgment, and have effects specifically on decision-making or other effects in global governance. The term is also used in other ways – for example to refer to a diagnostic characteristic (such as an indicator of a person who has been trafficked, or an indicator species for an ecosystem) – but these usages are outside the concept we are examining. Indicators often take the form of, or can readily be transformed into, numerical data. A key challenge is whether and how indicators ought to be distinguished from other compilations of numerically-rendered data. The differences lie in how indicators simplify ‘raw’ data and then name the resulting product. That simplification can involve aggregation of data from multiple sources. It can also involve filtering that excludes certain data, including outliers or other data deemed to be unreliable or irrelevant. Sometimes data are filtered out and replaced with statistics, such as means or standard deviations, meant to convey similar information. In still other cases missing data are filled in with values estimated from existing data. The specific name given to data that have been organized and simplified in these ways typically denotes the social phenomenon the data ought to be taken to represent. So for example, a census report containing data on the numbers of people between the ages of 0-14, 15-64 and 65+ is not in itself 6 an indicator. But suppose that data is aggregated in particular way, for instance by dividing the sum of the first and third figures by the figure for the number of people in the 15-64 group. If that number is then labeled a “dependency ratio”, and the same calculation is made for other units or other times, the collection of processed data is capable of being used for the purposes of inter-country or inter-temporal comparisons of “dependency” and qualifies as an indicator. Indicators can also be contrasted with other representations of social phenomena. In principle, any given social phenomenon can be represented in multiple ways. For example, the level of respect for the rule of law in a given country in a given year may be represented by an indicator such as a rule of law index. Alternatively, however, it might be represented by a paragraph of text describing patterns of respect or disregard for the rule of law during the relevant period, or by a series of striking photographs or a video recording. All of these representations may purport to capture the same phenomenon. Each involves some form of simplification (although the forms vary), and each may be given a suggestive name by its producer. However, the indicator is distinctive in the ways in which it represents and conveys compiled numerical data, and it has particular attractions as a means of representation for use in comparing or evaluating particular units of analysis. Different representations are likely to convey different impressions and stimulate different responses, in ways that vary with the type of audience. Indicators cater to demand for (and receptivity to) numerical, rank-ordered and comparable data. There is considerable room for variation within the scope of our broad definition of an indicator. Some indicators have names that are highly evocative of evaluative standards; some provide more complete orderings of the units being analyzed; some involve greater 7 simplification of raw data. Analyzing the significance of these kinds of variations is an important topic for further research but is beyond the scope of this paper. B. Salient characteristics of indicators Our working definition highlights several features of indicators, including (1) the significance of the name of the indicator and the associated assertion of its power to define and represent a phenomenon such as “the rule of law”, (2) the ordinal structure enabling comparison and ranking and exerting pressure for ‘improvement’ as measured by the indicator, (3) the simplification of complex social phenomena, and (4) the potential to be used for evaluative purposes. We elaborate on the significance of these features in the following paragraphs. 1. Naming the indicator The assertion that an indicator has been brought into existence and given life is typically marked by naming it. The name itself is usually a simplification of what the index purports to measure or rank. The name’s constancy may mask changes over time in the indicator itself. Calling an indicator a measure of “transparency” or “human development” asserts a claim that there is such a phenomenon and that the numerical representation measures it. An indicator may even create the phenomenon it claims to measure, as IQ tests came to define intelligence. Labeling this measure an Indicator, Index, Ranking, League Table, etc implies a claim to knowing and measuring a phenomenon. As a result, the indicator represents an assertion of power to produce knowledge and to define or shape the way the world is understood. 2. Rank-ordered structure 8 All indicators are fundamentally comparative, and some element of ordinal ranking is a feature of the indicators we are studying. Indicators usually enable comparison of different units but in a few cases only permit comparison of the same unit at different times. However, an indicator need not rank all data points or all units in a transitive way. Influential indicators are usually cardinal (attributing separately defined values to each unit), and most use one or other of a standard menu of scaling methods (e.g. a purely ordinal scale, an equal-interval scale, or a ratio scale), but it is possible to have an indicator which does not have these attributes. Some listings with most of the attributes of indicators may merely divide units into categories described nominally, identifying difference without ranking the categories. These do not fall within our definition of an indicator. Other nominal listings may have an element of hierarchy among broad categories (red, yellow, green). These do qualify as indicators for our purposes. 3. Simplification Simplification (or reductionism) is central to the appeal (and probably the impact) of indicators. They are often numerical representations of complex phenomena intended to render these more simple and comparable with other complex phenomena which have also been represented numerically. Indicators are typically aimed at policymakers and are intended to be convenient, easy to understand, and easy to use. Yet, the transformation of particularistic knowledge into numerical representations that are readily comparable strips meaning and context from the phenomenon. In this numerical form, such knowledge carries a distinctive authority that shifts configurations and uses of power and of counter-power. This transformation reflects, but also contributes to, changes in decision-making structures and processes. 9 Indicators also often present the world in black and white, with few ambiguous intermediate shades. They take flawed and incomplete data that may have been collected for other purposes, and merge them together to produce an apparently coherent and complete picture. Wendy Espeland and Mitchell Stevens identify this as a potential consequence of what March and Simon refer to as uncertainty absorption, which “takes place when inferences are drawn from a body of evidence, and the inferences instead of the evidence itself, are then communicated” (1958:165). As Espeland and Stevens describe this process, “‘Raw’ information typically is collected and compiled by workers near the bottom of organizational hierarchies; but as it is manipulated, parsed, and moved upward, it is transformed so as to make it accessible and amenable for those near the top, who make the big decision. This ‘editing’ removes assumptions, discretion and ambiguity, a process that results in ‘uncertainty absorption’: information appears more robust than it actually is…The premises behind the numbers disappear, with the consequence that decisions seem more obvious than they might otherwise have been. An often unintended effect of this phenomenon is numbers that appear more authoritative as they move up a chain of command. The authority of the information parallels the authority of its handlers in the hierarchy” (2008: 421-2). The degree of uncertainty beneath the surface of many of the most influential simplifying indicators in global governance is quite intensively scrutinized, but usually only in specialized scientific literature (Morse 2004; Hood, Dixon and Beeston, 2008; Hoyland, Moene and Willumsen 2011).1 4. Indicators as tools for evaluation 1 Hoyland, Moene and Willumsen 2011 reintroduce the uncertainty that is filtered out of the Human Development Index indicators and the Doing Business indicators when these are aggregated into rankings. They calculate, for example, that the ranking of the top four countries as shown in the 2008 HDI has less than 1% probability of being the true top rank. The estimated confidence intervals for Georgia in the 2007 Doing Business rankings are 11th place to 59th place, rather than the definitive 18th place Georgia was given in the report. 10