This chapter focuses on measurement activities related to open data: the tools, history, stakeholders, as well as the strengths and weaknesses of the approaches to date. Moreover, the chapter addresses the roles of the many actors involved in the measurement of open data and their efforts to address not only the methodological but also the political opportunities and challenges inherent in evaluating the impact and progress of open data. We will conclude with recommendations on how best to leverage open data measurement activities in the future.
Open data has the potential to encourage citizen participation, support better public services, and uphold government accountability. One way to observe the effects of open data over time is through measurement tools, such as indices which observe phenomena over time, often using numerical indicators or qualitative assessments frequently grounded in empirical case studies. Measurement tools provide different mechanisms to track change over time, to understand progress (or the lack thereof), and to better assess the readiness, publication, use, and impact of data.
Indicators are used, in particular, in quantitative methodologies (e.g. rankings). They also define objects of study as variables that get assigned a numerical value to measure longitudinal developments of these objects against a baseline value. Found in different contexts, indicators are part of national benchmarking, scoring, and rankings. After World War II, indicators, such as the Gross Domestic Product (GDP), became tools for benchmarking countries, and progressively expanded to many other areas of society, such as university rankings.1Fast forward to 2015, the Sustainable Development Goals (SDGs) with their 230 global indicators that countries use to report against 17 overarching sustainability goals,2epitomise this process of country benchmarking, scoring, and ranking.
Individual organisations and programmes develop theories of change to understand whether invested actions and resources lead to desired outcomes.3Theories of change establish relationships between “input”, “process”, and “output” indicators, and are used for monitoring and evaluation (M&E). In M&E, ongoing and time-bound processes collect data for many purposes, including assessing how well a project is performing, and whether inputs and processes lead to desired outcomes.
Lastly, “impact case studies” have received increased attention in assessing the societal effects of higher education. These case studies provide descriptive accounts of how, for example, research output has led to societal benefits. It is noteworthy that the viability of “impact” research methods is controversially discussed not only in fields like higher education but also in open data.4
- Measurement: In this chapter, we adopt a broad definition of “measurement”, including all structured methods to gather quantitative and qualitative information about readiness, publication, use, and impact of open data.
- Measurements: In general, techniques, methods, and methodologies used for quantitative or qualitative assessment.
- Measurement tools: Indices (observing development of phenomena over time, often using numerical indicators) and/or qualitative assessments (e.g. context-based case studies), each with its own methodology. Examples include the Global Open Data Index (GODI), Open Data Barometer (ODB), Open Data Inventory, and the GovLab’s impact case studies.
- Indicators: Define objects of study as variables which often get assigned a numerical value (referred to as normalisation) to measure longitudinal development of these objects against a baseline value.
- Theories of change: Models to understand the relationships between “input”, “process”, and “output” indicators. Theories of change are used for monitoring and evaluation.
- Monitoring and evaluation: Ongoing and time-bound processes to collect data for many purposes, including how well a project is performing and whether inputs and processes lead to desired outcomes.
Brief history of measurement and open data
Since the 1980s, measurement has played an increasingly important role in public sector management through the rise of “new public management”, managerialism,5and new institutional economics. In this climate, measurements have proliferated in many forms to support neoliberal governance schemes.67Measurements are applied across various types of institutions with different methodologies, analysing very different variables ranging from single organisations to entire countries. This “audit explosion” has been driven by numerous visions, including efficiency, value for money, managing for results, accountable government actions, and market-based incentives for improvement.8
Open data measurement tools are a continuation of this development. In 2007, the Sebastopol Principles9defined open government data as we know it. The establishment of the Open Government Partnership (OGP) in 2011 and the G8 Open Data Charter10have also played an important role in establishing the foundations of the open data movement. Open data then quickly became a popular open government commitment, garnering support for official open government initiatives (e.g. the United Kingdom (UK) and the United States (US)) that have transformed or evolved into many initiatives existing today.
In 2012, Open Knowledge International (OKI), formerly Open Knowledge Foundation, launched the Open Data Census.11In 2013, both the World Wide Web Foundation (Web Foundation) and OKI published their first editions of the Open Data Barometer (ODB)12and the Global Open Data Index (GODI).13In 2014, the Web Foundation and New York University’s GovLab convened a workshop to define the Common Assessment Framework14for open data, a framework for measuring readiness, publication, use, and impact of open data.
Organisations have, over time, created a plethora of measurement tools to assess open data, including indices, such as the Open Data Inventory (ODIN), the Open Useful Reusable Government Data (OURdata) Index, the European Open Data Maturity Assessment (EODMA), the Open Data Readiness Assessment (World Bank), and impact case studies (Sunlight Foundation, GovLab, Web Foundation).
In 2015, the Open Data Charter Principles15were collaboratively drafted and later launched at the OGP Summit in Mexico City. The Open Data Charter’s Measurement and Accountability Working Group (MAWG) currently convenes representatives of the largest open data indices. In 2018, MAWG developed the Measurement Guide,16inspired by the Common Assessment Framework, in an attempt to understand how open data indices are aligned or not aligned with the Open Data Charter Principles, including highlighting limitations and existing gaps.
The section provides an overview of the stakeholders that engage with open data measurements, the basis for their interest in engaging and/or working with measurements, and how they put that interest into action.
The interest in open data measurement spans multiple stakeholder groups. One large group is government, which includes data publishers, open data champions, policy-makers, civil servants, and other agencies/task forces within government. Interest groups outside of government include non-profits working on open data, civil society groups, and academia. Governments and civil society use measurement tools to benchmark government performance on publishing, sustaining support, and making use of open government data. For example, some measurement tools, such as the GODI, anticipate a direct relationship between civil society (the auditor) and government (the auditee), which reflects normative assumptions that these are tools of “sousveillance” and data activism.
Sociologists of quantification have noted that groups of people engage with measurements, such as indices or rankings, in complex ways, often adjusting their behaviour to align with measurement tools by meeting measured targets.17Research findings highlight that measuring government performance may create unintended incentives and encourage undesired behaviour to meet targets (e.g. CompSTAT).18Others have critically examined evidence-based decision-making and have argued that the contemporary trend to “trust in numbers”19serves as a discursive device to cover up political arguments hidden in the numbers.
Contrary to the wealth of research from academia, evidence from within government on how open data measurement tools are used or impact interest groups is scarce. However, recent case study-based research conducted in the UK, Argentina, and Ukraine suggests that civil servants use measurement tools to assess whether their organisations deliver on targets.20The indicators within these tools provide open data agencies with a baseline for discussion and for developing strategies to improve open data publication. Rankings may also incentivise ministerial support or sustain momentum for open data in the absence of an official open data policy. Indicators are a discursive tool with which governments can demonstrate their openness, yet case studies suggest that open data may be confounded with open government at large or even create incentives for lowering commitments in other areas of transparency.21The implications of these findings for future research are discussed later in the chapter.
It is often noted that global indices may favour some world regions over others22by setting globally applicable targets and standards (see Strengths and weaknesses section below). As a response, some tailored regional assessments are being produced that use adjusted indicators designed to better detect the levers needed for open data readiness, publication, and impact (e.g. Africa).23Similar to the GODI and ODB’s model, these assessments rely on partnerships with a small number of regional organisations who help decide what data should be analysed.
Noteworthy open data measurement producers include international non-profits (including Web Foundation, OKI, and Open Data Watch), government bodies (including the European Commission with support from consultancy firms such as CapGemini), multilateral organisations (including the Organisation for Economic Co-operation and Development (OECD) and the World Bank), academic research organisations and divisions (such as GovLab), and national civil society organisations (CSOs), civic groups, and non-profits (such as Article 19). In general, measurement tools are supported financially by funding from philanthropic donors or government institutions via commissioned contracts or supported through voluntary efforts via in-kind contributions by the open data community. There have been repeated discussions about resourcing the production of measurement tools, but also about providing resources for those being assessed, so that they can improve their policies and practices.
Funders engage differently with measurement of open data. Some provide funding for the development of measurement tools (e.g. Omidyar Network, Hewlett Foundation, and IDRC fund the GODI and the ODB). Others commission research on the impact of open data. If programmes are tied to external funding, organisations may be required to assess the impact of their programmes and agree on internal impact metrics with the respective funder. Measurements are not only relevant to assess open government data, but also to assess the capacity of organisations working with open government data to deliver impact.
An increasing number of measurement tools with their own indicators and methodologies are now available to better assess open government data. Quantitative open data indices (using indicators to measure progress against comparable phenomena and baselines) that produce a rank and score (see Table 1 below) are possibly the most prominent type of open data measurement; however, qualitative indicators are critical to the process. The ODB, OURdata, and EODMA use qualitative data sources (e.g. descriptive news articles and research articles). In addition, the ODB collects qualitative primary data (short answers via desk research and interviews) to assess policies and impact.
There are currently five prominent open data measurement tools (see Table 1) that assess a range of elements related to open government data. The GODI and ODIN focus on measuring data publication. The ODB and EODMA measure data publication and also provide cross-country metrics for readiness and impact. In 2016, the ODIN covered 173 countries, the ODB covered 115 countries, the GODI covered 94 countries, and OURData and EODMA covered mostly only OECD and European Union (EU) countries, respectively. These measurement tools apply different criteria and measure different aspects of open data, using more than 130 different indicators in total.
Measurement tools also exist for specific topics or regions, such as the National Democratic Institute’s Legislative Openness Data Explorer,24Article 19’s open data analysis on femicides (Brazil),25and Imaflora’s environmental open data assessment (Brazil).26There are also subnational assessments, such as city rankings by Canada’s Open Cities Index27and Sunlight Foundation’s U.S. City Open Data Census.28
There are also qualitative approaches to measuring the outcomes and impact of open data (see Table 2). Qualitative studies can have different objects of study, including: 1) the types of impact generated; 2) pathways and enabling conditions of impact; and 3) the types of data use. Some studies provide narrative accounts of impact, while others develop analytical models and tools for practitioners. These analytical models are practice oriented, attempting to map out the enabling factors which support open data impact and relate these factors to one another.29Some exemplify use cases based on existing open data projects and suggest data sources for monitoring purposes.30Other studies identify typologies of impact31or attempt to model how open data use translates into behavioural changes within and across organisations by applying methods such as outcome mapping.32
To support research on data use and impact, organisations also built repositories of data use cases useful for follow-up analyses. For example, Open Data for Development’s (OD4D) Open Data Impact Map33is a repository of organisations (e.g. companies, non-profits, and academic institutions) that use open government data for advocacy, to develop products and services, to improve operations, to inform strategy, and to conduct research.
Researchers do not, however, agree upon the best methods to capture outcomes and impact. Arguably, open data has not existed as a phenomenon long enough for substantive impacts to be observable. A common assumption is that, with time, evidence on impact will accrue, yet we note several challenges to capturing that impact. First, measurement tools that use indicators for longitudinal analyses (e.g. indices) may struggle to adequately capture change simply by observing changing indicators. This is partly because social context may change, which requires indicators to be attuned to these changes34and possibly prevents observing long-term changes solely based on indicators. Second, impact case studies may struggle to attribute impact specifically to open data release. Therefore, it is no surprise that there seems to be agreement that the causal connections between investments, outputs, and outcomes will be challenging to determine conclusively.
|Expert survey and secondary data. Assessment based on quantitative and qualitative data that combines contextual data, technical assessments, and secondary third-party indicators. Results are peer reviewed and have a QA process.
|Focuses on national governments. Expanded coverage from 77 countries in 2013 to 115 countries in 2016.
|Ongoing crowdsourcing with expert review to create an annual index. Discussions from survey and review displayed publicly. Checklist with qualitative justifications. GODI methodology.
|94 countries covered in 2016–2017 (focuses on national governments). Expanding coverage from 70 countries in 2013 to 94 countries in 2016–2017.
|Open Data Inventory37odin.opendatawatch.com
|Research carried out by trained researchers. Inputs from government officials taken into consideration. Two rounds of review conducted by Open Data Watch staff.
|180 countries covered in 2017; subnational data assessed at administrative levels 1 and 2.
|OECD OURdata Index38
|Government survey completed by public sector officials from OECD countries and partners with analysis by OECD Secretariat. Includes secondary third-party indicators. A high-level overview of the report can be found in the OECD publication Government at a Glance 2017, Section 10, Open Government Data (p. 192). Note that OURdata methodologies are not publicly available online.
|32 countries covered in the 2017 OURdata Index. 31 were OECD countries (focuses on national governments) and 1 was a country partner (Colombia).
|European Open Data Maturity Assessment39
|Government survey completed by officials with validation and analysis from the European Data Portal team in cooperation with government officials. The methodology is in Annex III of the 2017 report.
|39 countries covered in 2017, including the 28 EU member states, Liechtenstein, Norway, Switzerland, and Iceland. Also, EU accession countries. Focuses on national governments.
Table 1: Prominent measurement tools that assess open government data. Source: Open Data Charter Measurement Guide, 2018
|The GovLab research series on “The Impacts of Open Data”40
Type: Qualitative case studies
Description: Each case study highlights descriptions of data use cases and consults various types of data sources (e.g. fiscal, electoral, educational data) to understand what aspects led to outcomes.
Data sources: Interviews, government documents, academic papers, media reports
|The Web Foundation’s Exploring the emerging impacts of open data in developing countries41
Type: Reports, academic papers, case studies
Description: A multi-country, multi-year study to understand how open data is being put to use in different countries and contexts across the global South, informing the development of planned and ongoing open data initiatives and their emerging impacts.
Data sources: Interviews, government documents, academic papers, media reports
|The Web Foundation’s Open Data Barometer42
Description: The ODB collects qualitative stories of impact across social, political, and economic dimensions.
Data sources: Researchers collect “credible claims made in academic and scientific publications, mainstream media, or other accredited online sources”. These reports need to demonstrate certain impacts to open data publication and use.43
|Sunlight Foundation’s Social Impact of Open Data44
Type: Case studies
Description: The study employs outcome mapping to understand how open data leads to behavioural changes in organisations. The objects of study are behaviour, relationships, activities, or actions of those people, groups, and organisations.
Data Sources: Face-to-face interviews, workshops
|European Open Data Portal’s European Open Data Measurement Assessment45
Description: The European Open Data Portal evaluates evidence on the political, social, and economic impact of open data monitored by governments, and creates comparable metrics of impact across countries based on government’s estimations of impact.
Data sources: Impact reports issued by government, data use cases, applications, news articles
Open Knowledge International’s reports on the effects of open data use:
- Data and The City46
- Changing What Counts47
- From Evidence to Action48
Type: Qualitative case studies.
Description: Each case study contains problem-centred descriptions of data use and how that data use can be enabled to alleviate a problem (problem-centred outcome). The cases cover fiscal and procurement data, crime statistics, air pollution statistics, and others.
Data sources: Interviews, government documents, academic papers, media reports
Table 2: Ways of measuring the impact of open data
Strengths and weaknesses of measurement tools
This section highlights the strengths and weaknesses of existing measurement tools, discusses the political and societal effects of rankings as the most prominent measurement tools, and reflects upon research replication, organisational learning, as well as on how inclusion can be measured and how to embed participatory processes into measurement tools.
Open data funders and governments49may commission impact research to understand their return on investment. Other groups interested in impact research include civic tech and data for development communities that intersect with open data.50Benefits of these groups working together include organisational learning and improving programme design for impact.
However, there are a number of potential challenges to using measurement tools. First, the resources required to develop, apply, and report on measurement tools, projects, and programmes are significant. Second, there is a very broad and diverse range of content that can be taken into consideration for measurement at various levels, both vertically (e.g. global, national, subnational) and horizontally (e.g. sectoral). Measurement processes can also be very time consuming for both civil society and governments. These challenges in achieving an impact on policy affect how the impact of open data is measured globally. In addition, indicators rely on the existence of secondary impact research, yet comparative accounts of impact are difficult to generate and tend to have a narrow focus on broader long-term, socioeconomic impact.
Rankings have become a prominent tool for measurement as they simplify the interpretation of complex problems through categorisation and common metrics. Rankings are easy to understand and communicate, allow for comparison of performance, track progress over time, and are effective.51The use of measurement indicators to develop dashboards is also a common practice.
More substantive research is needed to understand how governments engage and respond to open data measurements. Such endeavours could address how the internal audit procedures of government and global open data measurement tools relate to one another. Even though measurement tools frequently measure the same phenomena, there are often no consistent criteria to define and measure readiness, publication, use, and impact of open data. More specifically, we refer to the type of impact that is being measured (e.g. economic, social, environmental) and the criteria for data to be considered timely, available, accessible, useable, and of good quality. In addition, given the existence of several global measurement tools, it is imperative to address how governments respond to different measurement tools. Do governments indeed discriminate and adjust their behaviour according to the tool that ranks them the highest as recent research on the effects of multiple rankings suggests?52
To understand how civil society that is not necessarily focused on open data engages with the GODI, an ethnographic study was conducted using the case of water advocates in South-East Asia. The study highlighted the issue that the GODI’s definition of water data is too output-oriented (the GODI assesses water pollutant concentration), while water advocates may have more interest in process-related data (water management schemes), and that government data is usually not trusted or is considered to be low quality in the region.53This points to a tension that global indices apply proxy indicators to measure an entire sector based on a selection of representative indicators that do not reflect the complexity of a given sector. These findings are based on a small sample of interviewees, but they suggest that more research is needed to understand when open data measurement tools become relevant at local levels and for what types of organisations.
There are ongoing debates on whether global rankings are the best tools to assess open data in low- and middle-income countries, whether more contextual analysis is needed to detect levers to advance open data programmes, and on how existing rankings could be improved by using more relevant metrics to highlight the steps needed for low-ranking countries to improve. Some argue that metrics do not take into account all the levers necessary to improve scores in environments where enabling conditions for open data are scarce. Proponents of quantitative metrics argue that low scores mobilise commitment and action to improve scores, while critics argue that by applying a common standard, rankings tend to disadvantage or de-incentivise countries that are not in the top tier. Therefore, rankings tend to benefit governments and champions from countries in the Global North who have greater means to promote change around open data.54
Given the normative power relations that often exist in global measurements, there is a complicated history of measurement in the Global South as illustrated by the role of measurement in development planning55and in colonial and post-colonial development.56These experiences further echo concerns over the potential for creating unintended incentives, encouraging undesired behaviour to meet targets, and understanding who benefits and who loses. This also relates to literature that has the explored problematic aspects of development metrics.57
Furthermore, global rankings can apply criteria that penalise low-tier countries when new countries (often from the Global South) are added to a measurement index for assessment. Indicators operate with globally applicable baselines to incentivise progress in top-tier countries. It is challenging, if not impossible, to create indicators that are consistently meaningful across all low-, middle-, and high-income countries.
Measuring data use to understand demand and impact
Open data organisations do provide evidence of impact; however, when this evidence and information is reused by researchers and other stakeholders, open data measurement research is unevenly represented, especially with regard to concrete examples of data use and follow up on success stories, which is integral to achieving the measurement of impact. In order to enhance the measurement of progress on open data impact, organisations need to go beyond advocacy efforts and conduct participatory action research that complements global measurement assessments.
In accordance with the broader shift to user-centric open data or “publishing with purpose”, measurement tools should consider data use with the goal of measuring the demand for, and impact of, open data. Quantitative metrics have experimented with data requests and data downloads to measure demand.58As the Sunlight Foundation discusses, these metrics may only provide fragmented insight into specific audiences and user groups. Furthermore, relevance, usefulness, and usability are data- and use case-specific.59This suggests that quantitative figures like requests or download metrics should be complemented by qualitative descriptions of data users. Case studies of data use and outcomes are part of researching impact and can provide additional descriptions of demand or justify the costs of data publication.60In addition, problem-focused research that explores what makes data fit for a specific purpose is encouraged to explore what makes data understandable, accessible, timely, or comprehensive enough for a given task.61
Partnerships, replication of research, and collaborative learning
Our review in support of this chapter suggests that there is an opportunity for measurement efforts to learn from one another and to conduct follow-up analyses to test their limitations. This issue may stem from a lack of collaboration and coordination between organisations and a lack of awareness about the different use cases of measurement methods and how they can inform many different audiences and use purposes. In addition, the replication of measurements is scarce with a few exceptions (e.g. the Open Data Survey62, and measurements do not tend to build on one another. Furthermore, impact studies remain one-off projects and are generally not replicated or tested again.
Case study-based impact research depends on the availability of reliable information to construct an account of how actions led to certain outcomes. As research by the GovLab notes, information on impact produced by open data organisations is aspirational, but without “concrete evidence [of] impacts at meaningful scale”.63Such information lacks reflection on intention, implications, and impact. This points to a larger problem that open data initiatives might not systematically monitor and evaluate their work or that the incentives to provide nuanced accounts of what works and what does not are not evident. This situation seems to continue despite the rise of organisational learning and programme design for impact as recurring topics in the open data, civic tech, and data for development communities.64
However, quantitative methodologies, such as the ODB and GODI, are reused by the Natural Resource Governance Institute’s Resource Governance Index65(gauging whether countries create an enabling environment for natural resource governance) and Blavatnik School’s International Civil Service Effectiveness (InCiSE) Index (Oxford University). Others reuse significant parts of these methodologies, including the Canadian Open Cities Index and EODMA.
Partnerships can be used to clarify the expectations of measurement tool assessments. The Open Data Charter Measurement Guide identifies that it may be challenging to match existing measurements with aspirational or ambivalent principles and the commitments they represent. This may pose challenges for measuring the implementation of a number of policy commitments. A broader discussion needs to be had on what cannot be feasibly measured, which is an often overlooked topic in this space. However, there are signs of improvement with the work of the Open Data Charter’s MAWG that intends to identify overlaps and differences across measurement tools.
Sustaining global measurement tools and the role of funders in supporting and refining this work is critical. There is an opportunity for funders to collaborate more on providing financial support for measurement tools and the sustainability of these products. Beyond the need to identify partners with whom to pool resources, it is necessary that measurement programmes and tools scope out to what extent they differ, as well as how different methodologies could build on each other to improve efficiency and complementarity.
One weakness of existing measurements of open data is that inclusion is not prominently measured by all tools. However, some progress can be noted in the methodologies of the ODB and ODIN, which test whether governments publish sex-disaggregated data. In addition, only the ODB measures the impact of open data on marginalised communities. Further investigation is needed into the evidence behind these measurements and the potential methodological drawbacks of solely relying on government self-assessments or secondary data sources like news articles.
Inclusion also refers to who has a say in defining what gets counted and measured. Before asking how local communities can participate in methodology development,66more research is needed to understand when and how open data measurement tools become relevant at the local level and for what types of organisations. Similarly, understanding how CSOs, researchers, and others engage with globally defined indicators and how open data measurement tools can provide a basis for new collaborations in which these topical experts can participate more strongly in the design of methodologies. This should be complemented by participatory governance models over measurement tools which ensure meaningful engagement opportunities during methodology design, data collection, analysis, publication, and use of results. There are initiatives that work to generate localised metrics from the Global South, such as the Open African Innovation research network,67but more can be done. We recommend addressing issues on whether measurement tool creators include multidisciplinary design teams with diverse backgrounds and whether these teams consult local communities in the design process to capture diverse perspectives.
Overall, progress has been made to create tools and methodologies for the measurement of open data. Over the last decade, the landscape has expanded with a proliferation of measurement tools; however, this does not necessarily lead to better measurement. To improve and continue the evolution of open data measurement, the expansion of collaborative efforts, such as the Open Data Charter’s MAWG, is necessary. Beyond virtual working groups, it is paramount for measurement practitioners to consider the politics implicit in any measurement approach, to listen, and to include people from different geographies, prioritising diversity and gender balance in measurement conversations and practices. Measurement practitioners need to also engage with stakeholder groups that use, or could potentially use, open government data in real-world applications to guide investment or to support policy development.
A balance between quantitative and qualitative open data assessment is needed to fully understand open data impact as there currently seems to be an instinctive preference for quantitative assessments of open data, which may be due to the methodological benefits of indicators that enable ranking and comparisons (including historical comparability). Discussions need to continue on which measurement methods most contribute to authoritative knowledge about open data. Future work should focus more on the governance of measurement tools and the demand side of measurements, including what measurements are most useful for different organisations and how organisations are currently making use of measurement results to support impact tracking. Organisations and funders should be attentive to the effects of performance targets on organisational operations and consider more flexible or qualitative assessments for organisational impact.68Moreover, organisations should conduct independent audits of existing measurement tools to support future improvements.
Furthermore, concrete steps toward the reuse of measurement tools, methodologies, and results are needed but this will only be possible by addressing the current lack of transparency around existing metrics. Most methodologies, as well as the granular results of the assessments themselves (including justifications), are not public. Finally, it is vital to not consider any measurement as the final assessment. Policy-makers, practitioners, and programme managers need to be able to make use of results data as an essential component of further reviews, as well as to identify opportunities for qualitative follow-up research on the impact of open data.