global innovation index methodology

A Systems View Across Time and Space

• Open access
• Published: 18 September 2023

Assessing the robustness of composite indicators: the case of the Global Innovation Index

• Khatab Alqararah   ORCID: orcid.org/0000-0002-7837-5246 1  

Journal of Innovation and Entrepreneurship volume  12 , Article number:  61 ( 2023 ) Cite this article

1893 Accesses

1 Citations

Metrics details

This research paper introduces a methodology to assess the robustness of the Global Innovation Index (GII), by comparing the rankings provided in it with those achieved using alternative data-driven methodologies such as data envelopment analysis (DEA) and principal component analysis (PCA). With it, the paper aims to reduce the level of subjectivity in the construction of composite indicators regarding weight generation and indicator aggregation. The paper relies on PCA as a weighting-aggregation scheme to reproduce the 21 sub-pillars of the GII before the application of DEA to calculate the relative efficiency score for every country. By using the PCA-DEA model, a final ranking is produced for all countries. The random forests (RF) classification is used examine the robustness of the new rank. The comparison between the new rank and that of the GII suggests that the countries positioned at the top or the bottom of the GII rank are less sensitive toward the modification than those in the middle of the GII, the rank of which is not robust against the modification of the construction method. The PCA-DEA model introduced in this paper provides policymakers with an effective tool to monitor the performance of national innovation policies from the perspective of their relative efficiency. Ultimately, the contribution made in this paper could be instrumental to enhance the effectiveness and the efficiency of the practice of innovation management at the national level.

Introduction

For a long time, continuous efforts have been deployed to improve composite indicators (CIs) as a tool for measuring national innovation systems’ performance (Bandura, 2011 ; Dutta & Lanvin, 2013; Edquist et al., 2018 ; Corrente et al., 2021 ; Alnafrah, 2021 ). Innovation, according to the Oslo Manual (OECD/Eurostat, 2018 , p. 20) is defined as ‘new or improved product or business process (or combination thereof) that differs significantly from the firm's previous products or business processes and that has been introduced on the market or brought into use by the firm’. In this regard, national innovation systems represent the harmonious combination of several facilitators and parties such as institutional structures, infrastructures, or any supporting activities and policies that orchestrate together to facilitate and create an appropriate environment to foster innovation (Lundvall, 2007 ). The need for innovation and knowledge-driven growth is no longer exclusively for developed countries, as developing countries are also acting to design policies aiming to enhance their innovation performance to achieve better economic and environmental growth (Broughel & Thierer, 2019 ; Nikolaidis et al., 2013 ).

CIs represent the aggregation of a set of individual indicators into a single index that aims to measure a multi-dimensional concept (OECD, 2004 ). In the last decades, a growing trend has emerged among policymakers, media, and all areas of research regarding CIs and the rankings derived from them—examples of CIs include education, health, government quality, innovation—(Bandura, 2011 ; Barbero et al., 2021 ; Crespo & Crespo, 2016 ; Greco et al., 2019 ; Hatefi & Torabi, 2010 ). These rankings made a significant impact on the perceived image of how decision-making units (e.g., countries, regions, cities, universities) are relatively performing (Cherchye et al., 2008 ; Zabala-Iturriagagoitia et al., 2007a ). Not only do they influence people’s desire to visit or to invest in these countries, but also to legitimize the policies adopted in these countries to support the development of certain activities (i.e., education, economy, health, etc.). In the case of innovation systems, increasing efforts have been devoted to developing tools and indicators to assess economic and political potential, in order to provide countries with scientific evidence that helps to develop policies and activities towards the development, diffusion, and adoption of innovation (Edquist, 2011 ). CIs play a fundamental role in such endeavors. Nevertheless, several concerns emerge from this general context, namely: how much does the ranking provided by a CI represent the actual performance of a particular decision-making unit? And, how robust are the results against the methodology used in such CIs? This paper aims to contribute to clarifying these research questions, by focusing on the analysis of the statistical information provided by the 2019 edition of the Global Innovation Index (GII).

The literature has evidenced how the underlying methodology followed in the construction of CIs has a direct influence on its final results and rankings (Edquist et al., 2018 ; Greco et al., 2019 ; Grupp & Schubert, 2010 ; OECD, 2008 ; Zabala-Iturriagagoitia et al., 2007a ). However, the CIs designed to measure such issues like innovation, education, health, etc., have remained blind to this evidence. The examination of more methodologies to assess whether the results of CIs change depending on the way in which data are processed, weighted, or aggregated is referred to as the robustness of the CI (OECD, 2008 ). Out of the multiple CIs that have been developed to assess innovation potential, this paper examines the robustness of the 2019 edition of the GII. To do that, this paper applies alternative data-driven techniques such as data envelopment analysis (DEA), principal component analysis (PCA), and random forests (RF) to produce a new rank for the GII countries to examine the robustness of the GII.

The structure of this paper can be summarized as follows: Sect.  The Global Innovation Index and the need for different perspectives presents the relevance of the GII and its main characteristics, as well as the main concerns about its structure and methodology. Moreover, it highlights the need to combine different perspectives to provide a sound methodological analysis of the GII. Section  Methodological challenges in the construction of composite indicators presents the main methodological challenges that face the construction of CIs. Section  Composite indicator weightin discusses the methodology of the new model (PCA-DEA) introduced in this paper. Section  Composite indicator aggregation presents the results obtained by the PCA-DEA model, as well as the explanation of the outcomes. It also provides a comparison between the original GII rank and the rank obtained with the new model. Section  Robustness of composite indicator discusses the main conclusions and the contribution of the paper.

The Global Innovation Index and the need for different perspectives

The GII is a global comparative study that involves 129 countries using detailed evidence of 80 indicators (i.e., in its 2019 edition), accounting for around 95% of the global Gross Domestic Product. It provides a tool to measure national innovation performance using two sub-indices, innovation input, and innovation output. There are five pillars under the input sub-index, which consists of Institutions, Human Capital and Research, Infrastructure, Market sophistication, and Business sophistication. In turn, the innovation output sub-index consists of two pillars, Knowledge and Technology Output and Creative Output. Under each pillar, there are several sub-pillars, and under each sub-pillar, there are several indicators (Dutta et al., 2019 , p. 307).

The GII represents a significant source of insight in relation to national level sustainable innovation and aims to evaluate the environment supporting innovation at the national level. At the same time, it also helps to determine conditions for the diffusion of innovation and its importance for a country’s development (Dutta et al., 2019 ). The GII is distinguished from the other composite innovation indices, as it emphasizes various components related to intangible assets, such as trademarks, global brand value, and industrial designs (Dutta et al., 2022 ). Owing to this feature, the GII can be used as a leading reference for researchers, business executives, and policymakers toward creative intangible asset-related innovation.

The GII is not designed to be the ultimate tool to reinforce rankings with respect to economies and countries. Rather, it provides a foundation to be able to continually evaluate the factors of innovation which can provide much insight for economies. In its 2019 edition, the GII provided a rich database of detailed metrics to redefine innovation policies, which are imperative for policymakers. There are many layers in measuring and understanding innovation, due to its multi-dimensional nature, which can help identify core and best practices and focus on holistic policies (Decancq & Lugo, 2013 ; Edquist et al., 2018 ; Zabala-Iturriagagoitia et al., 2007a ). The intricate data allow for economies to monitor performance over time and thus standardize developments concerning other economies, allowing comparisons and the identification of best practices, which further supports the aim of the GII. Additionally, the 2019 edition of the GII highlights that it should be “regarded as a sound attempt, matured over 12 years of constant refinements, to pave the way for better and more informed innovation policies worldwide” (Dutta et al., 2019 , p. 387). With this is in mind, it is important to recognize that the GII is not the be-all and end-all in rankings within innovation, but a resource that can be used to assist countries in policymaking.

The GII has persistently maintained a similar methodology in all the editions published over the last 10 years, with very slight changes in the weights attributed to its indicators. As a result, it has presented consistency in the results, yielding similar rankings with very small variations across countries. In particular, the aggregation methods followed by the GII are the arithmetic average and the weighted average. Also, it attributes the rationality of assigning the weights only for “statistical coherence” and “highest correlation” between indicators, sub-pillars, and pillars (Dutta et al., 2019 , pp. 371–373). In the construction of every CI, assigning weights is an imperative part of the process, and the level of subjectivity should be treated with caution since it has bearings on the final results (Greco et al., 2019 ). In the case of the GII methodology, the allocation and production of weights imply several concerns within the methodology. First, the GII reveals how weights have been generated at the level of the seven pillars, but it has withheld how weights have been generated for its indicators and sub-pillars. Second, the 2019 edition of the GII specifically adjusted the weights for three indicators to provide “statistical coherence”. Third, it needs to be taken into consideration that five of the indicators included in the 2019 edition were collected by subjective means such as using qualitative data, e.g., stakeholder interviews, which might unravel multiple layers of subjectivity (Dutta et al., 2019 ). Fourth, in the process of data aggregation, the arithmetic average has been used in multiple layers, which is an unreliable statistical tool when combining large amounts of data, since the arithmetic average disregards any significance of certain variables over the others (Ćudić et al., 2022 ). Interestingly, the GII does not indicate any methodological rationale for using the arithmetic average in the process of aggregation. Furthermore, when aggregating 80 indicators by using averages, there is a high probability that two or more indicators are correlated, creating an additional bias in the results. Retrospectively, duplicating weights may distort the soundness of the outcomes. Considering these methodological concerns, the application of methodologies such as PCA is highly recommended to produce these relative weights (Adler & Yazhemsky, 2010 ), as it can help to solve this problem, by grouping indicators along with a set of consistent components, each being given a relative weight.

As previously argued, the GII has presented stability in their rankings for a long period of time, which may not aid in dynamically redefining policies, as the challenges facing innovation systems evolve over time. Additionally, it is important to recognize that some countries that follow the direction of the GII, may reap the benefits of its ranking, but this does not necessarily aid the economic growth of that country, as the GII paints all countries with the same brush, disregarding each country on its own merits (Crespo & Crespo, 2016 ; Jankowska et al., 2017 ). Therefore, the literature has already tested new methods for data processing within the GII. For instance, Cui et al. ( 2020 ) used RF and Artificial Neural Networks to recalculate the GII scores for years 2019 and 2020 by using 14 indicators. Along the same line, Pence et al. ( 2019 ) estimated the GII countries’ scores, using the Artificial Neural Networks by choosing 27 indicators of the GII 2016. Consequently, the possibility to approximate the GII countries’ score by using data-driven approaches with fewer parameters may help countries to monitor the change of their innovation performance in a shorter cycle.

The use of the unweighted average between the GII sub-indices to calculate the general index hosts several issues. Firstly, it needs to be considered that the output sub-index disregards the number of innovation inputs utilized. Secondly, it does not seem appropriate to weight the contribution of the output sub-index equally to the contribution of the input sub-index, since the number of the GII pillars on both sides is not equal. Furthermore, the use of unweighted average to aggregate the data at the level pillars/sub-indices, unnecessarily suggests the same significance for all of them within all countries. Accordingly, implying that all countries can be clustered in one big group measured in the same way may yield an unfair scale (Stavbunik & Pelucha, 2019 ). Additionally, the GII has numerous missing data values, and hence, estimating or imputing these data points will lead to higher precision in results (Cui et al., 2020 ; Omer et al., 2020 ).

The GII produces efficiency scores. However, in determining the GII data structure (input and output), the efficiency of an innovation system can be paired with productivity, i.e., the amount of output a system can generate using a certain amount of input (Edquist et al., 2018 ; Jankowska et al., 2017 ). Hence,  in order to compare the efficiency of two systems,  it is necessary to measure the output that can be generated by utilizing the same input or less (Dutta et al., 2019 ). In stipulation to this notion, there is always room to improve the GII methodology by using data-driven techniques such as DEA, which seems a very appropriate technique to measure the efficiency of a national innovation system (Alnafrah, 2021 ; Barbero et al., 2021 ; Hatefi & Torabi, 2010 ; Omrani et al., 2019 ; Zabala-Iturriagagoitia et al., 2007b ), given that the generation of the weights while using the DEA is taking place without prior intervention, and the efficiency score for each country is relative to all the other countries.

Methodological challenges in the construction of composite indicators

CIs play an essential role in formulating the shape of the innovation policies and the awareness of them (Edquist et al., 2018 ). Meanwhile, it is crucial to set the characteristics of the policies that can fulfill the needs of the system (Zabala-Iturriagagoitia et al., 2007a ). Therefore, it is alarming to accept and use them simplistically without any scrutiny and discussion (Grupp & Schubert, 2010 ). Accordingly, there are at least three fundamental challenges that need to be considered when creating CIs: (1) the relative weights associated with the indicators included in it, (2) the methodology used in the aggregation of these indicators, and (3) the robustness of its results (Freudenberg, 2003 ; Greco et al., 2019 ; Munda, 2012 ; OECD, 2008 ; Saisana et al., 2005 ). The misuse of the first two factors has a direct impact on the robustness of the final outcomes.

Composite indicator weighting

In the process of constructing a CI, selecting or developing the most sensible weighting scheme is critical, due to the strong effect of the weights on the final ranking (Greco et al., 2019 ; Yang et al., 2018 ). For example, in the case of the Technological Achievement Index introduced by Desai et al. ( 2002 ), changing the weights of some indicators reflected noticeable changes in the overall ranking with its consequent (political) implications (Cherchye et al., 2008 ). According to the OECD (2018), a “weight” represents a coefficient associated with a certain variable in the construction of CIs. In other words, weights represent the contribution of a variable to the overall CI, or to the sub-indices that constitute that overall CI. Therefore, the selection of the weights represents a major challenge in the process of constructing CIs. This challenge is frequently referred to as the ‘Index problem’ (Cox et al., 1992 ; Freudenberg, 2003 ).

To address this challenge a range of weighting schemes have been developed in the literature. For instance, (1) “No or equal weights” suggests that equal weights are assigned for all indicators, which is equivalent to saying that no weights are assigned. This is often labeled in the literature as an Attributes-Based Weighting System (Freudenberg, 2003 ). This approach among others has been used by the European Innovation Scoreboard (European Commission, 2019 ). However, this technique neglects the variation of the relative importance among indicators. Another solution is to adopt (2) Budget Allocation Processes. In this scheme, a set of ‘n’ points are given to a group of experts so they can distribute them across a group of indicators (Moldan & Billharz, 1997 ). For example, this scheme used in the estimation of the weights used in the e-Business Readiness Index introduced by Pennoni et al. ( 2005 ). However, a problem of inconsistency might occur if the number of indicators is larger than 10, or if the group of experts is not carefully selected (Saisana & Tarantola, 2002 ). Also, a critical aspect of this scheme is that it involves a high level of subjectivity. Another alternative is to (3) adopt data-driven weight-assigning techniques. In this case, and contrary to the previously mentioned weighting schemes that contain a high level of subjectivity in the arbitrariness of weights selection, statistical techniques such as PCA and DEA are claimed to be desirable as they entail a high level of objectivity in the decision-making (Decancq & Lugo, 2013 ; OECD, 2008 ).

Composite indicator aggregation

As argued above, characterizing and measuring complex phenomena through a ‘simplistic’ CI may lead to flawed results and conclusions. An alternative would be to develop several indices that would measure the same phenomenon from different perspectives. However, this alternative would increase the difficulty to interpret the results, particularly for non-specialized audiences such as the media, or policymakers. They would rather rely on a single number that includes all the indicators, which provides a simpler understanding of a complex phenomenon, even if this conclusion may be biased and hence lead to wrong (policy) implications (Saltelli, 2007 ). As a result, the use of CIs as a measurement tool is subject to debate. Moreover, the extent to which a CI represents such a phenomenon is also contested (Greco et al., 2019 ).

An obvious debate emerges in the literature between aggregators versus non-aggregators. Aggregators support building synthetic indices to describe a whole (complex) phenomenon, by combining indicators using a certain aspect to produce a bottom line, which will result in a meaningful outcome (Cobb et al., 1995 ; Gadrey & Jany, 2003 ; Osberg & Sharpe, 2002 ). In turn, non-aggregators consider that the previous aggregation results will be statistically meaningless, and the process must stop at the level of having a set of indicators without combining them to a single outcome, their objection to aggregation being the arbitrariness of the process of weighting and combining (Atkinson et al., 2002 ; Henderson, 1974 ). Nevertheless, it is worth stating that most of the widespread indices such as the Human Development Index (UNDP, 2019 ) adopt a methodological framework that uses aggregation.

Aggregation is the last step in the process of constructing a CI. According to the OECD ( 2008 ) aggregation methods can be broken down into three different categories: linear, geometric, and multi-criteria. Moreover, there is yet another categorization of aggregation to be considered, namely, ‘compensatory’ or ‘non-compensatory’ (Munda, 2005 ). Compensatory approaches occur when there is a trade-off between two perceptions of weights (Paruolo et al., 2013 ). Understandably, this trade-off might cause a fixed compensability between pairs of dimensions. This could happen when one of the dimensions might cause a loss for another (OECD, 2008 ). In this context, Munda ( 2012 ) argues that in a hypothetical sustainability index, the dimension of economic growth could compensate for the loss in the environmental dimension. To conclude, the Linear or Additive utility-based approach is the most frequently used among the approaches of compensatory aggregation (Saisana & Tarantola, 2002 ).

The non-compensatory multi-criteria approach is less frequently used, due to the simplicity of applying compensatory approaches (Greco et al., 2019 ). Nevertheless, the Condorcet–Kemeny–Young–Levenglick non-compensatory approach is regarded as a plausible alternative to the frequently practiced linear aggregation. This approach has been applied to the Environmental Sustainability Index introduced by Esty et al. ( 2005 ), and it has produced remarkable differences in the rankings reported by the original compensatory approach (Munda & Nardo, 2005 ).

Robustness of composite indicator

The robustness analysis aims to guarantee the quality and authenticity of a CI during the different levels of its construction, such as the theoretical and methodological framework development (Corrente et al., 2021 ; Saisana et al., 2005 ). Therefore, in the absence of robustness analysis, CIs may draw conclusions that would deliver a misleading message to the audience (Barbero et al., 2021 ; Billaut et al., 2010 ; Corrente et al., 2021 ; Saisana et al., 2005 ). However, the significance of robustness analysis is often neglected by most widespread CIs (OECD, 2008 ). To assure a high level of robustness in the image projected by the CI, some techniques such as uncertainty analysis or sensitivity analysis should be used (OECD, 2008 ).

Uncertainty analysis refers to the magnitude of changes that may occur in the final outputs (i.e., conclusions) of a CI as a result of changes in the construction stages, such as weights assigning or aggregation (Greco et al., 2019 ; Grupp & Schubert, 2010 ; OECD, 2008 ). In turn, sensitivity analysis deals with how much variance of the final outcomes is yielded due to these uncertainties (Grupp & Mogee, 2004 ; Saisana et al., 2005 ). Hereby, it can be concluded that there is paramount demand to assess the influence of the methodological issues amid the construction of CIs. One of the methods that can be beneficial to examine the robustness of CIs, is the classification of the RF. Such classification can be an indicator of the level of a unit (country) ranking.

Methodology

The dataset.

This research uses the dataset provided by the 2019 edition of the GII (Dutta et al., 2019 ). This is the last report before the COVID-19 pandemic, so the data reflecting the innovation performance of countries may have changed since then, due to the need to reallocate some resources to other areas of public action (e.g., health) given the health emergency. The dataset consists of 80 indicators (53 inputs and 27 outputs) collected from 129 countries. The data are available only in PDF format, so we had to transfer it into an MS-Excel sheet manually. The reliability of the data combines with the element of being presented by well-known international agencies (i.e., WIPO, INSEAD, Cornell University) and recognized by institutes and established bodies such as the UN Economic and Social Council (WIPO, 2021 ). Furthermore, GII indicators are derived from distinguished socio-economic data such as government effectiveness, tertiary education enrollment, global R&D companies’ expenditure, and ICT usage.

When analyzing the data of the 2019 GII, there are only 16 indicators without missing data for all countries. The remaining indicators have different percentages of missing data points, where the highest percentage of missing data for one indicator (High-tech net exports) is 51.2%. Among the 64 indicators with missing data, 27 indicators have less than 5% missing data points, 11 indicators have 5–10% missing data points, 12 indicators have between 10 and 20% missing data points, 7 indicators have 20–30% missing data points, and the remaining 7 indicators have more than 30% missing data points.

To deal with the missing data, several steps were taken. Firstly, the countries were divided into four groups based on the level of income: high-, upper-middle-, lower-middle-, and low-income according to the World Bank. Following this, the mean of each indicator was calculated for each group. Secondly, the countries were divided into four groups again based on the Human Development Index (UNDP, 2019 ). Following this, the mean of each indicator was calculated for each group. Thirdly, each country was grouped with the five nearest neighboring countries, then the mean of each indicator was calculated for each group. Finally, the mean of the three means for each indicator was taken as an estimation for the missing data point. However, the previous steps did not solve the problem entirely, leaving the data with 40 missing data points, which were finally imputed by linear regression modeling.

• Data envelopment analysis

DEA is a linear non-parametric programming model developed by Charnes et al. ( 1978 ) in the field of operations research, which is often referred to as the CCR model. The idea behind this model is to evaluate the relative efficiency for homogenous decision-making units (DMUs) (i.e., companies, banks, universities, countries, etc.) by giving them scores between 0 and 1. Specifically, DMUs with a score = 1 are considered as “efficient” and DMUs with a score < 1 are considered as “inefficient”. In the 1980s, Banker et al. ( 1984 ) extended the method to develop a model that deals with multiple inputs and several outputs, which the literature refers to as the BCC-model. DEA relies on a frontier created from the observed DMUs by utilizing the so-called best-practice, based on the minimum extrapolation principle (Thanassoulis, 2001 ).

Shen et al.( 2013 ) conclude that DEA provides several features to the field of CIs such as: (1) it is a means to combine multiple indicators for countries without any prior awareness about the tradeoffs (i.e., the weights); (2) the country itself obtains its own best possible indicators weights; (3) if a country is underperforming compared to other countries, this cannot be attributed to the unfair weighting scheme, since every country has been put in its most beneficial position vis a vis all the other countries, (4) and any other weighting scheme would have generated lower weighting scores for that particular country. Additionally, DEA evaluates the relative efficiency of every country, taking into consideration the performance of all other countries (Cherchye et al., 2008 ). For the above-mentioned features, DEA has been broadly utilized to examine CIs to name but a few: Technology Achievement Index (Cherchye et al., 2008 ), the Macro-economic Performance Index (Ramanathan, 2006 ), the Human Development Index (Despotis, 2005 ) and the Knowledge Economy composite indicator (Guaita Martínez et al., 2021 ).

In the context of CI construction, the literature has broadly suggested an adjustment to the classic DEA formulation by considering all the indicators to be treated as outputs (Cherchye et al., 2008 ; Guaita Martínez et al., 2021 ; Hermans et al., 2008 ; Martin et al., 2017 ). This adjustment is known as the “Benefit of doubt” approach (Cherchye et al., 2007 ), and it shifts all input variables to become outputs, compromising the inputs with a dummy variable equal to one. It was initially adopted by Melyn and Moesen ( 1991 ) as a method to construct CIs to evaluate macroeconomic performance. This approach is to be considered if the underlying structure of the evaluated composite phenomenon is not definitive or if there is disagreement regarding the construction methodology, or if the input indicators are considered to be “achievements” (Cherchye et al., 2007 ). All these concerns are valid for any CI that endeavors to measure innovation performance. For example, Crespo and Crespo ( 2016 ), by applying a fuzzy-set qualitative comparative analysis, conclude that none of the GII input pillars is a necessary condition for anticipating high innovation performance. Meanwhile, in the high-income countries, only two of the pillars (i.e., Infrastructure and Human capital and research) are sufficient to secure better innovation performance. Over and above, Jankowska et al. ( 2017 ), Edquist et al. ( 2018 ), and Barbero et al. ( 2021 ) among others, evidence that the common assumption that the higher theGII input indicators, the higher the GII output indicators, is not confirmed.

For the above-stated, this paper relies on the DEA, using the benefit of doubt approach (see Eq.  1 ), with one dummy input variable equal to 1, and 21 output variables. Particularly, these 21 output variables will be generated by considering the linear combination of the indicators under each sub-pillar of the GII (i.e., \({V}_{q}=\left({u}_{1}. {I}_{1}+{u}_{2}. {I}_{2}+\dots +{u}_{n}. {I}_{n}\right)\) , \({I}_{n}\) = indicators under sub-pillar \({V}_{q}\) , \(q= (1,\dots ,21)\) , \(n=\) number of indictors under each sub-pillar, \({u}_{n}\) = the weight generated for indicator \(n\) by using the PCA one-component loadings for the sub-pillar that the indicator belongs to. Eventually, the linear combination of indicators under the sub-pillar “Political environment” will produce variable 1, and the linear combination of indicators under the sub-pillar “Regulatory environment” will produce variable 2, etc. (see Table 1 ). This PCA-DEA approach has been introduced by Adler and Yazhemsky, ( 2010 ).

where \({I}_{qc}\) is the normalized value of the \(q\) th individual variable \((q=1,\dots ,Q)\) for the country \(c\) \((c=1,\dots M)\) and \({w}_{qc}\) the corresponding weight (Cherchye et al., 2004 ). While \({I}^{*}\) is the “benchmark performance” (i.e., the hypothetical country that maximizes the overall performance (OECD, 2008 )).

However, due to the nature of the DEA, all efficient countries will obtain the same efficiency score equal to one (i.e., DMUs that lay at the frontier). Consequently, at least for these countries, the ultimate desired ranking will not be entirely discriminating. This limitation of DEA is known in the literature as the “discrimination power problem” (Adler & Yazhemsky, 2010 ; Barbero et al., 2021 ; Hatefi & Torabi, 2010 ). To address this problem, a sequence of sub-DEAs will be executed over the efficient countries only, by dividing the output variables for these countries into subsets according to the GII pillars. For example, the first sub-DEA will be performed over the output variables: “Political environment”, “Regulatory environment”, and “Business environment”, with a dummy input equal to 1. This will be repeated seven times for the seven pillars. Finally, the total efficiency score for each country will be calculated as the average of the seven sub-DEAs scores.

• Random forests

RF is a non-parametric supervised learning statistical method, introduced by Breiman ( 2001 ). It has been proven to be a reliable method for classification problems (Hamidi & Berrado, 2018 ; Hastie et al., 2009 ). RF develops a random bootstrap of a set of data, performing multiple decision trees according to identified features (variables), eventually by so-called ‘Bagging’ to vote for the best classification (Hastie et al., 2009 ). The relationship between CIs and RF emerged recently in the fields of data mining and machine learning, to examine the robustness of the classifiers (Setiawan et al., 2019 ). In this paper, the idea behind the use of RF is to assure the robustness of the PCA-DEA results, by using the 21 variables in Table 1 to classify the countries and see to what level this classification matches the PCA-DEA results. Another quality RF can provide, is the ability to assess the ‘importance’ of every variable in the production of the classification (i.e., what are the variables that played an effective role in the classification of the countries?).

Before running the RF, all countries were divided into three groups: (1) countries in the top quartile of the PCA-DEA ranking and labeled as ‘Efficient’; (2) countries in the lowest quartile of the PCA-DEA ranking and labeled as ‘Highly inefficient’. (3) Countries in the two remaining middle quartiles labeled as ‘Inefficient”.

• Principal component analysis

The purpose of PCA is to generate the weights for the linear combination \({V}_{q}\) . Thus, a one-component PCA was performed for every GII sub-pillar separately. The loading of every indicator on that component is considered to be the weight of that indicator in the linear combination. The results show that indicators in the sub-pillar Political environment have gained an equal weight of 0.97. This case of “equal weights” among all the indicators in a given sub-pillar has occurred in several sub-pillars such as Business Environment, Research & Development, Information & Communication Technologies, Investment, and Online creativity. These sub-pillars are across the board of the GII input and output. Meanwhile, Indicators such as GERD financed by abroad, Growth rate of PPP$ GDP/worker, and Creative goods exports have gained considerably low weights of less than 0.1. Table 1 provides the value of weights associated with each indicator.

After reproducing the 21 variables (sub-pillars), we applied the benefit of doubt DEA. The efficiency scores yielded by the model PCA-DEA are presented in Table 2 . It shows that 31 countries obtain a relative efficiency score = 1 and the rest of the countries obtain relative efficiency scores < 1. Concluding that, if all the GII indicators were to be considered as outputs (i.e., the benefit of doubt), 31 out of 129 countries tend to perform efficiently with regard to national innovation performance. On the other hand, the national innovation systems in the remaining 98 countries that obtained a score < 1 are considered to be performing inefficient at different levels, the closer the score to 1, the better the country is performing. The case of equal scores for the 31 countries is acknowledged because of the discrimination power problem of the DEA (see Sect.  Composite indicator weighting ).

However, to reach the ultimate final ranking for all the countries, the case of 31 equal efficiency scores must be resolved. To do that, only for the top 31 countries, we examine the efficiency of each country under each GII pillar separately. In practice, this implies running the previous benefit of doubt DEA model for every country seven times (one for every pillar). In each pillar, we consider the output variables to be the variables from Table 1 under that specific pillar and the input being equal to 1. This step evaluates the efficiency of every country with regard to every GII pillar one by one. Eventually, the average of these seven sub-DEAs for every country will be the discriminating value to form the final ranking among these 31 countries. As far as the GII pillars are individually concerned, the results show that the performance of the national innovation system in Switzerland is the best, followed by the USA and the Republic of Korea (see Table 3 ). Lastly, by replacing the score values = 1 in Table 2 with the average value of the seven sub-DEAs for these 31 efficient countries, a final ranking for all countries can be generated (see Table 4 ).

In pursuance of the comparison, Table 4 shows the PCA-DEA ranking of all countries compared to the GII 2019, where the negative sign of difference indicates moving backward and the positive sign indicates moving forward in the new ranking. The comparison adopts two aspects: (1) the absolute value of the difference between the country’s position in the GII rank and the PCA-DEA rank; (2) the distribution of the absolute value of difference over the GII rank.

The comparison shows a noticeable difference in the ranking between the GII 2019 and the PCA-DEA score. Specifically, the average absolute value of the difference between the two ranks is 10.9 positions; whereas some countries change their positioning by more than 40 positions of ranking, such as India and Macedonia. Other countries such as Switzerland maintain the same position. Moreover, the results show that countries that lay in the middle of the GII ranking display the greatest difference in the absolute value in their positions (i.e., the greatest change in the ranking both ways, forward and backward), while countries that lay at the top or bottom of the GII made less change in rankings (see Fig.  1 ).

Distribution of absolute value of the difference over GII 2019

To assure the robustness of the PCA-DEA results used in this paper, the RF technique was applied. To elaborate, all countries were divided into three groups. The first group consists of the 31 efficient countries in the PCA-DEA rank, this group being labeled as “Efficient”. The second group consists of the lowest 30 countries in the PCA-DEA rank, this group is labeled as “Highly inefficient”. Lastly, the third group consists of the remaining 68 countries in the middle of the PCA-DEA rank, which is labeled as “Inefficient”. The application of the RF shows that the PCA-DEA ranking matches its classification to 88%. In detail, for 26 out of 31 efficient countries in the ranking of PCA-DEA, the RF also has classified as efficient, while 5 countries shifted to be inefficient. As regards the highly inefficient countries, 26 out of the lowest 30 countries in the ranking of PCA-DEA, were classified as highly inefficient by the RF, while 4 countries were shifted to be inefficient. Finally, RF classified 62 out of the 68 countries in the middle of the PCA-DEA ranking as inefficient, while 3 countries shifted to efficient, and 3 countries shifted to highly inefficient (see Table 5 ).

Additionally, RF highlights the importance of each variable during the classification. For example, the input sub-pillars “Political environment”, “Education”, and “Research & development”’ play a significant role in determining the rank of the country, while the sub-pillars “Investment”, “Trade, competition, and market scale” play a minimal role. Likewise, the output side sub-pillars such as “Knowledge impact” and “Creative goods and services” are more important than “Knowledge diffusion” and “Intangible assets” (see Table 6 ).

Conclusions

Innovation is one of the economic factors leading to societal progress, technological development, and economic growth. As a result, the measurement of innovation performance and capacity has received increasing attention, not only at the academic level, but also at the policy and societal levels. Composite Indicators (CIs) such as the Global Innovation Index (GII) and the European Innovation Scoreboard are well recognized and accepted instruments for this task. Although they represent a vehicle of communication and an elevator for the awareness of innovation, they also play a significant role in shaping and directing innovation policies (Edquist et al., 2018 ).

The literature dealing with the construction of the CIs has had an important evolution in recent years, with new contributions emphasizing that the methodologies underlying these CIs should be considered carefully, in terms of weighting, aggregation, and robustness. In this paper, we introduce a data envelopment analysis (DEA) model that relies on the weights provided by the application of principal component analysis (PCA). This PCA-DEA model is applied to the 2019 edition of the GII, which measures innovation at the national level in 129 countries worldwide. The paper aims to examine the sensitivity of the results provided by the GII, to variations in the methodology. In addition, the robustness of the results provided by this alternative methodology is tested using the random forest (RF) methodology. The rationale for using data-driven techniques such as PCA, DEA, and RF, lies in that they eliminate the subjective intervention during the construction of the CI.

The PCA-DEA model used in the paper provides the relative efficiency of innovation of the countries considered in the GII, which has enabled us to draw a final ranking. This PCA-DEA model relies on the “Benefit of doubt” technique, which measures the effectiveness of the national innovation systems, since all the countries are attributed a single input with a value equal to one (i.e., no minimization function for the input side).

The comparison between the GII rank and PCA-DEA rank shows that the countries that lay in the middle of the GII rank have made a considerable change in their positions in terms of the absolute value of difference. Meanwhile, countries at the top or bottom of the GII rank have made a minimal change, despite the rank of some of the top countries such as Singapore dropped 10 positions on the PCA-DEA model. This is explained by the attributes-based weighting system (i.e., no weights or equal weights). The GII uses the unweighted average of the GII sub-indices to calculate the general index. Instead, this paper relies on the objective weights provided by the PCA to each of the GII sub-indices.

One of the conclusions of the paper is that the GII rank appears to be robust for the countries located at the top and the bottom of the ranking, but not for the countries in the middle. In other words, the positioning of the countries in the middle of the GII rank is more sensitive towards the modification of the construction method of the GII, compared to the countries at the top or the bottom. This can be attributed to the influence of the subjective means of generating and assigning the weights for the indicators such as the Budget Allocation scheme, since there is no consensus about the importance of an indicator over the others. In addition, what seems to be important for a certain group of countries is not necessarily important for other countries (Stavbunik & Pelucha, 2019 ; Zabala-Iturriagagoitia et al., 2007a ).

As regards the new ranking provided by the alternative PCA-DEA model, the results of the RF methodology applied in the paper reveal that the new ranking is statistically robust. As a matter of fact, the RF attributes 88% of the countries to the right sub-groups. In addition, the RF classification also helped to assess the relative importance of each variable (sub-pillar) in explaining the factors that determine whether each country is efficient, inefficient, or highly inefficient.

This paper represents, to the best of our knowledge, the first contribution in which the robustness of the GII is assessed, using alternative methodologies such as the PCA and the DEA. Consistent with the extant literature, particularly related to the European Innovation Scoreboard (Barbero et al., 2021 ; Edquist et al., 2018 ; Zabala-Iturriagagoitia et al., 2007a , 2007b ), our results also reveal that the methodological election has a direct impact on the results provided by composite indices. Accordingly, further editions of the GII should seek to introduce alternative methodologies to the existing data, in order to increase the robustness and reliability of the rankings provided by the GII. This would allow for emphasizing different dimensions of the innovation system, highlighting those areas in which each country may show relative strengths, and weaknesses. We contend that these alternative methodologies would provide additional information to policymakers, so that effective policies can be adopted on each innovation system.

Finally, one of the limitations of this research paper is that we used only the year of 2019 as a statistical point. Further research could thus explore the extent to which potential deviations may also emerge when larger periods of time are considered. This is an emerging field of research where limited evidence exists to date (e.g., Aparicio et al., 2020 ; Zabala-Iturriagagoitia et al., 2021 ), but which shows enormous potential to provide a dynamic assessment of innovation performance worldwide.

Availability of data and materials

The dataset used and analyzed during the current study is available in the 2019 edition of the Global Innovation Index report, [ https://www.wipo.int/edocs/pubdocs/en/wipo_pub_gii_2019.pdf ].

Abbreviations

Composite indicator

Decision-making unit

• Global Innovation Index

Adler, N., & Yazhemsky, E. (2010). Improving discrimination in data envelopment analysis: PCA–DEA. European Journal of Operational Research, 202 , 273–284.

Google Scholar  

Alnafrah, I. (2021). Efficiency evaluation of BRICS’s national innovation systems based on bias-corrected network data envelopment analysis. Journal of Innovation and Entrepreneurship, 10 (1), 1–28.

Aparicio, J., Kapelko, M., & Monge, J. F. (2020). A well-defined composite indicator: An application to corporate social responsibility. Journal of Optimization Theory and Applications, 186 , 299–323.

Atkinson, T., Cantillon, B., Marlier, E., & Nolan, B. (2002). Social indicators: The EU and social inclusion . Oxford University Press. https://doi.org/10.1093/0199253498.001.0001

Book   Google Scholar  

Bandura, R. (2011). Composite indicators and rankings: Inventory 2011. Technical report. New York: United Nations Development Programme (UNDP).

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Science, 30 (9), 1031–1142.

Barbero, J., Zabala-Iturriagagoitia, J. M., & Zofío, J. L. (2021). Is more always better? On the relevance of decreasing returns to scale on innovation. Technovation, 107 , 102314.

Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? Scientometrics, 84 (1), 237–263.

Breiman, L. (2001). Random forests. Machine Learning, 45 , 5–32.

Broughel, J., & Thierer, A. (2019). Technological innovation and economic growth: A brief report on the evidence . Mercatus research.

Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2 (6), 429–444.

Cherchye, L., Moesen, W., & Van Puyenbroeck, T. (2004). Legitimately diverse, yet comparable: On synthesizing social inclusion performance in the EU. Journal of Common Market Studies, 42 (5), 919–955.

Cherchye, L., Moesen, W., Rogge, N., & Puyenbroeck, T. V. (2007). An introduction to ‘Benefit of the Doubt’ composite indicators. Social Indicators Research, 82 (1), 111–145.

Cherchye, L., Moesen, W., Rogge, N., Puyenbroeck, T., Saisana, M., Saltelli, A., et al. (2008). Creating composite indicators with DEA and robustness analysis: The case of the Technology Achievement Index. Journal of the Operational Research Society, 59 , 239–251.

Cobb, C. W., Halstead, T., & Rowe, J. (1995). The genuine progress indicator: Summary of data and methodology . Redefining Progress.

Corrente, S., Bernabeu, A. G., Greco, S., & Makkonen, T. (2021). Robust measurement of innovation performances in Europe with a hierarchy of interacting composite indicators. Economics of Innovation and New Technology . https://doi.org/10.1080/10438599.2021.1910815

Article   Google Scholar  

Cox, D. R., Fletcher, A. E., Gore, S. M., Spiegelhalter, D. J., & Jones, D. R. (1992). Quality of life assessment: Can we keep it simple? Journal of the Royal Statistical Society Series A (statistics in Society), 155 (3), 353–393.

Crespo, N., & Crespo, C. (2016). Global innovation index: Moving beyond the absolute value of ranking with a fuzzy-set analysis. Journal of Business Research, 69 (11), 5265–5271.

Ćudić, B., Alešnik, P., & Hazemali, D. (2022). Factors impacting university–industry collaboration in European countries. Journal of Innovation and Entrepreneurship, 11 (1), 33.

Cui, R., Sun, J., Li, Y., Yang, K., & Wu, X. (2020). Data-driven approach with artificial neural network for Global Innovation Index re-evaluation. 2020 6th International Conference on Big Data and Information Analytics (BigDIA) , (pp. 88–93).

Decancq, K., & Lugo, M. A. (2013). Weights in multidimensional indices of wellbeing: An overview. Econometric Reviews, 32 (1), 7–34.

Desai, M., Fukuda-Parr, S., Johansson, C., & Sagasti, F. (2002). Measuring technology achievement of nations and the capacity to participate in the network age. Journal of Human Development, 1 (3), 301–311.

Despotis, D. (2005). Measuring human development via data envelopment analysis: The case of Asia and the Pacific. Omega, 33 (5), 385–390.

Dutta, S., Lanvin, B., León, L. R., & Wunsch-Vincent, S. (2022). Global Innovation Index: What is the future of innovation-driven growth? World Intellectual Property Organization.

Dutta, S., Lanvin, B., & Vincent, S. W. (2019). The Global Innovation Index 2019. Creating healthy lives—The future of medical innovation. Ithaca, Fontainebleau, and Geneva: Cornell University, INSEAD, WIPO. From https://www.wipo.int/edocs/pubdocs/en/wipo_pub_gii_2019.pdf

Edquist, C. (2011). Systems of innovation perspectives and challenges. African Journal of Science, Technology, Innovation and Development, 2 (3), 14–43.

Edquist, C., Zabala-Iturriagagoitia, J. M., Barbero, J., & Zofío, J. L. (2018). On the meaning of innovation performance: Is the synthetic indicator of the Innovation Union Scoreboard flawed? Research Evaluation, 27 (3), 196–211.

Esty, D. C., Levy, M., Srebotnjak, T., & De Sherbinin, A. (2005). Environmental sustainability index: Benchmarking national environmental stewardship. New Haven: Yale Center for Environmental Law & Policy.

European Commission. (2019). European Commission. Retrieved March 30, 2020 from https://ec.europa.eu/commission/presscorner/detail/en/ip_19_2991

Freudenberg, M. (2003). Composite indicators of country performance: A critical assessment. OECD Science, Technology and Industry Working Papers .

Gadrey, J., & Jany, F.-C. (2003). Les indicateurs de richesee et de developpement: Un bilan international en vue d’une initiative francaise . DARES, Ministere du Travail.

Greco, S., Ishizaka, A., Tasiou, M., & Torrisi, G. (2019). On the methodological framework of composite: A review of the issues of weighting, aggregation, and robustness. Social Indicators Research, 141 , 61–94.

Grupp, H., & Mogee, M. E. (2004). Indicators for national science and technology policy: How robust are composite indicators? Research Policy, 33 (9), 1373–1384.

Grupp, H., & Schubert, T. (2010). Review and new evidence on composite innovation indicators for evaluating national performance. Research Policy, 39 (1), 67–78.

Guaita Martínez, J., Martín, M., Ostos, M. S., & Pardo, M. D. (2021). Constructing knowledge economy composite indicators using an MCA-DEA approach. Economic Research-Ekonomska Istraživanja, 34 (1), 331–351.

Hamidi, S., & Berrado, A. (2018). Segmentation of innovation determinants: Case of the Global Innovation Index. 12th International Conference on Intelligent Systems: Theories and Applications (pp. 1–8). Rabat: Association for Computing Machinery. From https://doi.org/10.1145/3289402.3289548

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2 ed., Vol. 12). New York: Springer-Verlag. From https://hastie.su.domains/ElemStatLearn/printings/ESLII_print12_toc.pdf

Hatefi, M. S., & Torabi, S. A. (2010). A common weight MCDA–DEA approach to construct composite indicators. Ecological Economics, 70 (1), 114–120.

Henderson, D. W. (1974). Social indicators: A rationale and research framework. Ottawa: Cambridge University Press. From https://www.cambridge.org/core/journals/canadian-journal-of-political-science-revue-canadienne-de-science-politique/article/abs/dw-henderson-social-indicators-a-rationale-and-research-framework-ottawa-information-canada-1974-pp-vii-90/8DE50AF54F6D3452E0C7

Hermans, E., Bossche, F. V., & Wets, G. (2008). Combining road safety information in a performance index. Accident Analysis and Prevention, 40 , 1337–1344.

Jankowska, B., Jedrych, A., & Dabrowska, K. (2017). Efficiency of national innovation systems—Poland and Bulgaria in the context of the Global Innovation Index. Comparative Economic Research, 20 (3), 77–94.

Lundvall, B. A. (2007). National innovation systems—Analytical concept and development tool. Industry and Innovation, 14 (1), 95–119.

Martin, J., Mendoza, C., & Román, C. (2017). A DEA travel-tourism competitiveness index. Social Indicators Research, 130 , 937–957.

Melyn, W., & Moesen, W. (1991). Towards a synthetic indicator of macroeconomic performance . Center for Economic Studies.

Moldan, B., & Billharz, S. (1997). Sustainability indicators: A report on the project on indicators of sustainable development . Chichester John Wiley & Sons.

Munda, G., & Nardo, M. (2005). Non-compensatory composite indicators for ranking countries: A defensible setting—EUR 21833 EN. Ispra: Institute for the Protection and Security of the Citizen.

Munda, G. (2005). Multiple criteria decision analysis and sustainable development. In S. Greco, M. Ehrgott, & J. Figueira (Eds.), Multiple criteria decision analysis International series in operations research & management science (pp. 1235–1267). New York: Springer. https://doi.org/10.1007/978-1-4939-3094-4_27

Chapter   Google Scholar  

Munda, G. (2012). Choosing aggregation rules for composite indicators. Social Indicators Research, 109 (3), 337–354.

Nikolaidis, Y., Fouskas, K., & Carayannis, E. G. (2013). Assisting regional policy by rapidly comparing enterprise innovation between regions. Journal of Innovation and Entrepreneurship, 2 (1), 1–25.

OECD. (2004). The OECD-JRC handbook on practices for developing composite indicators . The OECD committee on statistics.

OECD. (2008). Handbook on constructing composite indicators: methodology and user guide . OECD.

OECD, Eurostat. (2018). Oslo manual: guidelines for collecting, reporting and using data on innovation (4th ed.). OECD Publishing.

Omer, M., El-Amin, M., Nasr, A., & Ahmed, R. (2020). Modeling, visualization, and analysis of African innovation performance. International Conference on Learning Representation (ICLR). Addis Ababa. Retrieved 3 11, 2022 from https://www.researchgate.net/publication/343735079_Modeling_Visualization_and_Analysis_of_African_Innovation_Performance/fulltext/5f3c9c16a6fdcccc43d31df6/Modeling-Visualization-and-Analysis-of-African-Innovation-Performance.pdf?_sg%5B0%5D=CEDIy0xN-y8EZs6

Omrani, H., Valipour, M., & Mamakani, S. J. (2019). Construct a composite indicator based on integrating common weight data envelopment analysis and principal component analysis models: An application for finding development degree of provinces in Iran. Socio-Economic Planning Sciences, 68 , 100618.

Osberg, L., & Sharpe, A. (2002). An index of economic well–being for selected OECD countries. The Review of Income and Wealth, 48 (3), 291–316.

Paruolo, P., Saisana, M., & Saltelli, A. (2013). Ratings and rankings: Voodoo or science? Journal of the Royal Statistical Society: Series A (statistics in Society), 176 (3), 609–634.

Pence, İ, Kalkan, A., & Cesmeli, M. (2019). Estimation of the country ranking scores on the Global Innovation Index 2016 using the artificial neural network method. International Journal of Innovation and Technology Management, 16 (4), 1940007.

Pennoni, F., Tarantola, S., & Latvala, A. (2005). The European e-business readiness index. Joint Research Centre (JRC). Retrieved March 20, 2020 from https://www.researchgate.net/publication/46446317_The_2005_European_e-Business_Readiness_Index

Ramanathan, R. (2006). Evaluating the comparative performance of countries of the Middle East and North Africa: A DEA application. Socio-Economic Planning Sciences, 40 (2), 156–167.

Saisana , M., & Tarantola, S. (2002). State of the art report on current methodologies and practices for composite indicator development. Ispra, Italy: European Commission, Joint Research Centre, Institute for the Protection and the Security of the Citizen, Technological and Economic Risk Management Unit.

Saisana, M., Saltelli, A., Schulze, N., Tarantola, S., & Duchene, V. (2005). Uncertainty and sensitivity analysis for the Knowledge-based Economy Index. Conference on Medium-Term Economic Assessment (CMTEA). Sofia. Retrieved 3 11, 2022 from https://www.researchgate.net/profile/Michaela-Saisana/publication/242180137_Uncertainty_and_Sensitivity_Analysis_for_the_Knowledge-Based_Economy_Index/links/579723b708aed51475e66de4/Uncertainty-and-Sensitivity-Analysis-for-the-Knowledge-Based-Economy-Inde

Saltelli, A. (2007). Composite indicators between analysis and advocacy. Social Indicators Research, 81 (1), 65–77.

Sebastian, L., & Gutierrez, E. (2008). Data envelopment analysis of the human development index. International Journal of Society Systems Science, 1 (2), 32–150.

Setiawan, B., Djanali, S., Ahmad, T., & Aziz, N. M. (2019). Assessing centroid-based classification models for intrusion detection system using composite indicators. Procedia Computer Science, 161 , 665–676.

Shen, Y., Hermans, E., Brijs, T., & Wets, G. (2013). Data envelopment analysis for composite indicators. Social Indicators Research, 114 , 739–756.

Stavbunik, Y., & Pelucha, M. (2019). The innovation system of Kazakhstan in the context of the Global Innovation Index. The Innovation Journal: the Public Sector Innovation Journal, 24 (1), 1–24.

Thanassoulis, E. (2001). Introduction to the theory and application of data envelopment analysis (1 ed.). Dordrecht: Kluwer Academic Publishers. https://doi.org/10.1007/978-1-4615-1407-7

UNDP. (2019). Human development index. New York: United Nations Development Programme. Retrieved 10 19, 2020 from https://hdr.undp.org/en/content/download-data

WIPO. (2021). www.wipo.int . Retrieved 2 10, 2022 from https://www.wipo.int/pressroom/en/articles/2021/article_0008.html#:~:text=About%20the%20Global%20Innovation%20Index&text=The%20GII%20has%20also%20been,Sustainable%20Development%20Goals%20(SDGs)

Yang, F.-C., Kao, R.-H., Chen, Y.-T., Ho, Y.-F., Cho, C.-C., & Huang, S.-W. (2018). A common weight approach to construct composite indicators: The evaluation of fourteen emerging markets. Social Indicators Research, 137 (2), 463–479.

Zabala-Iturriagagoitia, J., Aparicio, J., Ortiz, L., & Carayannis, E. G. (2021). The productivity of national innovation systems in Europe: Catching up or falling behind? Technovation, 102 , 102215.

Zabala-Iturriagagoitia, J. M., Jiménez-Sáez, F., Castro-Martínez, E., & Gutiérrez-Gracia, A. (2007a). What indicators do (or do not) tell us about regional innovation systems. Scientometrics, 41 (5), 661–672.

Zabala-Iturriagagoitia, J. M., Voigt, P., Gutierrez-Gracia, A., & Jimenez-Saez, F. (2007b). Regional innovation systems: How to assess performance. Regional Studies, 41 (5), 661–672.

Download references

Acknowledgements

The author is grateful to the editor and the anonymous reviewers for their feedback and the comments provided on earlier versions of the paper.

No source of funding.

Author information

Authors and affiliations.

Deusto Business School, University of Deusto, San Sebastian, Spain

Khatab Alqararah

You can also search for this author in PubMed   Google Scholar

Contributions

Single author.

Corresponding author

Correspondence to Khatab Alqararah .

Ethics declarations

Competing interests.

The author declares that he has no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Alqararah, K. Assessing the robustness of composite indicators: the case of the Global Innovation Index. J Innov Entrep 12 , 61 (2023). https://doi.org/10.1186/s13731-023-00332-w

Download citation

Received : 28 March 2022

Accepted : 17 August 2023

Published : 18 September 2023

DOI : https://doi.org/10.1186/s13731-023-00332-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Composite indicators
• Innovation performance

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy

SEARCH HISTORY

• Country Profile

Methodology

This GDI 2024 measures the digitalization of 77 countries that represent a total of 94% of the world's GDP and cover 82% of the world's population, which should generally reflect global progress in digital transformation. The 77 countries are grouped into three clusters — Frontrunners, Adopters, and Starters — according to their level of ICT maturity and economic development. The GDI makes specific recommendations for the countries of each clusters on the acceleration of their digitization process.

Digitalization does not progress in isolation. Adequate ICT supply and strong demand are required to tap into its full potential. The GDI considered both economic value and industry development, helps countries improve the ROI of ICT investments, identify the strengths and weaknesses of their own ICT industry, and inform the development of national digitalization strategies.

• Fiber Coverage (To C)
• Fiber Coverage (To B)
• 4G & 5G Coverage
• Mobile Broadband Experience
• Fixed Broadband Experience
• Enterprise Gigabit Campus penetration
• IPV6 Deployment
• Datacenter investments
• Advanced Storage Investment
• Business Continuity and Disaster Recovery (BCDR) Adoption
• Computing Power 
• Cloud Investment
• Renewable Electricity Investment
• Green Travel Ratio
• Charging Convenience
• ICT Investment
• Spectrum & Policy
• Digital Transformation Policies
• Green Energy Policy
• ICT Patents
• ICT Laws & Regulations
• Fixed Broadband Subscriptions
• Mobile Broadband Subscriptions
• loT Installed Base
• Mobile Broadband Affordability
• Fixed Broadband Affordability
• Mobile data per Connection
• Enterprise Export Bandwidth 10 Gigabit+ Deployment Rate
• Cloud Migration & Cloudification Rate
• Data Creation
• E-Government Service
• Industry digital transformation spending
• Economics of Renewable Electricity
• Internet Participations
• e-Commerce Transactions
• Smartphone Penetration
• Online Video Streaming
• Number of Startups
• STEM Graduates Ratio
• ICT Workforce 

These target values are extrapolated from market penetration projections based on the highest ranked countries, historical market performance, and expert opinions. Each country's score is then determined by its normalized raw data value in relation to this scale. In most baseline cases, a value that is less than 10% of the target value will be allocated a score of 1. A value of between 10% and 20% of the target value is allocated a score of 2, and so on.

The weighting of each pillar: Ubiquitous Connectivity, Digital Foundation, Green Energy, and Policy & Ecosystem account for 30%, 30%, 30%, and 10% of GDI, respectively. The final index score is then calculated by the scores of the four pillars.

GDI SCORE = \( \sum_{i=1}^{4}\left(\frac{\alpha_{i}}{n_{i}} \times \sum_{j=1}^{n_{i}} \text { Score }_{j}\right)\)   i：represents the four pillars ni：represents the number of indicators for each pillar   j：represents the indicators of each pillar αi：represents the weight of each pillar

Remarks: The GDP used for GDI calculated is based on purchasing power parity (PPP), which account differences in the cost of living across countries. The latest data from sources such as the Organization for Economic Cooperation and Development (OECD), the International Telecommunication Union (ITU), GSMA, the World Economic Forum (WEF), the World Bank, the United Nations, Ookla, IDC, and Huawei. Missing data were estimated based on conditions within similar geographic regions. The data in the chart may be different from the actual calculation result due to rounding adjustment.

Online Services

Consumer Products

Huawei Cloud

Sales: +852-800-931-122

Carrier Network

• Innovation Indicator 2024

By Stefan Schaible and David Born

Small countries, high innovative capacity: Switzerland, Singapore and Denmark lead the way

Innovation is essential for advanced economies to remain competitive, foster growth, and secure prosperity. It is also key to addressing global challenges like decarbonization, climate change, and pandemics. Against this backdrop, the Federation of German Industries (BDI) and Roland Berger present the Innovation Indicator 2024. This indicator compares the innovation capability of 35 economies using a structured measurement model. The analysis focuses on three key aspects: first, assessing the current state of innovation ("Creating Innovations"), second, exploring the technological future of the countries ("Developing Future Fields through Key Technologies"), and third, evaluating the sustainability of economic systems to determine if they operate within planetary boundaries and can be successful in the long term ("Sustainable Economy").

"Innovations are the driving force behind competitiveness, prosperity, and growth. A country that aims to succeed in the future must create the right conditions for research and innovation policies today, and initiate the necessary momentum."

The top positions in the overall ranking of the 2024 Innovation Indicator are once again occupied by smaller nations, as in previous years. Switzerland remains in first place with 71 points, followed by Singapore, which increased its score by three points to reach 68. Denmark also achieved 68 points, making it one of the most dynamic and innovative countries in the world. It improved its performance by an impressive eight points and leads in several key technologies, such as energy technology and biotechnology. The positive developments in Singapore and Denmark have brought the top three countries closer together, while the gap between them and the next-ranked countries—Sweden (58 points), Ireland (55), and Finland (52)—has widened significantly.

No major industrial nation ranks in the top 10

After several smaller countries, including Belgium and Austria, the first major industrial nation appears at rank 11: South Korea with 44 points. Germany follows closely behind with 43 points (-2), dropping from 10th to 12th place, and the UK comes in at 13th place with 42 points (+1). The United States, with 35 points (-7), ranks 18th, continuing the slow but steady decline in innovation capacity that began in the mid-2000s. The temporary stabilization of this trend was largely undone by the Covid-19 pandemic.

The pandemic also impacted China. As the only major economy to have shown a highly dynamic improvement in innovation capacity over the years, China consistently climbed the rankings. Currently, it holds 25th place, ahead of countries like Italy and Japan. However, the pandemic and the government’s strict isolation measures have led to stagnation in China’s innovation index since 2020.

Japan ranks 28th, at the lower end of the middle field. Japan's relatively poor performance in the innovation ranking is largely due to weaknesses in areas such as international networking, scientific performance, workforce availability, and government investment in research and development. Despite Japan's strong performance in R&D activities, patent applications, and high technology, these strengths are not enough to significantly improve its overall ranking.

The lower positions in the ranking are occupied by seven emerging economies, including four of the five BRICS countries (Brazil, Russia, India, South Africa) as well as Turkey, Mexico, and Indonesia. With indicator scores ranging between 21 and 12 points, they hold the lowest rankings on the list.

Small countries often highly specialized, larger ones more diversified

The size of an economy significantly influences its scientific and technological specialization, and consequently its innovation activities. Smaller countries need to specialize to effectively focus their limited resources and capabilities. This allows them to more easily excel in specific areas, as seen with Switzerland, Singapore, and Denmark—countries whose scientific institutions rank among the best in the world. Their industries focus on fields like pharmaceuticals, biotechnology, electronics, and automation, where new research is particularly crucial.

Larger countries, on the other hand, have a much broader range of activities. This diversity allows them to reach critical mass in many sectors, but it can also reduce their overall effectiveness. Larger economies are also more heterogeneous, meaning specific activities are often regionally concentrated, such as in clusters. When looking at individual U.S. states, for example, the country’s overall weak innovation ranking is somewhat mitigated. Massachusetts would rank 7th and California 13th if assessed independently, both much higher in the global comparison. Similarly, the German state of Baden-Württemberg would rank 4th, just behind Switzerland, Singapore, and Denmark. Therefore, while some regions in these countries are global innovation leaders, others lag significantly behind.

The 2024 Innovation Indicator reflects the state of global innovation activities, based on scientifically backed data collected and analyzed by the Fraunhofer Institute for Systems and Innovation Research (ISI) and the Centre for European Economic Research (ZEW). The report highlights that a country’s current position is the result of past decisions and investments. Conversely, the future is shaped by the actions taken today. Countries that fail to implement the necessary measures now will struggle to keep up with the innovation leaders in the coming years. In Germany, for instance, there is a concerning investment backlog in many areas. Cutting back on building new knowledge and technologies threatens Germany’s competitiveness, especially as more countries enter the innovation race.

The Innovation Indicator 2024, presented by Roland Berger and the Federation of German Industries (BDI), analyses the development of the innovation capacity of important economies.

Innovation is the key to a sustainable, prosperous future

Modern industrial societies require innovation to simultaneously grow and sustainably operate. Without radical transformation and consistent innovation, these countries cannot maintain their competitiveness and the prosperity of their citizens.

Sustainability & Climate Action

Roland Berger is a thought leader in environmental issues and the response demanded from organizations. Our publications cover all relevant areas.

Smart Mobility

Smart Mobility contributes to a more sustainable and value-adding state of mobility. Find out more about new mobility scenarios for our society here.

• Publications

Global Innovation Index

The Global Innovation Index (GII) ranks the innovation performance of some 131 countries and economies around the world, based on 80+ indicators.

Co-published by WIPO, Cornell University and INSEAD, the report provides an annual ranking of the innovation capabilities and performance of economies around the world.

Release of the GII 2021

Tracking Innovation through the COVID-19 Crisis

Global Innovation Index 2023, 16th Edition Innovation in the face of uncertainty

Author(s): WIPO; Publication year: 2023

The Global Innovation Index 2023 (GII) takes the pulse of innovation against a background of an economic and geopolitical environment fraught with uncertainty. Tracking the most recent global innovation trends, the GII finds that – despite a climate of disquiet and a decline in risk capital investment – opportunities abound as a result of the incipient Digital Age and Deep Science innovation waves. At its core, the GII 2023 reveals who is leading in global innovation, ranking the innovation performance of 132 economies and highlighting their strengths and weaknesses. In addition, it identifies the world’s top 100 science and technology clusters. The GII is a “tool for action” regarding innovation policy. Governments around the world have used the GII to benchmark innovation performance, perfect innovation metrics and, ultimately, to shape evidence-based innovation policymaking. In the context of the United Nations Sustainable Development Goals (SDGs), since 2019, the GII has been recognized by the United Nations General Assembly to be a benchmark for measuring innovation, including more recently in a post-pandemic environment.

Global Innovation Index 2023 Executive Summary

The GII 2023 Executive Summary provides key highlights and results presented in the full GII 2023: Innovation in the face of uncertainty report. The GII 2023 reveals who is leading in global innovation, ranking the innovation performance of 132 economies and highlighting their strengths and weaknesses. In addition, it identifies the world’s top 100 science and technology clusters.

Global Innovation Index 2022, 15th Edition What is the future of innovation driven growth?

Author(s): WIPO; Publication year: 2022

The Global Innovation Index 2022 (GII) tracks global innovation trends against the background of an ongoing COVID-19 pandemic, slowing productivity growth and other evolving challenges. The GII reveals the most innovative economies in the world, ranking the innovation performance of 132 economies, highlighting their innovation strengths and weaknesses, and pinpointing any gaps in their innovation metrics. This 2022 edition of the GII focuses on the effect of innovation on productivity and wellbeing of society over the coming decades.

Global Innovation Index 2022, 15th Edition Executive Summary

Global Innovation Index 2021, 14th Edition Tracking Innovation through the COVID-19 Crisis

Author(s): WIPO; Publication year: 2021

The Global Innovation Index 2021 takes the pulse of the most recent global innovation trends and ranks the innovation ecosystem performance of 132 economies, while highlighting innovation strengths and weaknesses and particular gaps in innovation metrics. In its new Global Innovation Tracker section, the report draws on a select set of indicators, including the effects on research and development expenditures or access to innovation finance, to provide a perspective on the impact of COVID-19 on global innovation performance.

Find out more: Press release | Home Page | Infographic

Global Innovation Index 2021, 14th Edition Executive Summary

Global Innovation Index 2020 Who Will Finance Innovation?

Author(s): Cornell University, INSEAD, WIPO; Publication year: 2020

The Global Innovation Index 2020 provides detailed metrics about the innovation performance of 131 countries and economies around the world. Its 80 indicators explore a broad vision of innovation, including political environment, education, infrastructure and business sophistication. The 2020 edition sheds light on the state of innovation financing by investigating the evolution of financing mechanisms for entrepreneurs and other innovators, and by pointing to progress and remaining challenges – including in the context of the economic slowdown induced by the coronavirus disease (COVID-19) crisis.

Find out more: Press release | Home page | Infographic – China

Global Innovation Index 2020 - Executive version Who Will Finance Innovation?

Global Innovation Index 2020 - KEY FINDINGS

Who Will Finance Innovation?

Global Innovation Index 2019 Creating Healthy Lives — The Future of Medical Innovation

Author(s): Cornell University, INSEAD, WIPO; Publication year: 2019

The Global Innovation Index 2019 provides detailed metrics about the innovation performance of 129 countries and economies around the world. Its 80 indicators explore a broad vision of innovation, including political environment, education, infrastructure and business sophistication. The GII 2019 analyzes the medical innovation landscape of the next decade, looking at how technological and non-technological medical innovation will transform the delivery of healthcare worldwide. It also explores the role and dynamics of medical innovation as it shapes the future of healthcare, and the potential influence this may have on economic growth. Chapters of the report provide more details on this year’s theme from academic, business, and particular country perspectives from leading experts and decision makers.

Find out more: Press release | Home page

Global Innovation Index 2019 - Executive version Creating Healthy Lives — The Future of Medical Innovation

Global Innovation Index 2019 - KEY FINDINGS

Creating Healthy Lives—The Future of Medical Innovation

Global Innovation Index 2018 Energizing the World with Innovation

Author(s): Cornell University, INSEAD, WIPO; Publication year: 2018

The Global Innovation Index 2018 provides detailed metrics about the innovation performance of 126 countries and economies around the world. Its 80 indicators explore a broad vision of innovation, including political environment, education, infrastructure and business sophistication. The GII 2018 analyses the energy innovation landscape of the next decade and identifies possible breakthroughs in fields such as energy production, storage, distribution, and consumption. It also looks at how breakthrough innovation occurs at the grassroots level and describes how small-scale renewable systems are on the rise.

Find out more: Press release | Global rankings | Videos on YouTube | Photos (on Flickr) | Social media kit | GII in the news | Launch event flyer | GII events

Global Innovation Index 2017 Innovation Feeding the World

Author(s): Cornell University, INSEAD, WIPO; Publication year: 2017

The Global Innovation Index provides detailed metrics about the innovation performance of 127 countries and economies around the world. Its 81 indicators explore a broad vision of innovation, including political environment, education, infrastructure and business sophistication. This year’s report reviews the state of innovation in agriculture and food systems across sectors and geographies. Chapters of the report provide more details on this year’s theme from academic, business, and particular country perspectives from leading experts and decision makers.

Find out more: Press release | Global rankings | Videos | Launch event photos

Global Innovation Index 2016 Winning with Global Innovation

Author(s): WIPO, Cornell University, INSEAD; Publication year: 2016

The Global Innovation Index ranks the innovation performance of 128 countries and economies around the world, based on 82 indicators. This edition explores the impact of innovation-oriented policies on economic growth and development. High-income and developing countries alike are seeking innovation-driven growth through different strategies. Some countries are successfully improving their innovation capacity, while others still struggle.

Find out more: Press release | Videos and infographics

Global Innovation Index 2015 Effective Innovation Policies for Development

Author(s): WIPO, Cornell University, INSEAD; Publication year: 2015

The Global Innovation Index ranks the innovation performance of 141 countries and economies around the world, based on 79 indicators. This edition explores the impact of innovation-oriented policies on economic growth and development. High-income and developing countries alike are seeking innovation-driven growth through different strategies. Some countries are successfully improving their innovation capacity, while others still struggle.

Find out more: Launch event | Press release | Videos and infographics

Global Innovation Index 2014 The Human Factor in Innovation

Author(s): WIPO, Cornell University, INSEAD; Publication year: 2014

The Global Innovation Index ranks the innovation performance of 143 countries and economies around the world, based on 81 indicators. This edition explores the role of the individuals and teams behind the innovation process. It sheds light on different aspects of human capital required to achieve innovation, including skilled labor; the intersection of human, financial and technological capital; talent retention; and the mobilization of highly educated people.

Global Innovation Index 2013 The Local Dynamics of Innovation

Author(s): WIPO, Cornell University, INSEAD; Publication year: 2013

The Global Innovation Index ranks the innovation performance of 142 countries and economies around the world, based on 84 indicators. This edition explores the impact of innovation-oriented policies on economic growth and development. High-income and developing countries alike are seeking innovation-driven growth through different strategies. Some countries are successfully improving their innovation capacity, while others still struggle.

Find out more: Press release

Global Innovation Index 2012 Stronger Innovation Linkages for Global Growth

Author(s): WIPO, INSEAD; Publication year: 2012

The Global Innovation Index ranks the innovation performance of 141 countries and economies around the world, based on 84 indicators. This edition explores the impact of innovation-oriented policies on economic growth and development. High-income and developing countries alike are seeking innovation-driven growth through different strategies. Some countries are successfully improving their innovation capacity, while others still struggle.

Global Innovation Index 2011 Accelerating Growth and Development

Author(s): INSEAD; Publication year: 2011

The Global Innovation Index ranks the innovation performance of 125 countries and economies around the world, based on 80 indicators. This edition explores the impact of innovation-oriented policies on economic growth and development. High-income and developing countries alike are seeking innovation-driven growth through different strategies. Some countries are successfully improving their innovation capacity, while others still struggle.