Composite indicators of labour market regulations in a comparative perspective

Abstract

Technological progress and the internet brought about new possibilities of creating, storing, exchanging, replicating, and using various kinds of data for research. This paper discusses some of the dangers embedded into the reuse of data produced by some institutions by other institutions through the combination and aggregation of initial data into various data products. To this end, we use an example of labour market regulations’ indicators developed by the World Economic Forum, the International Institute for Management Development (IMD), and the Fraser Institute, which are all partly based on the World Bank Employing Workers Indicator. We document how these three indicators compare and identify both their common and specific limitations. For each of these indicators, the choice of subcomponents and of aggregate techniques results in different pictures of labour market regulations, despite the use of the overlapping initial sources of data. Our comparative exercise calls for continuous efforts to improve the indicators of labour market regulations, as well as for cautious use of such indicators for research and policy advice.

JEL codes: J00, J8, Y1

1 Introduction

The past decades witnessed a significant burgeoning of empirical studies examining the effect of labour market institutions on various measures of economic and labour market performance.¹ This burgeoning was naturally propelled by the global economic crisis and the need to seek empirically grounded policy responses to persisting unemployment and widening inequalities. It was also facilitated by the development of cross-country time-series composite indicators of labour market institutions. Some of this development was facilitated by technological progress and the widespread use of the internet that brought about new possibilities of creating, storing, exchanging, replicating, and using various kinds of data for economic research. However, the question of the robustness of empirical studies based on such indicators has become increasingly critical.

This paper examines some of the dangers embedded into what can be called data repackaging—i.e. utilization of data produced by some organisms by other organisms through the combination and aggregation of initial data into various other data products, such as composite indicators. This process is sometimes accompanied by data misclassification and misinterpretation, as well as by poor acknowledgement of the nuances in the initial data. It can also be viewed as part of “mashuping” of indicators—a term coined by Ravallion (2012)² and defined as a process of creating composite indicators in which the data producer is only constrained by the availability of data in choosing what variables to include and by their weights, but not by theory or empirical practice.

To demonstrate our point, we chose to focus on three composite indicators developed for measuring labour market regulations, competitiveness, and efficiency, by three internationally renowned think-tanks. These are the Labour Market Efficiency Index developed by the World Economic Forum (WEF LME), the Government Efficiency Index and its labour regulation components developed by the International Institute for Management Development (IMD), and the Fraser Institute Labor Market Regulations (Fraser LMR) index.

These three indicators are chosen for several reasons. First, they are widely used in policy-oriented research on the role of labour market institutions. A very partial list of the most recent works based on either of these indicators includes Freeman et al. 2008; Feldman 2007, 2009; Javorcik and Spatareanu 2005; Bernal-Verdugo et al. 2013, 2015 (criticized by Aleksynska 2015a, 2015b); or Prados de la Escosura 2016 (who also discusses some of these indicators’ drawbacks). These indicators are also widely cited in the media and are seriously taken by businesses and by governments for formulating policy advice, especially in countries with rich reform agenda, such as Ukraine, or countries of the Arab Spring world.³ Second, all three indicators share the same characteristic: they aim at capturing labour market institutions in general and at describing them with one number. In this respect, they are different from, for example, the widely used and acclaimed OECD Employment Protection Legislation (EPL) indicator or a recently launched ILO EPLex indicator, which are aimed at describing in depth one specific labour market institution—employment protection legislation. It is for this reason that such other indicators measuring just one type of institution are not considered in the systematic analysis of this paper, though we occasionally refer to them. Lastly, all three indicators under review are partly or fully based on data produced by other institutions. This is a common feature of the three indicators, which is not shared by some other famous indicators in the field, such as the Deakin et al. (2007) indicators. One consistently used data source for the three indicators under review is the Employing Workers Indicator (EWI), a subindicator of the Ease of Doing Business indicators developed by the World Bank. The Fraser LMR indicator also contains data produced by the WEF and IMD.

In what follows, we provide a critical overview and a comparison of these indicators of labour market regulations. Our main findings are three.

First, we highlight that while each of these indicators claims to be original and having a somewhat different objective from others, each of them systematically uses the whole, or parts of, the World Bank EWI—a series of indicators developed by the World Bank and measuring some specific aspects of labour regulations. Originally, EWI was part of the aggregate Ease of Doing Business indicator used to rank countries. However, its use in the aggregate Ease of Doing Business indicator has been discontinued by the World Bank, though the Bank still collects the data. In systematically using the World Bank EWI data, the WEF, IMD, and Fraser take liberty in renaming and rescaling various components of the original World Bank Employing Workers Indicator. But mainly, they fail to properly acknowledge the appropriateness of using this source to construct their own composite indicators. They also fail to properly track and acknowledge the methodological changes in the original data series. In fact, the World Bank EWI has been extensively reviewed and assessed as not being suitable for inclusion into the overall aggregate Doing Business indicator or for ranking countries. Following initial important criticism from academia, civil society, and other international organizations (see notably Berg and Cazes 2008; Lee et al. 2008), it underwent an independent evaluation (World Bank 2008), an examination by a consultative group (World Bank 2011), and a review by an independent panel (World Bank 2013). These evaluations resulted in several significant methodological changes introduced to the EWI. Most importantly, it was determined that, although the World Bank may continue gathering raw data underlying EWI, the World Bank would suspend using the EWI to calculate the aggregate Ease of Doing Business indicator or rank countries based on it, as well as would stop referring to the EWI when formulating policy advice. According to the Final Report of the Independent Panel (ibid), “The Bank’s decision to suspend the EWI acknowledged the problems inherent in measuring only the costs of labour-market regulation and not the benefits. The Panel agrees with the Bank’s reasoning [World Bank, 2009] that ‘a comprehensive approach in advice on labour market policies is needed’”. However, not only do the methodological notes accompanying the three indicators under the review of this paper not acknowledge these important debates but also they use the EWI data to construct their own composite indicators and to rank countries, thus disregarding significant recommendations and the decisions taken by the World Bank itself. Thus, another objective of this paper is to raise awareness among the WEF, IMD, and Fraser data users about the debates surrounding the data used to construct these composite indicators.

Second, we uncover additional caveats in the three respective composite indicators. These include trivial aggregation of de jure and de facto measures, the use of opinion surveys that may not represent all types of firms, and frequent methodological changes, which render data incomparable over time. These methodological changes include adding or removing individual components from aggregate indicators from one data year to another and changing the methodology and definitions of these components. Some of the composite indicators contain sample-dependent data, which should neither be aggregated with data from other sources nor used in comparative purposes over time if the sample changes—and this is not always the case.

Third, we show how, despite the use of the same common source, resulting composite indicators produced by the three intuitions differ from each other. As a result, each of the indicators provides a different outlook of labour market regulations, potentially creating confusion for policymakers. The uncovered limitations and shortcomings in the three composite indicators do not help advancing the controversial debate on the role of labour market institutions.

While previous research has already highlighted several problematic areas with the composite indicators that we consider (for example, Ochel and Röhn 2006 review the overall IMD, WEF, and Fraser indices prior to 2005, not just their labour regulations subcomponents), the wide recent use of subcomponents measuring labour market regulations for ranking countries and providing policy advice calls for a more in-depth overview of such specific components based on the post-2005 data. Our paper also rejoins a growing literature that is critical of composite indicators more generally, such as Ravallion (2012) and Stiglitz et al. (2009) who provide a general perspective on advantages and disadvantages of composite indicators in various fields, without focusing on labour market regulations specifically.

The three reviewed indicators, and the databases to which they are associated, are somewhat different from the “internet” data in the strict sense that has recently emerged in social sciences (Askitsas and Zimmemann 2015), as they are not generated by simple internet users, for example, through their search activities. However, they are similar to the “internet data” in the sense that the fast development of technologies permitting instantaneous, open, unlimited, and mostly free access to data generated by some organisms to other organisms facilitates the development of secondary online databases and composite indicators based on them. It helps to save on costs of original data collection and also of data use, as original sources provide data in a ready-to-use standardized format. The digital revolution also permitted virtually unlimited storage of newly created data products and allowed saving costs associated with their publicizing on the net. Thus, we can expect to see a growing number of similar databases and indicators in the future, an expectation also flagged by Ravallion (2012).⁴

The rest of the paper is organized as follows. Section 2 provides a comparative overview of the selected indicators. It shows how the World Bank Employing Workers Indicator is systematically found among the Fraser, WEF, and IMD composite indices’ ingredients. It also examines the appropriateness of the underlying subcomponents, data sources, and aggregation techniques adopted by the three indicators. Section 3 explores how the selected indicators correlate with each other and with key labour market outcomes. Finally, Section 4 concludes.

2 Comparative overview of selected composite indicators

2.1 Methodological description of the three composite indicators under review

In a world with unlimited information and numerous and growing measures of institutions and environments, composite indicators gained popularity because of their ability to describe a complex reality by means of few, sometimes just one, number. They are often easier to interpret and compare across countries and over time than a battery of separate indicators. Composite indicators are used by businesses to rank countries in order to assess their investment attractiveness, as well as by governments to see their relative position and benchmark best practices. They are also extensively used by academic researchers in need of cross-country time-series parsimonious data. However, if poorly constructed or misinterpreted, composite indicators may also send misleading policy messages and lead to simplistic conclusions (Saisana and Tarantola 2002; OECD 2008; Stiglitz et al. 2009).

The three composite indicators considered in this paper are produced by three international think-tanks in order to rank countries on competitiveness and obstacles to growth (WEF and IMD) or on economic freedom (Fraser). Each of these indicators is composed of dozens of baseline indicators, which are usually first aggregated into subindicators. The WEF and Fraser contain such composite subindicators of labour market institutions that we will thoroughly review. While the IMD does not have a separate subindicator for labour area, individual components related to labour institutions are part of the government efficiency subindicator that in its turn constitutes part of the global efficiency and competitiveness indicator.

Table 1 presents a summary overview of the composite subindicators examined in this paper and governing labour regulations area. The exact topics covered differ across the indicators and range from survey questions on labour-employer relations and wage determination to statistical data on female labour force participation, to de jure information on hiring and firing regulations. At the same time, significant topical overlaps can be observed. Each indicator under review also gives a different logic both to the collected data and to the labour market indicators based on them, though a broad name “regulations” or “institutions” may apply. Table 1 also includes the description of the World Bank Employing Workers Indicator, because, as will be shown further, it constitutes an important part of the three indicators under review.

Table 1 Overview of selected institutional databases and composite indicators, labour market subcomponents

As detailed in Table 1, the subindicator produced by the World Economic Forum, the Labor Market Efficiency (LME), is part of the broader Global Competitiveness Index (WEF GCI) which measures and ranks competitiveness throughout the world, defined as a “the set of institutions, policies, and factors that determine the level of productivity of a country” (WEF Reports, various issues). Along with the aggregate GCI index, the WEF LME index is used to rank countries and track changes in rankings over time. While some of the information pertaining to the functioning of the labour markets has been collected by the WEF for over two decades⁵, it is not before 2004 that the LME index emerged formally as one of the 12 “pillars of productivity and competitiveness” of which GCI is comprised. The overall CGI aggregation procedure is fairly complex, but the aggregation of the LME index is straightforward and is based on an arithmetic mean. In Appendices 1, 2, and 3, we carefully document the number of underlying components of the LME index (nine in the latest available report), their definitions, data sources, and evolution over time, which is particularly remarkable.

The aggregation methodology behind the WEF GCI overall composite indicator varied over the years. Starting from the 2009–2010 report, a uniform aggregation methodology was applied for computing the GCI in the following three steps. First, individual variables that form the basis for the index are split into 12 topical areas. They are averaged within these topical areas using a simple arithmetic mean to produce the values for the 12 “pillars of competitiveness”, LME being one of them. Second, these pillars are aggregated into three subindexes, each pillar getting a fixed weight. For example, the LME pillar is aggregated into the Efficiency Enhancers subindex with the weight of 17 %. Third, the three subindexes are aggregated into the overall CGI but with weights that vary across countries, depending on the development stage of an economy. To determine this stage, countries are subdivided into five groups based on their GDP per capita and exports of mineral goods. Further, a maximum likelihood regression of GDP per capita is run against each subindex for past years, allowing for different coefficients for each stage of development. Finally, econometric estimates from these regressions are rounded to produce the stage-dependent weights for the three subindexes. The robustness and validity of this methodology was assessed and confirmed on numerous occasions (see, for example, WEF 2010–2011 Report, Chapter 1.1). However, the fact that the outcome measure (GDP) is used as an input measure (for producing weights for CGI subindexes) means that, in principle, the CGI should not further be used in regression analysis, especially when involving the assessment of its impact on development or on macroeconomic outcomes highly correlated with GDP.

The IMD World Competitiveness Yearbook has been published since 1987, and until 1996, it was a joint publication with the World Economic Forum. The IMD does not have a special index governing labour area, but four individual components related to this topic—labour regulations, unemployment legislation, immigration laws, and redundancy cost—are used to construct a broader Government Efficiency index. Appendix 4 outlines the definitions and sources of these components. Starting from 2001, the methodology behind these components, and behind the Government Efficiency index, unlike the WEF or Fraser indices, remained relatively unchanged. The IMD ranks and analyses the capability of nations to provide a framework in which companies can compete. The IMD also reports country rankings separately for each individual component.

The IMD aggregation methodology is based on the following principles. The national environment is divided into four main factors: (1) Economic (domestic) Performance, (2) Government Efficiency, (3) Business Efficiency, and (4) Infrastructure. Each factor is divided in 5 subfactors, for a total of 20 subfactors. These 20 subfactors consisted of over 300 criteria in 2012. The information on labour markets comes under the Government Efficiency Factor and Business Legislation Subfactor (Labour Regulations sub-subfactor) but also under the Business Efficiency factor and Labour Market subfactor. It is the Labour Regulations sub-subfactor that is the most relevant to measuring labour market institutions and also the most comparable with other indicators; hence, our focus is exclusively on it. In contrast, the Labour Market subfactor measures such issues as labour costs and remuneration, working relations, and skills. It also contains questions related to assessing brain drain (a similar component is part of the Labour Market Efficiency Index for GCI). From 2004, hard data and survey data are accorded different weights in the overall ranking. From 2003, 8 regional economies were added to 51 national economies (the regional economies were removed from 2007 onwards). In addition, ranks were split in two groups by population size (over 30 million and smaller than 30 million). From 2004, a customized ranking based on GDP level, population size, and region is provided in addition to the overall ranking.

Finally, the Labor Market Regulations (LMR) index produced by the Fraser Institute ranks countries according to their labour market regulations rigidity and is part of their Economic Freedom of the World index. As economic “freedom” is considered to be an ultimate value, any institutional or custom regulation is viewed as reducing such freedom. The Fraser Economic Freedom of the World database has been collected since 1997, with data related to labour market regulations since 2001. In the 2012 edition, the Fraser Economic Freedom of the World index was based on five topical areas: Size of Government, Legal System and Property Rights, Sound Money, Freedom to Trade Internationally, and Regulation. Labour institutions is one of the three subcomponents of the Regulation area, and is based, at the lowest level of disaggregation, on six components outlined in Appendix 5. In total, the Fraser Economic Freedom of the World index is made up of 24 subcomponents and 42 distinct components. Each subcomponent and component is measured on a scale from 0 to 10, 0 reflecting most restrictive regulations, or the lack of economic freedom. The six components of Labour institutions are averaged (by taking a simple average) to produce a Labour Market Regulation index subcomponent. Likewise, simple averages are taken to produce indices of topical areas and the overall index. Country rankings are based on these averages of subcomponents, areas, and the overall index.

Both the WEF and IMD are partly based on primary data coming from the WEF and the IMD Executive Opinion Surveys, respectively, though they also contain data from other sources, such as international organizations (ILO and the World Bank). Fraser, however, is just a data repository, containing WEF and IMD questions, as well as the World Bank data and also data produced by other institutions.

2.2 Comparative review of the subcomponents of composite indicators. The resilience of the World Bank Employing Workers Indicator

The quality and reliability of any composite indicator is necessarily dependent on the choice of the underlying components, their sources, and on the aggregation scheme. In general terms, for any given indicator to be sound, it should be based on a solid theoretical framework for both the selection and combination of single components into a meaningful composite index. Individual components should also be selected on the principles of analytical soundness, comprehensiveness, pertinence to the measured phenomenon, and also relationship with other individual components (OECD 2008). Thus, in what follows, we examine the pertinence of the selected indicators according to these criteria.

Appendices 1, 2, 3, 4, and 5 provide the full list, description, and changes over time of the components for selected composite indicators. One of the most striking features of the indicators under review is that—despite the differences in their stated objectives of what they intend to measure—they contain overlapping ingredients. Fraser is just a data repository, containing WEF and IMD questions. Importantly, all three datasets systematically include the World Bank Employing Workers Indicator (WB EWI) as one of their main sources.

Table 2, column 1, shows the WB EWI structure. It consists of the Rigidity of Employment index, which is based on three subindicators: the Difficulty of Hiring index, Rigidity of Hours index, and difficulty of redundancy index, all of which have several components in their turn. In addition, the World Bank also publishes the Redundancy Cost indicator. Table 2 further shows how these data have been used by other institutions in the construction of their indices. It reports data starting from 2007, as this is the year when the World Bank data started being used in other datasets. Three main features can be noted.

Table 2 The resilience of the World Bank Employing Workers Indicator

First, there is a significant variation both across indicators and over time in the type and the number of WB EWI components used. Both the WEF and IMD initially use fully the WB EWI data. In 2012, however, both use only Redundancy Costs, most probably because, starting from 2011, the WB does not report the Rigidity of Employment index anymore but only the raw data which underlies this index. In contrast, note how Fraser uses only some, but not all, subcomponents of the Rigidity of Employment index, without justification of the choice.⁶ Note also that the Nonwage labour costs component is collected and reported by the World Bank, but has not been used for constructing indicators or country rankings—while both the WEF and Fraser, in some editions, have done precisely this.

Another issue relates to the time inconsistency with which the WB EWI data are used in the WEF and IMD composite indicators. Whenever the year of the WB EWI data can be inferred from the accompanying methodological reports, one can see that the WB EWI data was used without consistent updates (as in WEF reports 2009 and 2010), or with time jumps (as between WEF reports 2011 and 2012), or with some of the components relevant for one year and other components relevant for another year (IMD report 2011). Note that Fraser does not report exactly which WB EWI edition it uses. Given this, whenever countries experienced labour market institutional reforms, the timing of these reforms cannot be tracked properly in the composite indicators using the WB EWI.

Last, but not least, the WEF, IMD, and Fraser change the names of WB EWI components to adjust them to the logic of their indices. Both Fraser and the IMD also systematically apply transformation formulas to these data (see Appendices 4 and 5).

These observations are important for several reasons. The fact that the EWI data appear systematically as part of other composite indicators means that the very significant decisions taken by the World Bank not to use EWI for aggregating it with other indicators or for ranking countries are disregarded by the IMD, WEF, and Fraser. Moreover, this also means that the technical data concerns in the original source are carried over to other composite indicators. Renaming and rescaling the ingredients by the WEF, IMD, and Fraser also means that the problematic areas are more difficult to track and that potential problems accumulate. One remarkable example is the changing methodology in the EWI data in several editions of the WB reports. Especially significant changes were made in 2011 following up the WB consultative process⁷ and explicitly acknowledged by the World Bank.⁸ These changes also meant that there were important breaks in the data series, but they are especially difficult to track in the used composite indicators if changes are not explicitly acknowledged.

Rather, the WEF 2012–2013 report devotes as much as a footnote acknowledging “further minor adjustments to the data” (footnote 23 on p. 44 of 545). In the IMD, there is no attention drawn to the changing methodology in the Redundancy Cost variable, the IMD only adjusts the definitions from one report to the other. Neither does the Fraser 2012 Report attract the user’s attention to the changes in the WB components, although it does report new definitions in the appendices. Notably, the definition of the Fraser Hours Regulations component is accompanied by the following note: “This component was previously denoted ‘Mandated cost of hiring a worker’ […] In recent years, the entire labour market area has been dropped from the Ease of Doing Business project; however, they continue to present the data in a separate section. In order to maintain as much consistency over time as possible, we have revised the dataset back to 2002 with these data replacing the previous values”. The report remains silent on how this revision was effectuated: through additional data collection, extrapolation, or any other means. The World Bank Ease of Doing Business data for this component are available only from 2006 onwards.

While the WEF does not make any adjustment to methodological changes, the IMD states that “whenever there is a major change in methodology, all results are recalculated for the past 5 years in order to ensure a high degree of compatibility with past results”. However, comparisons across reports suggest that changes in this variable were not considered as “major” in order to recalculate past values in new reports. The Fraser Institute makes more efforts in this field with its chain-linked methodology in constructing the overall indices to correct for changing components and missing values for comparisons over time. However, the chain-linked index is only available for those countries that exist in the composite indicator since 2000, but not for those that were added later. In the 2012 edition, the chain-index methodology is applied only to 74 out of 144 countries.

Apart from containing the World Bank EWI subcomponents, all three datasets include different ingredients, without any apparent justification of how and why these specific variables are chosen. Arguably, the chosen components do not cover the topic of labour market efficiency, regulations, or institutions exhaustively, leaving aside other important aspects such as worker protection, safety and health, social security provisions, and labour administration. At the same time, there is a significant repetitiveness of questions within the same indicators, which is especially pronounced in the case of Fraser: hiring and firing regulations in the latest available editions are measured in various ways by three out of six variables that make up part of the aggregate LMR index (Appendix 5: Hiring regulations and minimum wages and Mandated cost of worker dismissal based on the WB EWI, as well as Hiring and Firing question from the WEF). The lack of theoretical background or sound analytical approach for the choice of specific components compromises the indicators’ comprehensiveness. Repetitiveness of questions, in its turn, means that certain included concepts are double-counted and that the aggregated indicators are biased in the direction of these over-represented concepts.

All three composite indicators contain subcomponents based on opinion surveys. Such questions can only be valuable when they are properly phrased and contain clear and exhaustive concepts. However, in some instances, the overall phrasing of the questions does not seem fortunate. For example, the IMD Labor relations question is phrased as follows: “Labor regulations (hiring and firing practices, minimum wages etc.) … 1 = are not flexible enough, 6 = are flexible enough”. The questions seems to cover too many issues which may be contradictory: some countries may have flexible firing and hiring practices but inflexible minimum wages, either in their level, setting, or degree of formality. It is thus not clear what aspect of labour regulations the response actually reflects. Another example is the WEF Flexibility of Wage Determination question, which concerns the degree of centralization of the wage setting. It ignores, however, the coordination of wage bargaining, and no supplementary question exists to measure it (Ochel and Röhn 2006).

Finally, there is also a significant variation of components not only across but also within the indicators, both in terms of their number, type, or changing definition of the same component over time. These changes are especially apparent in the case of the WEF and Fraser. Since its first appearance in the 2004–2005 report, the WEF LME indicator underwent six changes in the number and types of included components (Appendices 1, 2, and 3). Since 2001, the Fraser LMR index underwent four changes in the number and types of included components and also numerous additional changes in the definitions and data sources of these components (Appendix 5). As both the WEF and Fraser use secondary sources to collect data, a regular change in the subcomponents creates a feeling that there is a “shopping for questions” driven by concerns such as data availability from other sources, rather than by theoretical underpinning or empirical relevance. These regular changes also compromise the use of these data for time-series analysis.

But to illustrate further how critical this issue of changing components within indicators can be, consider the following example. For the WEF, both the overall GCI and the specific LME index are used to track changes in country rankings from one year to another, without taking into account the almost annual changes in the methodology (components). That is, each WEF annual report provides annual changes in the GCI, as well as changes in the country rankings based on it, showing the “progress” of countries towards competitiveness or away from it. However, it is impossible to judge whether countries move in the rankings because of the genuine changes in the underlying components or because of the changes of the components.

The issue can be illustrated using two of the latest available reports, which witnessed changes in two LME components (Appendix 3): the Rigidity of Employment component is used in 2011–2012 but not in 2012–2013 report, and the definition of Redundancy Costs was modified. It is also helpful to find a country which, say, has an increase in the LME score but a decrease or no change on all those individual components making part of the index that remain the same between the two reports. Such is the case of France, for instance (Table 3). It experienced an upward change in the aggregate LME index (Table 3, column 1). However, it experienced a downward change in seven out of ten individual components constituting LME in 2011 (columns 2–8). The change in the eighth component, Female Labour Force Participation (column 9), is too small to imply significant changes in the indicator. Dropping the ninth component, Rigidity of Employment component (column 11), between the two reports necessarily drives the index further down. With the jump from 32 to 12 weeks, Redundancy Cost (column 10) is the only component responsible for the reported upward change in the LME index.⁹ This change happened exclusively because of the change in the component definition; no actual change in the Redundancy Cost in France took place in that year. The change in this component is sufficiently large to offset both the opposite-direction changes on other components and the opposite-direction change of the index due to dropping of the Rigidity of Employment component.

Table 3 Evolution of the LME and its underlying components. Example of France

All in all, because with each change in the methodology the data are not revised backwards, neither LME, nor GCI, nor country rankings based on these indices are suitable for comparisons over time.

Given all of the above, it remains highly questionable to what extent the three composite indicators under review can be used for comparing countries and their regulations over time. Furthermore, the World Bank no longer uses these data for ranking countries and neither do they aggregate them with other Ease of Doing Business data, as part of the strategic methodological decision. In contrast, this is precisely what Fraser, the WEF, and the IMD are doing. They provide contemporaneous country rankings and, in some instances, comparisons of how the rank positions of countries change over time, ignoring the above-mentioned debates and their constructive outcomes.

2.3 Shortcomings linked to certain data sources

The sources of the underlying data play a critical role in determining the appropriateness of any given indicator. Beyond the systematic inclusion of the WB EWI, two further issues can be highlighted: the use of opinion surveys and the use of different data sources for the same variable.

Indeed, one of the common features of the reviewed datasets is their use of opinion surveys as one of the main sources of information. Both the WEF and IMD (and hence Fraser as a repository) are using their own Executive Opinion Surveys to collect data on labour market rigidities and on other relevant aspects of labour markets. As suggested by their name, respondents are business executives ranging from top to middle management, and the reported answers hence reflect only the employers’ viewpoint on labour relations and sometimes with the priors reflecting other countries’ experiences. Chor and Freeman (2005) contrasted the WEF/Fraser responses with the responses of union officials, activists, and professors of labour law and industrial relations, to actually find a strong similarity of the viewpoints on de facto labour practices across these different groups of stakeholders.

However, sampling modalities and respondents’ selection are different between the WEF and the IMD. For example, the WEF is carrying out a randomized sampling which accounts for the sector and firm size distribution of economies. Unlike the WEF, the choice of firms participating in the IMD survey accounts only for the industries or sectors, but does not seem to account for the size of the enterprise. This is rather critical for assessing the regulations, as both provisions and their enforcement may vary across firms of different size. Collective bargaining outcomes also differ tremendously along this dimension: they may have significant consequences in sizable firms but be virtually inexistent in small firms. These differences may severely affect the way business executives perceive overall labour regulations in a country, not just in their firm. Furthermore, the IMD survey is conducted with “nationals or expatriates, located in local and foreign enterprises in the country and which, in general, have an international dimension … We try to contact most IMD alumni” (IMD 2012)¹⁰. While for some of the questions measuring economic performance such respondents may provide valuable insights, for questions of labour market regulations, they may not be the most suitable group. This is because firms with an international exposure may have labour practices different from the rest of the economy (at least because they are bigger in size), they cover mostly the formal sector, and the business executives of such companies—especially non-nationals—may have their own priors as to the functioning of labour markets.

Of note is also the fact that in countries where a large portion of disputes are solved in courts or with mediation, decisions ruling in favour of workers may be pro-cyclical (Bertola et al. 2000; Ichino et al. 2003). This means that the reported perceptions of strictness of regulations may reflect these decisions, and hence the economic conjuncture, rather than the factual strictness of regulations. The latter is particularly important if the data are further used in the regression analysis of economic conjuncture: simultaneity and reverse causality will likely be an issue.

Another problematic area with some of the composite indicators, and especially with Fraser, is the use of different data sources for the same variable. This is done in order to provide considerable time coverage to some of the LM components. Thus, the Fraser unemployment insurance component at some point was based on data from two different surveys, the IMD and WEF. However, as shown in Appendix 4, the IMD reports to users converted, rather than actual, survey data, where the converted values for each country are sample-dependent. The IMD uses a special transformation formula—a standard deviation method—which measures the relative difference between economies’ performances. It is this standardized value for each economy that is reported. As a result, even if the WEF and IMD have a similarly phrased question on unemployment legislation, the WEF reports survey responses that are independent of the responses provided in other countries of the sample, while the IMD reports converted scores that are dependent on the responses provided in other countries of the sample.¹¹ Thus, the IMD cannot be used as a complementary source of data. Furthermore, it cannot be aggregated with any other data covering out-of-the-IMD sample countries, because the IMD data values are sample-dependent.

2.4 The choice of aggregation schemes

There are (at least) three options to aggregate the different dimensions and create a synthetic indicator: use weights based on theoretical assumptions, give equal weights to each dimension, or define weights according to their statistical significance. The choice of a particular weighting scheme may give rise to substantial differences between indicators. Some similarities and differences in aggregation techniques across the reviewed composite indicators can be highlighted.

While weighting according to an underlying theoretical framework would be an ideal and robust choice, it is often easier and a priori legitimate to give equal weights to all dimensions.¹² This is the choice made by the WEF and Fraser to simply average the data, while the IMD gives higher weights to hard data as compared to survey data.

A related issue is then the type of data that is aggregated into an index. The WEF, IMD, and Fraser all aggregate de facto data, such as survey questions on the flexibility of regulations, with de jure information, such as weeks of advance notice and severance pay, also adding aggregated indices of the World Bank and, in some instances, “hard” statistical data. This different nature of the aggregated variables, and the way they are grouped together, raises concerns. Simply averaging different types of data does not allow to account for possible endogeneity between these types of information nor for the fact that some of the variables reflect outcomes (outputs) of the underlying processes measured by other variables (inputs) with which they are averaged. This is the case of the WEF LME, for example, which imbeds female-to-male employment rates and information on the difficulty of hiring and firing. The more there are different types of data included into an index, and the less the endogeneity of the components is taken into account, the more different the resulting indices are. In addition, aggregating the “outcomes” data with measures of institutions or perceptions is problematic if a composite indicator is further used to analyse the outcomes. This partly explains why different indices correlated differently with unemployment measures.

A further problem with simple averages is that, in the case of overlapping or repeating information in subcomponents, an aggregate index is biased towards that dimension (Ochel and Röhn 2006). This latter concern is especially high in the case of the Fraser LMR index, which contains questions measuring similar phenomena (three out of six, in the case of Fraser: see Appendix 5). A high correlation between some of these questions (shown below) suggests that there is a considerable double-counting when these individual components are averaged into aggregate indices: more weights are given to hiring and firing practices as opposed to other aspects of labour regulations. Despite this evidence, the authors of Fraser reports claim that such simple averaging of six components is the most objective way to reflect the unknown weights that may also be interdependent.

In addition to this, the Fraser composite indicator has one further serious shortcoming that makes it less than ideal for assessing the evolution of labour market regulations. It relates to the fact that the indicator is computed as a simple ad hoc average of components according to data availability. Clearly, some of the overall results are thus driven by data availability and by the values of available components. This is especially aggravated by the fact that one of the components—conscription—is remarkable for its discrete values (it takes only values such as 1, 3, 5, and 10, on a scale from 0 to 10) and for jumps in values (such as from 3 to 10 from one year to another).

The implications for cross-country comparisons of both averaging of available components and of including a technically different variable “conscriptions” are illustrated in Table 4. This Table compares Azerbaijan and Barbados in 2010. Barbados has significantly lower values of “Hiring and firing regulations” and of “Centralized collective bargaining” as compared to Azerbaijan. The data on three other components relevant for understanding labour markets in Barbados are missing. It is the particularly high value of the “conscription” variable that fully drives the overall result and gives the same overall Labor Market Regulation score to both countries—in a very misleading way. In total, we were able to count that, out of 143 countries in the Fraser sample with available data in 2010, 16 have missing data on at least one subcomponent, while aggregate index is reported for all of them. Out of 121 countries with data in 2003, 25 have missing data on at least one of the subcomponents, while the aggregate index is reported for all of them.

Table 4 Fraser 2012 report, data for 2010: comparing selected countries

The authors of Fraser reports use a correcting methodology (chain-linked index) in constructing the overall indices to account for changing components and missing values for comparisons over time. Also, revisions backwards are effectuated. However, they are sometimes effectuated only partially. In such a setting, knowing the historical evolution of components reported in Appendix 5 may be of help even for those users who use only the latest editions of the data revised backwards.

This issue is illustrated in Appendix 6. It records the way in which the data reported in accompanying databases varied (not just the data themselves) and uses Argentina as an example of data values. In 2010, new components were introduced by Fraser. Based on them, the composite indicator was revised back to 2002, with new components replacing the old ones. However, prior to 2002, data on old components are still reported in the 2010 and in 2012 editions of the Fraser composite indicator. This is the case for the same topical components, such as Impact of minimum wage component, which was based on a WEF GCI survey question in 2001 and which became Hiring regulations and minimum wage of the WB DB after 2001, or for the Centralized collective wage bargaining component, which prior to 2002 reflected the Share of labour force whose wages set by centralized collective bargaining. It is also the case for discontinuous series, such as Unemployment insurance, which was replaced by another series, Hours Regulations, from 2002 on. In other words, the new components replaced the old components, but the actual reported data series behind the new names reflects old components prior to 2002 and new components after 2002 (see shaded cells in Appendix 6). Despite the Fraser disclaimer, this confounding of data series creates artefacts and may lead to wrong interpretations, notably identifying 2002 as a year of important reforms, which was not the case (see, for example, a discussion between Bernal-Verdugo et al. 2013 and Aleksynska 2015a).¹³

3 How do selected indicators compare?

Given the above-mentioned considerations, do the composite indicators under review tell us the same story about labour markets? There are at least three ways to look at this issue: examine the degree of correlation between these indicators; assess whether they provide similar ordering results, such as country rankings; and gauge their pertinence to the phenomena they are supposed to influence, such as the “health” of the labour markets.

We first examine the correlations between labour market indicators across the composite indicators under review. In Table 5, aggregate indices are presented in bold and selected subcomponents are in italics. Data are displayed for 2009 as this year offers the most comparable data availability. We take the IMD sample as a basis for comparisons, both because it has the smallest sample and because its rankings are sample-dependent.¹⁴

Table 5 Correlations between selected indicators, 2009

From Table 5, both World Bank subindicators—Rigidity Index and Redundancy Cost—have a relatively strong correlation with the WEF and Fraser aggregate indicators that are partly based on them (the minus sign is due to the reverse scale of this indicator: higher scores indicate more rigidity). Likewise, the WEF and Fraser (which also partly uses WEF) aggregate labour market indices show a correlation of 0.61. The similarly phrased WEF survey question on ease of hiring and firing and the IMD survey questions on flexibility of labour market regulation correlate rather well (0.78). Relatively high correlations may be reassuring from the point of view of the correctness of the measurement of similar concepts across individual components. However, they also point to a certain double-counting when these individual components are averaged into aggregate indices: more weights are given to hiring and firing practices as opposed to other aspects of labour regulations. At the same time, some correlations, such as between the WEF Flexibility indicator and the World Bank Redundancy Cost or between the WEF Hiring and Firing component and the World Bank Rigidity Cost, are very moderate. The latter is quite surprising, because WEF Hiring and Firing and the World Bank Rigidity Cost are supposed to measure very similar concepts, though the former measures perceptions of regulations, while the latter measures the strictness of the legal text. In Table 5, we also report correlations between these indicators and the OECD EPL indicator, as one of the most acclaimed indicators in the field, though only measuring one specific institution (and being most comparable, from this viewpoint, to the WB Redundancy Cost indicator). Indeed, the OECD EPL’s correlation is strongest with the WB Rigidity indicator and WB Redundancy Cost; it is, however, quite low with the WEF and IMD individual subcomponents.

Given these correlations, are countries ranked in a similar way across indicators?¹⁵ We use the WEF and Fraser aggregate labour market indices together with the IMD individual question on labour market relations to list the “top 20” countries according to their competitiveness, flexibility, efficiency, or ease of dismissal—as provided by each indicator. We also add the Redundancy Cost calculated by the World Bank for illustrative purposes. As all datasets have different country samples, the analysis is further restricted to a comparable sample, again taking IMD as a reference, because it has the smallest sample size, and because IMD country rankings are sample-dependent. On its basis, we order the Fraser, WEF, and WB data to produce our own ranking of the IMD-sample countries.

Table 6 reflects these rank positions of the IMD “top 20” countries within the four samples. Evidently, these rankings differ dramatically across indicators: the IMD and the WEF give the same rank only to one country (Hong Kong), while the IMD and the World Bank give the same rank only to two countries (Denmark and Singapore). The most consistently ranked country across Fraser, the WEF, and the WB is the USA. Apart from this, each indicator gives a different ranking to the IMD “top 20” countries, and countries like Malaysia or Israel receive particularly different rankings across indicators. Nine of the IMD “top 20” countries are not among “top 20” Fraser countries at all, five are out of the “top 20” WEF ranking, and eight do not enter the “top 20” according to the WB criteria.

Table 6 Countries rankings across indicators. “Top 20”, 2009

Finally, we check the correlations between the selected indicators and total unemployment rates, long-term unemployment rate, and youth unemployment rate in 2009 (Figs. 1, 2, and 3).¹⁶ While correlation does not imply causality, one can observe that, in Fig. 1, stricter regulations are positively correlated with higher unemployment when using the IMD (individual component), WEF LME (aggregate), and WB Rigidity of Employment index but negatively when based on the Fraser LMR (aggregate) and WB Redundancy Cost. In Fig. 3, the Fraser and WB Rigidity indicators have a positive correlation with youth unemployment rate, though not the IMD, WEF, or WB Redundancy Cost. More generally, across the three figures, there is not a single outcome that is affected in a consistent way by all of the indicators.

Fig. 1

Correlations between selected labour market indicators and total unemployment rate, 2009 Note: IMD, Fraser, WEF: higher values indicate more competitiveness, freedom, or efficiency. WB: higher values indicate more strictness and rigidity. Source: Own computations based on selected composite indicators and the ILO KILM

Fig. 2

Correlations between selected labour market indicators and long-term unemployment rate, 2009 Note: IMD, Fraser, WEF: higher values indicate more competitiveness, freedom, or efficiency. WB: higher values indicate more strictness and rigidity. Source: Own computations based on selected composite indicators and the ILO KILM

Fig. 3

Correlations between selected labour market indicators and youth unemployment rate, 2009 Note: IMD, Fraser, WEF: higher values indicate more competitiveness, freedom, or efficiency. WB: higher values indicate more strictness and rigidity. Source: Own computations based on selected composite indicators and the ILO KILM

Given the above, the choice of one particular database and indicator seems to provide a substantially different picture of labour market regulations. Even if all three composite indicators have one strong common feature—they are all partly based on the World Bank EWI—they provide a different outlook of regulations. This is both because they include other different components and use different aggregation schemes and also because they use the World Bank EWI in different and not always appropriate ways, as described above. Overall, this finding calls for a careful use of the different indicators and for further improvement of existing indicators. Otherwise, the “hidden costs and downside risks, including the diversion of data and measurement efforts, […] risk distorting development policy making” (Ravallion 2012).

4 Conclusions

This paper reviewed three influential composite indicators measuring labour market regulations and their evolution across the world. The ultimate goal of our exercise was to identify the key areas for improving current data as well as refining results of empirical research based on them. We illustrated that these indicators, while useful for summarizing a complex reality, suffer from several shortcomings. The main ones include a less than careful reuse of original data produced by others—a pattern that we argue is facilitated by, and will be growing with, the expansion of technology, easy access to data, easy storage, and data sharing. They also include an ad hoc choice of underlying components and their frequent changes, as well as deficiencies in weighting schemes. As a result, different composite indicators measuring seemingly similar institutional environments actually produce a different outlook of regulations. Because policy advice on labour market regulations can be substantially affected by the choice of an indicator, it should be done with great caution. The review undertaken in this paper suggests that none of the considered indicators can be claimed as sufficiently precise or sufficiently sound for basing coherent policy advice.

Despite the outlined problematic areas with the reviewed composite indicators, it is still possible to use them for some specific analysis and needs—provided the usage is based on a clear understanding of what can, and cannot, be done with the data. For example, cross-country comparisons within the same year, and based on disaggregated components, can in many instances provide interesting insights on phenomena measured by these components. In contrast, other types of data use, such as examining changes in rankings over time, or time-series analysis based on reviewed composites indicators, are substantially more problematic.

Our findings do not imply that indicators measuring the extent of regulations, or strength of institutions, should not be produced. Rather, they suggest that such indicators should be based on more balanced conceptual frameworks and robust methodological choices. This paper also showed that there is room for improving existing indicators. Already a decade ago, Bertola et al. (2000) urged that further research to find reliable means of capturing the complexity of the theoretical and empirical issues involved in creating employment protection indicators be undertaken. Perhaps more than ever before, there is a great need for creating transparent, integral, and comprehensive indicators that would provide a more solid basis for policy advice. There is also a need to raise further awareness about the debates underlying the indicators, what methodological changes these debates imply, and how different these changes are from genuine reforms reflected in the data.

Appendices

Appendix 1

Table 7 Global Competitiveness Report labour market indicators and changes over time (2001–2004)

Appendix 2

Table 8 Global Competitiveness Report labour market pillar and changes over time (2004–2008)

Appendix 3

Table 9 Global Competitiveness Report labour market pillar and changes over time (2008–2013)

Appendix 4

Table 10 IMD components and their changes over time

Appendix 5

Table 11 Fraser Economic Freedom of the World: labour market regulation components and methodology

Appendix 6

Table 12 Fraser Economic Freedom of the World: components and methodology. Example of Argentina