The authors present a dataset that contains the positions of 231 political parties across 28 countries on 30 policy issues that were considered salient for the 2014 elections to the European Parliament. The party position estimates were originally used in a voter information tool which compared the policy preferences of citizens to those of political parties. The paper discusses the estimation method in the context of the literature on estimating party positions, outlines the coding methodology, and introduces the value of the dataset for third-party users interested in studying political participation and representation.
Related data set “Political parties’ policy positions for the 2014 elections to the European Parliament” with doi http://doi.org/10.7802/1594 in repository “gesis datorium”.
Voting Advice Applications (vaas) are contemporary technological solutions that aim to compare the positions of voters to the positions of candidates or political parties on a set of relevant policy issues and informing voters about how well their preferences match those of the parties or candidates.
Government-funded bodies originally developed vaas as civic education tools with the intention of enhancing political knowledge and increasing electoral participation. The most popular vaa in the Netherlands, StemWijzer, was first developed in 1989 as an educational tool for high schools. Since the early 2000s, StemWijzer has served as a blueprint for many other vaas, such as Wahl-O-Mat in Germany, as well being the precursor to other types of vaa designs, such as those of the Swiss-based Smartvote. The rise of the internet brought with it an ever-increasing number of users while also fuelling the spread of vaas across Europe. vaas have now become a more prominent and generalized feature of the European political landscape, often attracting millions of users during electoral contests in larger European countries (Garzia & Marschall, 2014).
In a similar way as for national elections, vaas have been fielded during elections for the European Parliament. The decreasing levels of voter turnout in such elections have long alarmed those who hoped that the increase in the European Parliament’s powers under the co-decision procedure could dampen the indifference towards or disaffection with the European Union. Research has often indicated that vaa use during national election campaigns may boost turnout (Marschall & Schultze, 2012; Gemenis & Rosema, 2014; Garzia, Trechsel, & De Angelis, 2017; Germann & Gemenis, 2019), so it comes as no surprise that the 2014 elections to the European Parliament saw a proliferation of voter information tools such as VoteMatch Europe (http://www.votematch.eu/), EUandI (http://euandi.eu/), EUvox (http://www.euvox2014.eu/), and Electio2014 (http://www.votewatch.eu/electio/). Starting with EU Profiler in 2009, most vaas that were implemented at the supranational level (like the aforementioned tools) have allowed citizens to compare their policy preferences to those of political parties not only in their country of residence but in other EU member states as well. The central assumption of such vaas, however, is that the positions of these parties can be estimated with considerable certainty. If they cannot, vaas would provide citizens with misleading information.
In an attempt to contribute to the extensive academic debate on this subject (see Mair, 2001; Volkens, 2007; Laver, 2014), this paper describes the procedures and methods used to estimate party policy positions in the context of the EUvox vaa and presents the resulting database, which can be used by third-party researchers to investigate questions of broader interest in political science.
2. Research Problem
One of the biggest methodological challenges in the design of vaas and voter information tools in general is to find an unbiased and reliable method of estimating the positions of political parties on the policy statements that are used to match voters to parties. For several decades, the primary reference source for party policy preferences was based on the laborious hand-coding of party manifestos using methods and codebooks developed in the 1970s (Budge, Klingemann, Volkens, Bara, & Tanenbaum, 2001). Such hand-coding, however, has been shown to be unreliable against conventional content analysis standards (see Mikhaylov, Laver, & Benoit, 2012; Gemenis, 2013a).
The advent of the internet brought with it a plethora of political documents in machine-readable format. Political scientists can now pre-process such documents and use them to estimate “quantities of interest” (Grimmer & Stewart, 2013). Supervised machine-learning algorithms appear to be particularly promising with regard to coding party manifestos. However, the outcomes of supervised machine learning to classify the content of party manifestos only moderately correspond to those by human coding (Bilbao-Jayo & Almeida, 2018; Wiedeman, 2018). Moreover, the applications are concerned with pre-defined policy categories that indicate the relative emphasis parties place on different issues instead of their expressed position on issues which is of interest in vaas.
In contrast, the supervised computational scaling methods (Grimmer & Stewart, 2013, pp. 291–292) have been shown to lack validity, especially with regard to estimating positions on specific policy statements employed in vaas (Bruinsma & Gemenis, forthcoming; Ruedin & Morales, 2017). Despite their promise, automated text analysis methods have not become a standard tool in political analysis due to the lack of rigorous studies demonstrating their widespread validity (Grimmer & Stewart, 2013). Moreover, the automated text analysis algorithms are not directly accessible or easily understandable by users, which makes the use of such automated methods problematic in terms of the standards set by vaa designers (e.g. the Lausanne Declaration on vaas: Garzia & Marschall, 2014).
For these reasons, the vaas operated by government-funded bodies rely on direct input from the parties or candidates, while the vaas that are designed by political scientists rely on estimates given by other political scientists in so-called ‘expert surveys’ (Gemenis & Van Ham, 2014). The latter, however, have been criticized in terms of ideological bias, as experts have their own political preferences (Curini, 2010), and in terms of reliability, as experts often disagree with one another (Lindstädt, Proksch, & Slapin, 2018).
To address the problem of subjectiveness and unreliability in estimating parties’ positions on a wide variety of policy issues across EU member states, EUvox used an approach best described as an ‘iterative expert survey’ (Gemenis, 2015). The approach consists of an interactive forecasting technique that relies on the anonymous deliberations of a panel of coders and controlled feedback between multiple rounds of estimation, known as the Delphi method (Dalkey & Helmer, 1963). In many respects, this structured aggregation of responses has clear advantages over taking the mean (Steenbergen & Marks, 2007) or modal (Lindstädt, Proksch, & Slapin, 2018) response, or more sophisticated statistical aggregation approaches (Marquardt & Pemstein, 2018). A considerable body of evidence has shown that the Delphi method gives more accurate estimates compared to alternative methods (Rowe & Wright, 1999), and its application in estimating party policy positions has produced valid and reliable results (Gemenis, 2012; Gemenis & Van Ham, 2014). The premise of the Delphi method is that the iterative process induces coders to converge on an estimate by filtering out biased and/or uninformed estimates. Anonymity plays a crucial role, as it guarantees that the consensus reached among the coders is due to the quality of information and not to the coders’ personalities or political preferences. To achieve this, EUvox employed a web-based system that allowed panellists to estimate party positions in an anonymous, distributed, and asynchronous manner (Djouvas & Gemenis, 2019).
For the 2014 elections to the European Parliament, party positions were coded on a set of 30 questions representing policy issues that were considered salient for each country. For each party, the goal was to assign five coders to complete the task over at least two rounds of coding. In a few countries (Austria, Belgium/Flanders, France, Malta, Netherlands, Sweden), time constraints did not allow for a second round, while for a few parties the country teams of the project were unable to recruit five persons per party and the task was therefore carried out by fewer coders. Coding for EUvox took place on the SmartCoding web platform, which was designed for use with the ‘iterative expert survey’ method as shown in Figure 1.
Bulgaria should join the Eurozone
Statements such as the above represent an opinion. Coders had to decide whether or not the relevant party agreed with this opinion. They therefore needed to make a judgement using documented sources, their own expertise, or other contextual information. They had a five-point response scale with a ‘no opinion’ option for their judgement.
Neither agree nor disagree;
Completely disagree; or
Coders were asked to use the (6) ‘No opinion’ position for parties when the party’s position on the issue was not mentioned in, and could not be inferred from, any relevant sources. Coders were instructed that coding results should contain as few as possible ‘no opinion’ positions.
Positions (2) ‘Agree’ and (4) ‘Disagree’ were used when a party took sides on the issue. For example, in the above statement, position (2) should be used if the party agreed that Bulgaria should join the Eurozone, while (4) should be used if the party disagreed and believed that Bulgaria should not join the Eurozone.
Positions (1) ‘Completely agree’ and (5) ‘Completely disagree’ were used when a party strongly emphasized the issue and did not place any restrictions on or have any doubts about the validity of the issue. In the example given, (1) should therefore be used when the party appeared to express a strong and unambiguous opinion that Bulgaria should join the Eurozone, while (5) should be used when that party expressed an equally strong and unambiguous position that Bulgaria should not join the Eurozone.
Coders were also instructed that position (3) ‘Neither agree nor disagree’ was an argued position, unlike (6) ‘No opinion’. Position (3) was to be used either i) when the party only addressed an issue indirectly or vaguely and did not express an opinion clearly in favour or against; or ii) when the party clearly emphasized both sides of the issue (i.e. positive and negative aspects) and made a point of not taking sides; or iii) when a party was divided on an issue (i.e. equally prominent figures within the party took different sides of the argument).
Coders were instructed to reach their decisions on how to code each party on each statement anonymously and not to collaborate. In addition, they were asked to do all they could to be objective and not let their prejudices about certain parties get the better of them. For example, a coder who was sympathetic to pro-Euro party X should not code party Y ‘Disagree’ on whether Bulgaria should join the Eurozone based on a prejudice that party Y were pro-Russia communists who obviously opposed the Euro (even if they say they support it). If party Y’s leaders said that they wanted Bulgaria to join the Eurozone, the party must be coded accordingly.
In addition, coders were instructed to indicate how they arrived at their judgement in each case by entering some information in a box entitled ‘Justification’. Coders were told that they could make use of all possible sources of information on party positions (including manifestos, websites, news reports, and press releases). EUvox did not employ a set ‘hierarchy of documents’ (cf. Garzia, Trechsel, & De Sio, 2017), which meant that the coders were free to decide where to source the most relevant information. When these sources proved to be insufficient or otherwise unavailable, coders were asked to make a personal judgement based on their expertise or other contextual information. They were told that the information provided in the ‘Justification’ box should help convince other coders during the next round of coding that the judgement provided in that case was correct. Moreover, coders were instructed to provide the source of this evidence (e.g. a specific url, page number of party manifesto, etc.) in the box entitled ‘Source’.
Coders were also asked to indicate how confident they felt about their coding of a party for each statement. The options on a drop-down menu on the SmartCoding web platform were:
Not confident at all.
Once coding for the first round was completed, a moderator selected for each country inspected the file with the aggregated results produced by the SmartCoding platform to check whether sufficient consensus had been reached among the panel. For each party/statement combination, SmartCoding had been programmed to export the median estimate as well as Van der Eijk’s A coefficient of agreement (Van der Eijk, 2001). Coefficient A ranges from 1 to -1, where 1 indicates perfect agreement among the coders and -1 indicates polarization, with half of the coders placing the party at (1) ‘Completely agree’ and the other half at (5) ‘Completely disagree’. Moderators were asked to check for instances where A was under 0.7 and to review the individual responses in these party/statement pair. It appeared to be often the case that coders mistake the direction of the response scale. For example, two coders might give exactly the same source and justification, but one will code the party as (1) ‘Completely agree’ and the other as (5) ‘Completely disagree’. This probably means that one of the coders has mistaken the direction of the 1–5 response scale that was given for the coding. Moderators were instructed that, in such cases, it was advisable to enter a comment under the justification given by the coder in question to point out this inconsistency. It was at the moderator’s discretion to enter as many such comments on estimates/justifications given by coders as necessary. The moderators could therefore point out possible mistakes, inconsistencies, biases, and misunderstandings of the meaning of the question or the response scale.
The moderator then initiated a second round of coding, in which coders were able to see the estimates, justifications, and level of confidence given by each of the other anonymous coders, as well as the comments made by the moderator. In the second (and any subsequent) round, coders were asked to review their initial (first round) estimates in the light of the information (estimates, confidence, justifications, and sources) given by all other coders and of the (possible) comments given by the moderator. Also, in the second (and any subsequent) round, SmartCoding allowed coders to comment on estimates given in previous rounds, and to respond to comments left by the moderator. SmartCoding allows this process of moderation and iteration to be repeated over several rounds until sufficient consensus is reached. In EUvox, two and (rarely) three rounds were usually sufficient, although, as previously noted, in a handful of countries time constraints did not allow for coding of parties over more than one round. When coding for the parties in a particular country ended, the moderator was asked to arbitrate over any unresolved cases where agreement in specific party/statement pairs was below 0.7 in terms of Van der Eijk’s (2001) perceptual agreement coefficient, as well as in cases where the majority of coders gave a (3) ‘Neither agree nor disagree’ or a (6) ‘No opinion’ position. To resolve these cases, moderators were instructed to take into account the justifications and levels of confidence given by the coders.
EUvox party positions dataset deposited at gesis – doi: http://doi.org/10.7802/1594
Temporal coverage: 2014
The EUvox party positions dataset, available from the German social science data archive, gesis (doi:10.7802/1594), contains two main files: the party positions file, and the questionnaire file. In the party positions file, each row represents a political party, with the columns giving the name of the party in English and the original language, the number of rounds and coders that were employed in coding the positions of this party, and the positions of the party on each of the 30 policy statements. Out of these 30 statements, 21 were common across all countries (see Appendix). Among the common questions, there were seven for each of three broad dimensions: economic ‘left‐right’, cultural ‘liberal‐conservative’, based closely on the tan/gal scale proposed by the Chapel Hill Expert Survey team (Hooghe, Marks, & Wilson, 2002), and a pro/anti‐EU dimension. A notable exception to the common questions was the wording of the statement about the Euro, which differed between Eurozone and non-Eurozone countries (see Appendix, Statement 1). Among the nine questions that differed among countries, some were common across several, but not all, countries. An example of such a statement was ‘Protecting the environment is more important than fostering economic growth’, which was given in Finland, Ireland, Germany, Italy, Spain, Poland, Hungary, Estonia, and Bulgaria, but not in other countries. Other questions were unique to a single country, such as ‘To combat unemployment, workers must accept signing individual contracts’, which was given only in Greece. The wording of all policy statements that were coded for each party can be found in the questionnaire file.
In total, the dataset contains the positions of 231 parties across 28 countries and on 30 policy issues, making it comparable to other major datasets in the field (e.g. the 2014 edition of the long-established Chapel Hill Expert Survey (http://www.chesdata.eu). Moreover, the use of multiple coders and, in most cases, multiple rounds of coding offers increased confidence in the reliability of the estimates compared to other datasets with a comparable breadth of coverage (cf. Garzia, Trechsel & De Sio, 2017).
5. Concluding Remarks
Third-party users can employ the party position data offered in this dataset as is or scale the individual items into general dimensions of interest using data reduction techniques such as principal component analysis, multidimensional scaling, Mokken scale analysis, or Bayesian item response models (Louwerse & Otjes, 2012; Gemenis, 2013b; Germann, Mendez, Wheatley, & Serdült, 2015; Germann & Mendez, 2016; Linhart & Shikano, 2015; Dalton, 2016; Nyhuis & König, 2018) to study a wide variety of substantive questions in political science.
For instance, vaa data on party positions has been used to assess government coalitions (Linhart & Shikano, 2015), polarization across policy issues (Wall & Williams, 2018), the dimensionality of political space (e.g. Katsanidou & Otjes, 2016; Garry, Matthews, & Wheatley, 2017; Nyhuis & König, 2018), continuity and change in parties’ ideological profiles (e.g. Dalton, 2016), and political representation (e.g. Rose & Borz, 2013), especially in combination with citizen data from election studies (e.g. Lefkofridi & Katsanidou, 2014). In addition, since vaas employ the same questionnaire for parties and citizens, vaa datasets on party positions can be combined with datasets of vaa users (e.g. Wheatley, 2015; Wheatley & Mendez, forthcoming) in order to measure electoral utilities that can be used to test spatial models of voting (Mendez, 2017). Either way, vaa data on party positions offer unique opportunities to study political participation and representation beyond the vaa designers’ original intention of helping citizens make informed choices during elections.
Gemenis K. & Van Ham C. T. (2014). Comparing methods for estimating parties’ positions in Voting Advice Applications. In Garzia D. & Marschall S. (Eds.) Matching Voters with Parties and Candidates: Voting Advice Applications in Comparative Perspective (pp. 33–47). Colchester: ECPR Press.