Makerfield as it was then

We need to know the age distribution of the 2024 Makerfield constituency as it was in 2016. We can find the necessary data at Nomisweb. The query tool allows us to get the 2021-based small area estimates of population. Because the people at Nomisweb are wonderful, and foresee all eventualities, they make it possible to aggregate these small area estimates not just to the 2010-2024 boundaries that were in use at the time, but also to the 2024- boundaries which were not even a twinkle in the eye of the Boundary Commission.

This age distribution is available for men and women separately. Because I never want to download things manually, I’ve been greedy and downloaded information for all of the country, and only latterly filter this to Makerfield.

Here’s what a random selection of rows of that data look like.

Age and gender and referendum vote choice

We know the number of people of different ages and different genders in Makerfield in 2016, but no official source will tell us how these people voted (or indeed whether they voted). We must therefore learn from national data.

Here I’m going to build a model of referendum vote choice based on the post-referendum wave of the British Election Study. The model is a generalized additive model, which allows for a smooth relationship between age and voting behaviour, where that relationship is different for men and women.

With generalized additive models, the best way to understand the model is to plot it. So here are the predicted probabilities of voting Leave and not voting at all, by age and by gender.

There are two problems with these model derived predictions. First, the proportion of people not voting is far too low. Although turnout in the EU referendum was high, it was still less than 75%. The predicted probability of not voting is always less than 25% for all ages and either gender.

Second, this model derived prediction doesn’t know anything about Makerfield, and any predictions we make for the voters in Makerfield won’t match to the results for Makerfield.

In order to make our predictions match know results, we’ll use a logit shift. The logit shift is one of those techniques in political science which is unreasonably effective. Instead of working on the probability scale, we turn those probabilities into “logits”. We then add on a little extra (or subtract a little extra), convert it back into a probability, and see what this does to the result. Here’s my code which will adjust our probabilities so that everything matches the results of the referendum .

We can now get the number of people in 2016 Makerfield, of different ages and genders, who voted for each option, or did not vote at all. Here is another random draw of rows from our data, with those counts now added.

Introducing mortality

Given this information, we can now start introducing information on mortality. Once more, we return to NomisWeb, where we can find mortality statistics by sex and age by using the query tool (“Life events”). Working with this data is slightly awkward: unlike the figures for the age distribution, these figures are not baked into ready aggregates. We must download the data at a low level, and aggregate back up.

The data itself gives the count of deaths by age group, gender and geography. Here’s a random sample of rows, for lots of different constituencies, showing a large number of zero returns.

Unfortunately the data only includes mortality figures for 2017 to 2023. I have therefore had to assume that mortality in Makerfield in 2024 and 2025 was the same as mortality in 2023.

I can aggregate my predictions to the age ranges shown in the mortality data, adding up the numbers of Remain voters, the numbers of Leave voters, and so on. I can then proportionally allocate the deaths in each age-by-gender group, and working out how many voters of each type have died.

How many then?

Overall, I estimate that 6167 Leave voters might have died since 2016, compared to around 2003 Remain voters. That’s a mortality rate amongst Leave voters of 17% amongst Leave voters, compared to a rate of 11% amongst the (on average younger) group of Remain voters.

This means that amongst the still living in Makerfield, the Leave:Remain split moves from 66:34 to 64:36.

These are estimates. To the extent that they are wrong, they are likely to under-estimate the number of Leave voters who died, and over-estimate the number of Remain voters who died. Bad health was a significant individual- and aggregate-level predictor of voting for Leave, and that would affect mortality estimates. However, more sophisticated modelling, or including mortality for 2026 to date is unlikely to change significantly the qualitative conclusion.

Ultimately, these actuarial exercises don’t affect the main reason why continuing to cite these numbers feels odd, and that’s that some people have changed their mind since then.

Problems with calculating disproportionality on the basis of the list share

There are at least three reasons why calculating disproportionality on the basis of the list vote alone – or what I’ll call standard practice – is bad. These reasons apply to all commonly used indices of disproportionality.

First, standard practice does not take account of the fact that the list share of the vote may not be principal determinant of seat shares. In particular, standard practice makes little sense in parallel systems where a majority of seats are awarded based on the nominal tier, or in compensatory systems where the nominal tier makes up more than three-quarters of the total seats (Bochsler 2023, 102557). Japan is an example of a parallel system where the nominal tier is more important (289 of 465 seats awarded in the nominal tier); Korea is an example of a notionally compensatory system where the proportional tier makes up just 15% of the legislature. These systems are not designed to make seat shares fully match list shares, and so presenting information on the gap between list shares and seat shares seems to miss the point. For these systems we might consider calculating disproportionality on the basis of the nominal tier share of the vote, but this would create a demarcation problem (when is the nominal tier share so large that the relevant vote share is the nominal tier share rather than the list tier share?)

Second, standard practice does not take account of the fact that the set of parties competing in the nominal and list tiers may vary for reasons good or bad. Take as an example the Non-Partisan Solidarity Union in Taiwan, which in the 2012 election ran only in single member districts, winning two seats. Because its list vote share was technically zero, the gap between its seat and vote share contributes significantly to overall disproportionality as measured by the Gallagher or Loosemore-Hanby indices. Worse still, the fact that the party won seats on zero percent of the vote creates a division-by-zero error in the formula used to calculate the Sainte-Laguë index, meaning that disproportionality is undefined!

Nominal and list-tier parties may differ for bad reasons. Decoy lists – where a party runs constituency candidates in a separate list to frustrate compensatory mechanisms in the list tier – are commonly held to be a bad thing (Bochsler 2012). These lists are bad because they frustrate the intent of the electoral law, but they are also bad because they make it harder to calculate a meaningful measure of disproportionality if the decoy list technically has negligible list vote share.

Third, standard practice ignores the fact that some voters “lend” or “rent out” their vote precisely because there is another approved party which is likely already to benefit from the electoral system (Meffert and Gschwend 2010, 2011; Gschwend et al. 2016). CDU/CSU voters in Germany often lend their vote to the FDP so that it can pass the electoral threshold (Roberts 1988); SNP voters in Scotland similarly lend their vote to the Green party so that they can broaden the set of pro-independence parties. Should that strategy fail – as it did in the 2013 German elections – then it would seem strange to regard those lent votes as major contributors to disproportionality: the people who cast those lent votes already got a good return on their nominal tier vote.

Sadly it is not possible to remedy these problems easily. We can’t calculate separate indices of disproportionality for the different tiers and aggregate these together: the disproportionality in a nominal tier will advantage one set of parties, whilst the disproportionality in a compensatory tier will advantage a different set of parties, and because disproportionality is not signed, these two disproportionalities will add to something greater than any plausible measure of overall disproportionality.

We also can’t average the vector of vote shares across the two tiers. Averaging shares across the two tiers makes sense, but it fails to deal with the problem of parties that don’t run in both tiers, and parties that create decoy lists. It also fails to address “rental” mechanisms which mean that the vote share in one tier may be “low” precisely because support is strategically being directed to the other tier.

An extension to the Sainte-Laguë index

I suggest extending an existing measure of disproportionality (the Sainte-Laguë index) to deal with the problem of multiple tiers. Because the Sainte-Laguë index can be represented as though it aggregated over voters rather than parties, it can be extended to analyse disproportionalities across multiple voter types, where a voter type is defined by the set of parties voted for across multiple tiers. This party/party-supporter duality allows me to combine the two tiers and present a measure of disproportionality that summarizes inequality amongst voters and disproportionality between parties.

The Sainte-Laguë index has two inputs: a vector of vote shares , and a vector of seat shares . The index can be written using two formula which look different but give identical results. The index can be written using squared differences between seat and vote shares:

where each party is indexed by , and the summation runs over all parties . This way of writing the index makes the index resemble the chi-squared index used to measure association.

Alternately, the index can be written as a weighted average of advantage ratios. The advantage ratio for a party, , is equal to its seat share divided by its vote share: . Because shares always sum to one, the weighted mean advantage ratio across all parties must equal one. Departures from an advantage ratio of one therefore indicate disproportionality. The index is:

Writing the Sainte-Laguë index in the form given by Equation 2 allows us to think about what an extended index might look like if voters, not parties, were the unit of account (Van Puyenbroeck 2008). Rather than speak of individual voters, I am going to aggregate and deal with voter types.

A voter type is defined by the set of parties they vote for across multiple tiers. The voter who votes for the CDU in the constituency vote and for the FDP in the list vote is one type. So too is the voter who votes for the SPD in the constituency vote and the AfD in the list vote. The voter who votes for the CDU in both the constituency vote and in the region vote is a particular type of voter: a straight-ticket voter.

Voter types are defined by unordered sets of parties, and don’t take account of which vote was cast for which party. This means that the voter who votes for the CDU in the constituency vote and the FDP in the list vote is of the same type as the voter who votes for the FDP in the constituency vote and the CDU in the list vote.

If there are parties competing, then the number of voter types is equal to the square of the number of parties. This is the number of logically possible voter types, but not all of these types are politically plausible: there are some voters who voted for the SPD in the constituency vote but AfD in the list vote, but this type is rare.

Having introduced the language of voter types, I’ll now make a further conceptual shift, and talk about advantage ratios for voter types rather than parties. I’m going to stipulate that the advantage ratio for a voter of type is equal to the average of advantage ratios of the parties that they supported, and that the advantage ratio for each party is now calculated based on the seats they won, divided by the proportion of voters who voted for them on either vote.

Formally,

where is a party that makes up voter type , and where is the proportion of people that voted for party in either ballot. The proportion is always greater than a party’s list share or their nominal tier share, and the sum of all values of ranges between 100% (all voters are sincere and do not split their votes) to 200% (all voters split their ballot).

It should be clear from these definitions that we cannot work out how many people belong to each different voter type from the results data alone. If a party wins 10% of the constituency vote and 20% of the regional vote, then it’s likely that the overwhelming share of the constituency voters also gave their regional vote to that party, but even that is not guaranteed, and we have no way of knowing where the other ten percentage points came from.

I therefore work with survey data to develop this extended measure of disproportionality. My example is drawn from the 2021 Scottish Election Study. The figures in this section are based on the post-election survey wave, except that the survey weights for respondents have been raked so that the sample proportions of nominal and list votes for each party exactly match the reported results.

Table 1 gives information on the break-down of types in the 2021 election. The largest single type of voter (38.16% of all voters) is the voter who cast their vote for the SNP in both contests. There are large entries on the diagonals, which represent straight-ticket voters, but some of the off-diagonal entries are clearly relevant, like the 7% of voters who split their votes between the SNP and the Greens.

We can use this information to calculate advantage ratios for different voter types:

• for the voter type {SNP}: the number of seats won by the SNP in the 2021 election was ; the proportion of people who voted for the SNP on either ballot was 0.499 (this being the sum of all entries in the yellow column); the party’s advantage ratio is therefore 0.994.

• for the voter type {SNP, Green}: the advantage ratio for the SNP is as previously calculated. The advantage ratio for the Green party is their seat share (6.2%) divided by the proportion who voted for them on either ballot, or 0.0879153 (this being the sum of all entries in the L-shaped group of cells with a green border); the party’s advantage ratio is therefore 0.705; the average of these two advantage ratios is 0.85

Because sum to more than 100%, the average advantage ratio is less than one. That means we can’t apply the Sainte-Laguë formula above in Equation 2, which relies on the accounting identity that the population weighted advantage ratio is one. We must normalize by dividing by the share-weighted average of advantage ratios, or 0.827. Call this adjusted figure . We can then use in Equation 2, and replace the vote share with the proportion of voters in each type.

Formally,

where , , and is as described in Equation 3.

Given Equation 4, then disproportionality in the 2021 Scottish Parliament election was
5.3%, rather than the value of 0.069 that one would get by following standard practice and calculating the Sainte-Laguë index using the list share alone. Extending the Sainte-Laguë index in this allows us to recognize that many people voted for the SNP in the nominal tier and lent their votes to other parties in the list tier, and that therefore the SNP is not as over-represented as one might think – except that now we must say, “voter types involving the SNP” are not as advantaged as the list share alone would suggest.

Comparative evidence

We can repeat this exercise for other countries which use mixed systems. Here I draw on the Comparative Study of Electoral Systems Integrated file and two elections from module 6 of the CSES. I am able to use all mixed elections save one: the 2005 survey of Albanian voters does not include anyone who voted for some seat-winning parties. As such, I am unable to construct the joint distribution for all relevant parties for this election. I return to this issue in the conclusion.

Figure 1 shows the values for all the mixed member elections in the CSES. In general, disproportionality under the extended Sainte-Laguë index is around two-thirds of disproportionality under the basic Sainte-Laguë index. Most elections have lower values under the extended index, but there are some exceptions. I discuss the Korean case below in more detail, as it is the most obvious exception to the pattern. Other elections where the extended Sainte-Laguë index exceeds the basic index result from complexities of the nominal tier: in Hungarian elections, the higher figure under the extended Sainte-Laguë arises because of disproportionalities in the two-round nominal tier, where first-round shares can be a poor guide to seat outcomes.

The two measures are visibly correlated (), and so elections which have high disproportionality based on the list votes alone are also likely to have high disproportionality based on this extended index – but the overall levels are lower. This in turn has implications for the level of disproportionality achieved in mixed systems compared to other simpler systems.

Whether the mixed system is mixed member proportional or mixed member parallel does not seem to affect the relationship between basic Sainte-Laguë and the extended version. Mixed member parallel systems generally have higher disproportionality, and so they are further to the right side of the graph. However, mixed member proportional systems can also have high disproportionality, as in Germany in 2013, when the FDP, AfD, and Piraten all fell foul of the five percent threshold. That disproportionality concerned the proportional tier alone, and this election result looks disproportional on either the basic Sainte-Laguë index or the extended version.

We can also compare the extended version of the Sainte-Laguë index that I have argued for to the simpler version which involves averaging over vote shares in the nominal and list tiers before calculating disproportionality. This is shown in Figure 2. These two measures are more strongly correlated (), but there are differences in the levels. Levels of disproportionality under the extended Sainte-Laguë index are generally lower, and are on average around 87.3 percent of the values of the index when averaging over nominal and list tier votes.

Whether we compare the extended Sainte-Laguë to the basic Sainte-Laguë index or to a version which averages over list and nominal shares, there are pairs of countries which have very different values on the two indices. The most obvious contrast is between New Zealand in 2023 and South Korea in 2012. These two elections were equally disproportional if we are to judge by the standard Sainte-Laguë index: the values are almost identical (0.066 for Korea, 064 for New Zealand). However, the extended index suggests that the South Korean result was much more disproportional. Why is this?

The answer lies in a combination of the gap between list and nominal shares, the dispersion of advantage ratios; and the size of split-ticketing groups which choose “poorly”. First, note that in the 2012 Korean election, the gap between the list tier share and the nominal tier share was small for the two largest parties – of the order of a percentage point. The extended Sainte-Lagüe index can make an outcome that seems disproportional on the basis of the list tier look better by incorporating information from the nominal tier. Here, there is no gap to make good. Second, note that the Korean system is heavily majoritarian – the nominal tier contributes 85% of seats. This means that the votes-seats curve is extremely steep at its inflection point. The contrast is with the New Zealand system which has a discontinuity in the votes-seats curve at the threshold of five percent, but is otherwise fairly smooth. Whether they lend their votes to the Greens, the ACT, or NZ First, voters are similarly treated, even though these parties differ in vote share by five to six percentage points. Third, although many voters in the Korean election were straight-ticket voters, a sizeable proportion of Saenuri and Democratic United Party voters experimented with other parties. Because the votes-seats curve is so steep, any defection from one of the two big parties means that these voters advantage ratio couples one fairly good advantage ratio to one fairly disastrous advantage ratio. In other words, the extended Sainte-Laguë index shows that in a more majoritarian mixed system, trying to split your vote risks wasting at least one vote. The difference in values is therefore indicative of the difference in the two systems which, although both formally of the same mixed member proportional type, differ substantially in their composition.

Conclusions

In this note I’ve introduced an extension of the Sainte-Laguë index which allows the measurement of disproportionality across multiple votes given information on the joint distribution of those votes.

This index has several desirable features. First, it allows us to incorporate information from both votes cast by voters, rather than privileging one vote and ignoring the other. Second, the index collapses to the normal Sainte-Laguë index if all voters are straight-ticket voters. Third, the index deals with decoy lists by exploiting the empirical overlap between nominal slates and upper-tier lists.

The major disadvantage of the index is that it requires information on the joint distribution of votes across tiers. In this note I have used survey information to construct that joint distribution, but I have also noted that in one case (Albania 2005) the survey data was not detailed enough to construct the marginal distributions for at least one seat-winning party. I would suggest that estimating the joint distribution of votes is a basic descriptive task for anyone involved in studying elections in countries which use mixed systems, and so this information ought to be available. If it is not available then it would in principle be possible to estimate the joint distribution using ecological inference, particularly where electoral results are available at a very granular level.

This extension to the Sainte-Laguë index is not the only possible way of extending the index. In particular, I have made some design choices that seemed to me appropriate, but that are not the only valid choices. I have defined voter types as the unordered set of parties supported by each voter, rather than the set ordered by tier. In some senses, the voter who votes for the CDU/CSU in the nominal tier and lends their vote gets the same representation as the voter who votes the other way around, but it seems odd to say that they are equally advantaged. Paying attention to ordering raises the question of whether we should attach the same weight to advantage ratios for “list-tier” parties and “nominal tier” parties. Our present standard for calculating disproportionality attaches zero weight to the constituency tier, and I think that is wrong, but it does not follow from this that the best weighting is an equal weighting. I have not gone beyond equal weighting because unequal weighting seems to move us beyond disproportionality and into an assessment of how well election outcomes match voters’ cardinal utility over parties, and that is definitely a much bigger topic than can be handled here (Monroe 1995; Blais et al. 2022).

Are the benefits of PR overstated?

David writes:

“claims of PR’s downstream benefits (in terms of higher turnout, higher trust, or lower inequality) are often highly overstated, and often relate more to contingent party system dynamics or to the historical circumstances of its initial introduction to a given country than to inherent properties of proportionality itself”.

I don’t wish to defend all claims made in favour of PR. No doubt campaigning organisations present statistics on turnout in PR countries without any rigorous causal identification strategy. That’s a shame, because there is good evidence that PR boosts turnout and lowers inequality, evidence which by design rules out “contingent party system dynamics” or “historical circumstances”.

In France and Poland, researchers have used regression discontinuity designs to show that local elections held under proportional rules have higher turnout than local elections held under plurality rules. We know that this higher turnout is a consequence of the electoral rule, because the electoral rule is stipulated by a population threshold, and because local councils which just fall short of the population threshold don’t have different characteristics or histories to local councils which narrowly exceed the population threshold. PR boosts turnout by either 1.5 percentage points (France) or 4 percentage points (Poland).

This is high quality evidence. The evidence base for the positive effect of PR on turnout is, in my judgement, stronger than the evidence base for the negative effect of voter ID requirements on turnout. You can say that effects of one or four percentage points aren’t large: they are certainly small compared to the effects of David’s preferred solution of compulsory voting. But small does not mean negligible, and meaningful improvements in turnout can be achieved by many marginal gains.

Similarly, there are multiple studies which identify the effect of the adoption of PR upon inequality. Terminally online Britons may be used to the synthetic control method, since this is the method most commonly used to estimate the negative impact of Brexit on the UK economy. That same method has been used to estimate that New Zealand’s adoption of PR meaningfully increased the amount of redistribution. Other studies have used historical transitions to PR, finding that Norwegian councils which were required to use PR redistributed more and that Swiss cantons which switched to PR started spending more on public health programmes [reducing child mortality](https://docs.iza.org/dp12729.pdf.

Does this proposal match what’s happening in French politics?

David writes:

“The French system… accommodates multi-partyism while letting electorates make clear and decisive choices. It allows smaller parties to use their bargaining power to secure parliamentary representation, while working against extremist forces (who are both likely to struggle to form alliances, and against whom voters for other parties can unite in the second round)”

The two-round system is not unique to France, but France has held more democratic elections under the two-round system than any other country. According to Bormann and Golder, between 1919 and 2020 France held seventeen democratic legislative elections under the two-round system. No other country using that system held as many, and many of the remaining countries are either very small (Comoros: six elections 1992-2020; Kiribati: 12 elections 1982-2020) or have since ceased to be democracies (Mali: seven elections 1992-2020). If French politics has a strongly “decisionist” character, that would be prima facie evidence that the two-round system is conducive to “decisionism”.

Unfortunately for David’s argument, contemporary French politics doesn’t demonstrate much evidence of “decisionism”, and the description given above would cause even a Dr. Pangloss to shrink back. The most recent legislative election created a tripolar legislature where a governing “centrist” block is opposed by groupings on the far left and far right. It yielded the shortest-lived French government ever – quite an achievement for those of us required, willing or not, to study the record of the Fourth Republic – and the current Prime Minister (the same guy as the previous Prime Minister) lurches from no-confidence motion to no-confidence motion

One might be generous, and describe the current period of ongoing crisis as an anomaly resulting from Macron’s decision to call a snap election in 2024. But this generosity benefits the giver more than the receiver: if we can say that some negative effects only appeared under the two round system due to the President, then we can also say that any positive effects under a two round system might also be due to the presidency. Legislative elections held shortly after presidential elections benefit from presidential coat-tails or a honeymoon effect. French legislative elections are almost always held shortly after presidential elections. It seems at least possible that the presidential election shapes the legislative election and makes any government formation process much simpler.

In summary: the most obvious example of politics under a two-round system doesn’t display lots of decisionism, and the decisionism it does display might be a result of a directly elected president rather than the electoral system.

More generally, I have a visceral negative aesthetic reaction to the two-round system and the fragmentation it creates relative to either first-past-the-post or the alternative vote. I have spent enough time creating placeholder categories for “divers gauche” and “divers droit”: I do not need any more of it.

This block can fit so many parties

One starting point for the proposal is the claim that British electoral politics is becoming “block politics”, with greater attachment to “left” or “right” than the parties which make up those blocks. Nothing is new under the sun, and block politics has always been with us. What does seem new is the absence of a clearly dominant party within each block. It’s this feature which threatens to cause chaos in the translation of votes to seats.

Although I dislike first-past-the-post, I think it’s wrong to hold the current high effective number of vote-winning parties as an argument against it. We might be in an out-of-equilibrium position because no party is dominant within its block. But out-of-equilibrium positions can return to equilibrium if the system is allowed to continue in operation. If we have an election with a very high effective number of parties, it’s very hard for the distribution of seats within each bloc to be proportional: it would require a very particular electoral geography. Each block would likely find its dominant party through the crushing operation of the plurality rule. I’d like electoral reform to be based on the general statement that SMDP treats some voters much better than others.

The converse of this is that it’s probably not a good idea to pick an electoral system based on a pattern of vote shares found in contemporary opinion polls three years out from the most likely date of the next general election. The current distribution of vote intention is a product of the electoral system; if we were to have a different electoral system, we’d have a difference distribution of vote intention, and possibly a different set of relevant parties. If people were like sticks of Blackpool rock marked “left block” or “right block”, then a two-round system would be a good choice. But most of what people think is structured by idiosyncratic and instability, rather than ideology.

Conclusion

I don’t know what electoral system is best for the UK. If I could decide unilaterally, I’d choose closed list proportional representation with district magnitude of between three and five. I’d accept most other systems of proportional representation, up to and including mixed systems which are sometimes not so proportional. But the two round system is one of the few systems I think is worse than first-past-the-post. If anyone wants to come out in favour of multimember plurality, you could troll me worse than David has, but the game might not be worth the candle.

Building blocks

Output areas

The modelling I do is based on output areas.

Output areas are the smallest areas for which Scotland’s Census produces statistics.

There are over 46,000 output areas, with an average population of around 120 people.

Output areas don’t have names, only codes.

An example output area is S00145472, which is plotted in Figure 1.

This output area is a residential area in Gilmerton, in the south of Edinburgh. The houses there are recently built. At the time of the 2022 Census, the population was 153 individuals, of which 122 were of voting age.

Output areas are characterised by two things: their boundaries, and their population weighted centroid.

The boundary of an output area is not hard to understand. It shows where and how far the area extends. There are complications to do with shorelines, and how detailed these boundary lines can be, but they’re not relevant here.

The centroid of any flat shape is the point upon which the shape would balance perfectly, if the weight of the shape were the same at all points.

The population weighted centroid is the point at which the shape would balance perfectly if we located each person on the shape and give them equal weight.

The population weighted centroid for S00145472 is shown in the plot with a star. This is the best information we have about where most people in that output area are. There is no lower level source of official information about where people in this area live.

For this reason, I’ll be mapping output areas to existing and proposed parliamentary constituencies on the basis of their population weighted centroids. Some output areas might technically intersect constituency boundaries, but

• any intersections might be the result of coarsely drawn boundaries;
• the proportion of people living in those intersections is likely to be negligible; and
• there is no better official source which could resolve this issue.

Post-stratification frame

The reason I described output areas is because information on these output areas allows me to create a post-stratification frame.

You can think of a post-stratification frame as a spreadsheet with a row for each combination of person characteristics, which records how many people with that set of characteristics live in each area.

For example: a post-stratification frame might record how many (i) men are (ii) aged between 60 and 64, are (iii) from an ethnic minority, (iv) have a university degree, are (v) religious, (vi) don’t identify primarily as Scottish, and (vii) live in rented accommodation.

Obviously, the smaller the areas and the more numerous and variegated the characteristics, the smaller these counts will be.

Constructing a post-stratification frame is an involved process. Although the Census tells us how many people in a given area are religious, and how many people in a given area live in rented accommodation, it doesn’t tell us how many people are religious renters. To use terms from statistics: the Census (generally) tells us about marginal distributions, but not the joint distribution.

To create a joint distribution, I use microdata from the 2011 census. This gives information on a five percent sample of 2011 census returns, or just over a quarter of a million individuals. With such a large volume of individual returns, we can work out the association between being religious and being a renter.

I use this data to create initial weights for each combination of characteristics. I then repeatedly multiply or divide these weights so that the sum of weights for each characteristic, taken one at a time, matches the marginal distribution reported in the Census. For example: I might have four proportions:

• religious renters;
• religious owners;
• irreligious renters;
• irreligious owners;

and I might multiply or divide the weights in first two categories to match the Census reported proportion of religious people in the relevant area. I might also multiply or divide the weights in the first and third categories to match the Census reported proportion of renters in the relevant area. Through repeated processes of multiplication and division – in statistical terms, “iterative proportional fitting” or “raking” – I can start with an arbitrary joint distribution and get something that matches what I know about each particular area.

Additionally, where the Census does report associations – as they do when they release tables like “Religion by sex by age” or “Ethnic group by age”, I can ensure that the combination of those characteristics is matched almost exactly at the local level.

In describing post-stratification frames, I have spoken about areas generically. For this report, I’ve constructed a post-stratification frame defined on intersections between old and new constituencies. There are 114 such intersections. This means that the post-stratification frame is defined by:

• area [114 intersections of old and new constituencies]
• age [14 distinct categories]
• sex [2 distinct categories]
• membership of an ethnic minority [2 distinct categories]
• university degree [2 distinct categories]
• religiosity [2 distinct categories]
• national identity [2 distinct categories]
• housing tenure [2 distinct categories]

This means that the post-stratification frame has just over 80,000 non-zero rows in it. After removing combinations with absolutely no people, the average (median) count in each cell / unique combination of attributes is fourteen.

This post-stratification frame covers the voting age population rather than the voting eligible population. There are individuals in the voting age population who are not eligible to vote because they are imprisoned for sentences longer than twelve months or because they are not a citizen of a qualifying country. The difference between the voting age population and the voting eligible population is much smaller in Scotland than in England because of less restrictive rules on the franchise.

In my opinion, this is the most granular post-stratification frame that can reasonably be constructed for this purpose. Whilst it would be desirable to model voting behaviour at a lower level, the SES only records information on respondent’s 2022 parliamentary constituency. Although it records information on postcode sector, postcode sectors can correspond to multiple output areas and can span (current and proposed) constituencies.

The degree of change between old and new constituencies

Because we need to count how many people live in the intersections between old and new constituencies, we can calculate how much each new constituency has changed relative to its closest predecessor.

I define an index of change as one minus the proportion of the new constituency’s population which lived in the old constituency with the greatest population overlap, divided by the new constituency’s population.

I illustrate this with Edinburgh Southern. Figure 2 shows the proposed boundary for Edinburgh Southern and the boundaries of its four predecessor constituencies.

The figure shows thart the new Edinburgh Southern has shifted southwards, taking in the eastmost part of Edinburgh Pentlands, and the southern fringe of Edinburgh Eastern, which had previously snaked round to include the Gilmeron area. The new Edinburgh Southern includes a tiny portion of Edinburgh central, although this may be due to inaccuracies in the map data.

Table 1 shows the count of individuals coming from each of the four predecessor constituencies. The largest single contributor is the old Edinburgh Southern constituency, which contributes almost half of the new constituency population. The index of change is therefore .

We can calculate this for all new constituencies. Table 2 shows the value of this index for the ten constituencies which have seen the greatest change by this metric. Although it is natural to focus on areas which have seen the most change, 46 constituencies have seen negligible change (index values less than one percent).

Modelling

I estimate two models: one model of list vote, and one model of constituency vote.

In the model of list votes, the outcome variable is a categorical variable with eight possible values (in alphabetical order: Alba, Conservative, Did not vote, Green, Labour, Liberal Democrat, SNP, Other).

In the model of constituency vote, the outcome variable is the same, except that there are seven possible values because Alba did not stand constituency candidates.

I model these options because these were the only parties that won more than one percent of the vote, and because these are the only parties chosen by more than a dozen respondents in the Scottish Election Study.

I model these outcomes using different predictor variables. I include all of the demographic variables which feature in the post-stratification frame. These variables are modelled as fixed effects, except that age is modelled as though it were a continuous variable using a spline. I also include constituency random intercepts and constituency level predictors. The constituency level predictors for the model of list votes includes the shares of the voting age population who voted for the Conservative, Labour, Liberal Democrats and Greens. The constituency level predictors for the model of constituency vote also includes dummy variables which record whether a Green candidate or any other candidates stood for election in that constituency.

The statistical model I use is a multinomial logit model estimated using Bayesian methods using the brms package.

Although the primary purpose of the model is to generate predictions, it is helpful to understand the predictions implied by the model when we change respondent characteristics. For example: we might “give” a respondent a degree and ask how that changes the predicted probability of voting for the SNP.

Figure 3 shows the changes in the predicted probabilities for discrete changes in respondent characteristics. People with degrees are much more likely to vote for the Greens. Ethnic minorities are less likely to vote for Labour or the Conservatives. Scottish identifiers are much more likely to vote for the SNP. Men are more likely to vote for the Conservatives, and less likely to vote for the SNP. Respondents who are religious are more likely to vote for the Conservatives, but renters are much less so.

Results

The output of this procedure is a set of notional results for (i) the constituency vote and (ii) the list vote for the seven largest parties. Table 3 shows the result for Renfrewshire North and Cardonald.

We see a slight discrepancy in the numbers who did not vote, implying that some individuals only cast a valid vote in the constituency contest. That’s possible, and the discrepancy is just within the range of invalid votes.

The main point of comparison, however, must be the overall seat tallies (i) on the actual results, and (ii) with these notional results. These are shown in Table 4. Labour “lose” one seat, failing to win the redrawn Edinburgh Southern. The Greens win a list seat in the South of Scotland. Further detail is given in Table 5, which breaks down seat allocations by region.

Comparison with alternative estimates

I have based these notional estimates on a model of voter behaviour estimated on individual responses to a survey taken shortly after the 2021 Scottish Parliament election. This is not the only way to create notional constituency results, and indeed it has not been the dominant way. Most notional results have been put together using information from local elections, since results from local elections are compiled at a more granular level than are results at parliament level.

For the new Scottish Parliament boundaries, Ballot Box Scotland has produced notional results which are based on polling district level results collected directly from local councils.

Creating notional results based on local election results makes a great deal of sense. Local elections often show similar patterns of party support to national elections. Working with local election data also has the advantage that analysts are working with counts of actual behaviour rather than counts of reported behaviour. In the specific case of local election results collated at the polling district, this data is very granular.

Having said that, I think there are several reasons why working from individual level data is preferable.

First, local elections are held under a different electoral system. It seems plausible that the distribution of party support might be similar when comparing the distribution of first preferences under the single transferable vote and list votes. However, this is a big and unverifiable assumption. It’s also much less plausible that the distribution of party support is similar when comparing the distribution of first preferences under the single transferable vote and constituency votes.

Second, many local elections didn’t feature Alba, which means that they’re uninformative about the distribution of Alba support. Modelling Alba support correctly isn’t consequential for Alba’s seat tally, which remains at zero under any plausible redistricting, but is important insofar as Alba may draw votes away from other parties.

Third, local elections do feature relevant independent candidates, which can’t be said of the Scottish Parliament elections. What is worse, these independent candidates do not draw evenly from other parties, but are disproportionately likely to appear and win votes in areas which are more Conservative, as judged by the results of Scottish Parliament elections. This means that in some cases areas will appear as though they were void of Conservative support, when actually that support is masked by strong support for independents.

Fourth, local elections were held a year after the Scottish Parliament election. Whilst geographic patterns of party support are unlikely to have changed within the course of a year, it seems preferable to use 2021 data to work out what would have happened if the 2021 election had been held on 2026 boundaries.

These are reasons why in principle we should prefer notional results based on contemporaneous individual data from the election rather than notional results based on data from separate elections held a year later under a different electoral system. In practice, sets of notional results are extremely similar.

Figure 4 plots counts from this report against notional counts from Ballot Box Scotland for the five largest parties. The dashed line shows the results of the best fitting linear regression line, the equation for which is reported at the top of each panel. There is, to two significant figures, a one to one correspondence between the two sets of notionals. The high association is artificial in the sense that the association is based on all constituencies, including constituencies with minimal change. However, when we remove these constituencies the correlation across parties’ list votes ranges between 0.939 (Greens) and 0.971 (Conservatives), and the mean absolute difference in list votes ranges between 261 votes (Liberal Democrats) and 542 (Conservatives).

High associations can be consistent with differences in seat tallies if a small number of seats are decided by a handful of votes, but there is only one constituency where the two sets of notional results disagree regarding the constituency winner. On the basis of my notional results, the SNP would have won Edinburgh Southern rather than Labour. This is because, as shown in Figure 2 and Table 1, the new Edinburgh Southern seat draws upon Edinburgh Eastern. In Edinburgh Eastern a majority of voters (52%) cast their constituency vote for the SNP’s candidate, Ash Regan. Whilst both these notional estimates and the nationals from Ballot Box Scotland show that the notional constituency race in Edinburgh Southern would have been closer as a result of the influx of voters from Edinburgh Eastern, in my model this influx is sufficient to tip the seat.

Although the constituency winners are almost everywhere the same, there is one further difference in the list seats. Ballot Box Scotland have the Greens winning two list seats in Glasgow, whereas I have them at just one list seat. This is not driven by differences in the estimates of the Green vote: my estimates have the Greens winning 94 more votes than in the Ballot Box Scotland estimates. Rather, the difference in seats is caused by the votes won by Labour. In my notional results, Labour wins more votes, and crucially wins enough votes that its vote share is twice that of the Conservatives or the Greens. This means that when seats are allocated proportionally, Labour gets two bites at the cake for every one bite taken by either the Conservatives or Labour. Although systems of proportional representation reduce the incidence of sharp discontinuities, they do not remove them – and if Labour, across all of Glasgow, had won 368 fewer votes the Greens would have won an extra seat.

Initial code

Here’s some set up that’s common to all simulations. I set a seed, draw 10,000 voters, and say that I want one hundred simulations for each batch, or each size of party system. One hundred simulations is not a lot, but it helps illustrate the point without making this document too slow to render.

library(tictoc)
set.seed(236435)
n_voters <- 1e4
voters <- rnorm(n_voters)
n_sims <- 100

Here’s my first implementation.

tic()
out <- sapply(1:6, function(n_parties) {
    res <- sapply(1:n_sims, function(i) {
        parties <- rnorm(n_parties, mean = 0, sd = 1)
        d <- outer(voters, parties, `-`)
        d <- abs(d)
        min_d <- apply(d, 1, min)
        return(mean(min_d))
### Create all the distances
    })
    return(res)
})
toc()

8.293 sec elapsed

In this code, the outer loop works over the number of parties in the system. The inner loop works over the simulations. Within each iteration in the inner loop, I simulate positions for n_parties parties. I then calculate all the pairwise distances between the vector of parties and the vector of voters using the outer function. I convert these differences into distances by using the absolute value function. This gives me a matrix with n_voters rows and n_parties columns. In order to get the “nearest” party, I apply the min function over the rows of this matrix.

As you can see from the calls to tic() and toc(), I’ve timed the execution time using the tictoc library. I remember being disappointed by how slow this was. What if I wanted 1,000 simulations for each size of party system? Sure, I could parallelize this, but since it involves tailoring code to different machines with different numbers of cores, parallelization is almost always my last resort.

Profiling

In an effort to improve the code, I used the built-in R profiler. Here’s the code and the output I got.

Rprof()
out <- sapply(1:6, function(n_parties) {
    res <- sapply(1:n_sims, function(i) {
        parties <- rnorm(n_parties, mean = 0, sd = 1)
        d <- outer(voters, parties, `-`)
        d <- abs(d)
        min_d <- apply(d, 1, min)
        return(mean(min_d))
### Create all the distances
    })
    return(res)
})
Rprof(NULL)
summaryRprof()$by.self

                self.time self.pct total.time total.pct
"apply"              7.68    72.05      10.52     98.69
"FUN"                2.12    19.89      10.66    100.00
"unlist"             0.36     3.38       0.36      3.38
"lengths"            0.22     2.06       0.22      2.06
"aperm.default"      0.12     1.13       0.12      1.13
"outer"              0.08     0.75       0.10      0.94
"aperm"              0.02     0.19       0.14      1.31
"abs"                0.02     0.19       0.02      0.19
"mean.default"       0.02     0.19       0.02      0.19
"rep.int"            0.02     0.19       0.02      0.19

Let’s focus on the first two lines of the output. The second line talks about FUN, which is R’s way of talking about an anonymous function. It would be better if I had written out my functions in separate function declarations like a proper programmer. I’m sure there must be an R style guide out there which deprecates anonymous functions longer than two lines.

It’s the first line, however, which is key. Calls to apply are taking up the lion’s share of the compute time. It turns out that using the general function apply is overkill, and that more specialized functions can get row minima more quickly. Here’s a rewrite which replaces min_d <- apply(d, 1, min) with min_d <- rowMins(d), where rowMins() comes from the matrixStats package.

library(matrixStats)
tic()
out <- sapply(1:6, function(n_parties) {
    res <- sapply(1:n_sims, function(i) {
        parties <- rnorm(n_parties, mean = 0, sd = 1)
        d <- outer(voters, parties, `-`)
        d <- abs(d)
        min_d <- rowMins(d)
        return(mean(min_d))
### Create all the distances
    })
    return(res)
})
toc()

0.316 sec elapsed

As you can see, there’s a substantial improvement. Using rowMins is faster by a factor of twenty-five. If we look at the result of summaryRprof() below, we can see that the rowMins and outer function are the two big functions that are taking up most of the time.

               self.time self.pct total.time total.pct
"rowMins"           0.18    60.00       0.18     60.00
"mean.default"      0.06    20.00       0.06     20.00
"outer"             0.04    13.33       0.04     13.33
"abs"               0.02     6.67       0.02      6.67

Make it go faster

Is there any scope for speeding up further? Since we were able to speed things up by using a specialised function from a separate package, maybe we should use a specialized function from a specialised package, one which delights in making things go faster? Such a package is the Rfast package. I don’t know anything about the authors, but the package loads with such a ridiculous l33t startup message that it feels like they might have been interested in the demo coding scene in the 1990s.

library(Rfast)

Loading required package: Rcpp

Loading required package: zigg

Loading required package: RcppParallel

Attaching package: 'RcppParallel'

The following object is masked from 'package:Rcpp':

    LdFlags

Rfast: 2.1.5.1

 ___ __ __ __ __    __ __ __ __ __ _             _               __ __ __ __ __     __ __ __ __ __ __   
|  __ __ __ __  |  |  __ __ __ __ _/            / \             |  __ __ __ __ /   /__ __ _   _ __ __\  
| |           | |  | |                         / _ \            | |                        / /          
| |           | |  | |                        / / \ \           | |                       / /          
| |           | |  | |                       / /   \ \          | |                      / /          
| |__ __ __ __| |  | |__ __ __ __           / /     \ \         | |__ __ __ __ _        / /__/\          
|    __ __ __ __|  |  __ __ __ __|         / /__ _ __\ \        |_ __ __ __ _   |      / ___  /           
|   \              | |                    / _ _ _ _ _ _ \                     | |      \/  / /       
| |\ \             | |                   / /           \ \                    | |         / /          
| | \ \            | |                  / /             \ \                   | |        / /          
| |  \ \           | |                 / /               \ \                  | |       / /          
| |   \ \__ __ _   | |                / /                 \ \     _ __ __ __ _| |      / /          
|_|    \__ __ __\  |_|               /_/                   \_\   /_ __ __ __ ___|      \/             team

Attaching package: 'Rfast'

The following objects are masked from 'package:matrixStats':

    colMads, colMaxs, colMedians, colMins, colRanks, colVars, rowMads,
    rowMaxs, rowMedians, rowMins, rowRanks, rowVars

The following object is masked from 'package:tictoc':

    Stack

Let’s try their implementation of rowMins.

tic()
out <- sapply(1:6, function(n_parties) {
    res <- sapply(1:n_sims, function(i) {
        parties <- rnorm(n_parties, mean = 0, sd = 1)
        d <- outer(voters, parties, `-`)
        d <- abs(d)
        min_d <- Rfast::rowMins(d)
        return(mean(min_d))
### Create all the distances
    })
    return(res)
})
toc()

0.22 sec elapsed

Once again, we’ve shaved some time off. It may not appear like a lot of time, but this implementation is now 1.5 times quicker.

What doesn’t work (1)

This reliance on sampling might seem strange. Why, if we are postulating that voter distributions follow a normal distribution, and if we are setting the number of voters to A Big Number, don’t we aim for an exact solution?

You can, if you like, write out a function that will take a vector of party positions, calculate midpoints between them, and integrate areas between these midpoints – but it’s incredibly slow. R likes matrix operations; it doesn’t like integration.

What doesn’t work (2)

In this code, I’ve used two loops. R programmers are often told that loops should be avoided, if at all possible. One way of avoiding an outer loop is to endogenize the number of parties – to simply sample a number of parties per iteration. This involves a certain loss of control – we never get exactly 100 simulations per party system size – but does it help speed things up?

tic()
sf <- function() { 
    n_parties <- sample.int(6, size = 1)
    parties <- rnorm(n_parties, mean = 0, sd = 1)
    d <- outer(voters, parties, `-`)
    d <- abs(d)
    min_d <- Rfast::rowMins(d)
    return(c(n_parties, mean(min_d)))
### Create all the distances
}
out <- replicate(6 * n_sims, sf(), simplify = 'array')
toc()

0.261 sec elapsed

This loss of control doesn’t give an increase in speed. It seems that the overhead of the outer loop – which only runs over six elements – simply isn’t significant.

Conclusions

The meta level conclusion is that it’s always important to time things and check that your intuitions about what works or doesn’t work are correct.

The only-slightly-less-meta-level conclusion is that some “natural” R programming patterns aren’t always that quick. Although base R functions have been optimized a lot, they are generic tools, and some tools with a narrower remit can do the job quicker.

What is wrong with linearizing relationships?

Sweet summer children might believe that linearizing relationships cannot have any harmful consequences if it is so widely used. However, whenever we take logarithms we’re changing the type of mean we predict. In an ordinarily least squares regression, we’re predicting an average arithmetic mean. If we’re working on the log transformed scale, we’re still predicting an average arithmetic mean, but on the log scale. That means that when we reverse the log operation, the mean we get back will be smaller than the mean of the original, un-transformed data.

library(kableExtra)
x <- c(1, 10, 100, 1000, 10000)
y <- log10(x)
tab <- data.frame(`Original data` = x,
                  `Base 10 log transformed data` = y,
                  check.names = FALSE)
tab <- rbind(tab, colMeans(tab))
rownames(tab) <- sapply(seq_len(nrow(tab)), function(i) {
    paste0(rep(" ", i), collapse = "")
    })
rownames(tab)[5] <- "Mean"
             
kbl(tab)

Table 1 an example from Smith (1993). We have a set of numbers which range from one to ten thousand. Their average is a little over two thousand. Because these numbers are all positive, and span a couple of orders of magnitude, we might think to log-transform them. Here I’ll use log to the base 10 to keep the numbers simple. We might then estimate an intercept-only regression on the log-transformed values to get a mean. The mean of the log-transformed values is just 2, and when we back-transform we get , far less than the mean we “ought” to get.

The response to this “transformation bias” is very much to suggest that this theoretical problem is not a practical problem. Xiao et al. (2011) collect a large number of allometric datasets and fit models to them using the traditional method of log transformation and by using nonlinear least squares fits. The conclusion is that, for two thirds of data-sets, the fit between the model and data was better with a log-transformed model than with a nonlinear least squares model.

Additive and multiplicative error

Not everyone is convinced that the old ways are best. Gary Packard, who as far as I can tell has been waging a one-man battle against logarithmic transformations for allometry, argues that Xiao et al. (2011) make nonlinear regression look bad, because they assume that nonlinear regression modelling uses additive error, and that when nonlinear regression uses multiplicative error it looks much better.

What do I mean by additive or multiplicative error? I can explain it best by starting with the log transformed equation

where the error terms epsilon come from some normal distribution with a mean of zero and an estimated standard deviation .

In this equation, the error term is additive on the log-transformed scale. By additive, I just mean, “we add it on”, even though when we add on negative errors it is more natural to talk of subtraction.

However, when the error on log-transformed scale is additive, the error is multiplicative on the original scale. In order to show this, we exponentiate everything in the above equation to get

Here, any positive or negative values of are exponentiated, and multiply the term . Instead of adding or subtracting some error from our prediction, we scale it up or down by some factor.

Multiplicative error seems more appropriate when dealing with outcomes than span several orders of magnitude. Suppose we were dealing with deaths in wars. In some wars dozens of people die; in other wars millions of people die. Being able to predict war deaths “plus or minus a thousand” would be ridiculous in the case of a conflict which we predicted to involve a dozen deaths, and spuriously precise in cases where we predict millions of deaths.

Unfortunately, most nonlinear regression routines in statistical software don’t estimate multiplicative error. The variance structure recommended by Packard, and which is implemented in some R routines, is to set error proportional to the predicted value, raised to some power:

where is a parameter to be estimated, and where values greater than zero indicate that variance increases proportionate to the predicted value. A value of zero for implies constant additive variance on the original scale. Packard refers to this as additive heteroscedastic error, which is true in the sense that the error is added on, but confusing in the sense that the term here is clearly multiplying .

Worked example (1)

Let’s see how this might work in practice. I’m going to work with data from electoral studies, and specifically the National and District Level Party System Datasets described in Struthers et al. (2018). I’ll use this data to test some claims made by the Seat Product Model, as set out in Shugart and Taagepera (2017).

I begin by loading the libraries I need – haven for file import, modelsummary to show the results of my regressions, and nlme for the modelling.

library(tidyverse)
library(haven)
library(nlme)
library(modelsummary)
library(ggthemes)

I’ll then load in the data, and filter only to simple (i.e., single ballot, single tier, non-preferential) electoral systems where we know the seat product, or the average district magnitude times the assembly size.

dat <- read_dta("NLPS_v1_10-2018.dta") |>
    filter(simple == 1) |>
    filter(compens == 0) |>
    filter(parallel == 0) |>
    filter(!is.na(basictier1)) |>
    filter(!is.na(tier1_avemag)) |>
    filter(!is.na(Ns)) |>
    filter(!is.na(Ns0)) |>
    mutate(seat_product = basictier1 * tier1_avemag) |>
    dplyr::select(Ns, Ns0, seat_product) |>
    as.data.frame()

I’m going to begin with an example where three different methods reach the same conclusion. That is, I’m going to test the claim that the effective number of seat-winning parties is equal to the seat product, raised to the power of one sixth.

Let’s test the most basic claim, that the effective number of seat-winning parties is related to the seat product. We can (and should!) visualize these variables before we do anything else. Here I’ll use log-transformed axes, without by this implying any commitment to log transforming the variables in the regressions I’ll run shortly.

Code

ggplot(dat, aes(x = seat_product,
                y = Ns)) +
    geom_point(colour = "#ebebeb") +
    geom_smooth(se = FALSE) +
    scale_x_continuous("Seat product") +
    scale_y_continuous("Effective no. seat-winning parties") +
    coord_trans(x = "log10", y = "log10") +
    theme_solarized_2(light = FALSE)

The outlier to the right is Ukraine between 2006 and 2007, where the effective number of parties is low but where there were a tonne of independents.

m <- lm(log(Ns) ~ log(seat_product), data = dat)

my_cm <- c("(Intercept)" = "Constant",
           "log(seat_product)" = "log(Seat product)")
modelsummary(m,
             coef_map = my_cm,
             statistic = "conf.int",
             gof_map = c("nobs", "r.squared"))

We find that everything works – the confidence interval for our constant includes zero, and the confidence interval on the logged seat product includes one-sixth. We seem to be in the world described by Xiao et al. (2011), where log transforming “just works”.

We can now try modelling things in nls, R’s default routine for nonlinear least squares. This uses additive error, and so wouldn’t be recommended by Packard. As with all invocations of nls, we have to provide some starting values for the parameters, but with a relatively simple model these are innocuous.

m2 <- nls(Ns ~ seat_product ^ k,
          data = dat,
          start = list(k = 1/2))
m2b <- nls(Ns ~ a * seat_product ^ k,
           data = dat,
           start = list(a = 1, k = 1/2))

modelsummary(list(m2, m2b),
             statistic = "conf.int",
             gof_map = c("nobs", "r.squared"))

Despite the (probably incorrect) use of additive error , everything seems to work as it did before. In the first model, which uses just the exponent , the confidence interval includes one-sixth, even though it’s somewhat on the high side. In the second model, which includes an additional term to match the two term log log models just discussed, the confidence interval on the extra term includes , which is as it should be.

Finally, let’s try modelling it in nlme, with variance proportional to a power of the predicted value.

m3 <- gnls(Ns ~ seat_product ^ k, 
           data = dat,
           start = list(k = 1/2),
           weights = varPower(form = ~ fitted(.)))

m3b <- gnls(Ns ~ a * seat_product ^ k, 
           data = dat,
           start = list(a = 1, k = 1/2),
           weights = varPower(form = ~ fitted(.)))

modelsummary(list(m3, m3b),
             statistic = "conf.int",
             gof_map = c("nobs", "r.squared"))

Now in the two parameter model the exponent is exactly what it was predicted to be, to three decimal places. What the modelsummary output doesn’t show is the parameter , which is estimated to be somewhat greater than one, suggesting that error is slightly greater when we reach higher values of the seat product.

Worked example (2)

So far, so boring – modelling the relationship in three different ways gives substantially the same results. That might be a signal that we’ve found a robust relationship, but equally it might be sign that we are spending time on modelling details that don’t matter. What about an example where the three methods give different results?

We can stick with the same data-set, and model not the effective number of seat-winning parties but rather the raw number. The claim made by Shugart and Taagepera (2017) is that the raw number of seat-winning parties should be equal to the seat product to the power of one quarter. Taagepera and Shugart actually make their prediction about the effective number only after making a prediction for the raw number – so if these worked examples recapitulated the theoretical order of the claims, we should have started here with the raw number.

Let’s start with taking logs on both sides.

m <- update(m, log(Ns0) ~ .)

modelsummary(m,
             coef_map = my_cm,
             statistic = "conf.int",
             gof_map = c("nobs", "r.squared"))

So far so good – we have an intercept which could be zero, and a coefficient which could be one quarter, as predicted.

What about nonlinear least squares with additive error?

m2 <- nls(Ns0 ~ seat_product ^ k,
          data = dat,
          start = list(k = 1/2))
m2b <- nls(Ns0 ~ a * seat_product ^ k,
           data = dat,
           start = list(a = 1, k = 1/2))

modelsummary(list(m2, m2b),
             statistic = "conf.int",
             gof_map = c("nobs", "r.squared"))

Here something seems to have come off the rails. Although the single parameter model conforms to expectations, in the sense that the value of could be one quarter, the two parameter model has a value of that is far too low.

Can we rescue this – and nonlinear least squares – by incorporating error that is proportionate to the predicted value?

m3 <- gnls(Ns0 ~ seat_product ^ k, 
           data = dat,
           start = list(k = 1/2),
           weights = varPower(form = ~ fitted(.)))

m3b <- gnls(Ns0 ~ a * seat_product ^ k, 
           data = dat,
           start = list(a = 1, k = 1/2),
           weights = varPower(form = ~ fitted(.)))

modelsummary(list(m3, m3b),
             statistic = "conf.int",
             gof_map = c("nobs", "r.squared"))
m3 <- gnls(Ns0 ~ seat_product ^ k, 
           data = dat |> filter(seat_product < 1e5),
           start = list(k = 1/2),
           weights = varPower(form = ~ fitted(.)))

m3b <- gnls(Ns0 ~ a * seat_product ^ k, 
           data = dat |> filter(seat_product < 1e5),
           start = list(a = 1, k = 1/2),
           weights = varPower(form = ~ fitted(.)))

modelsummary(list(m3, m3b),
             statistic = "conf.int",
             gof_map = c("nobs", "r.squared"))

In model (2), the value of is once against consistent with the predicted value of one quarter, and the value of is (just) consistent with a value of one. Thus, although a log log regression gave the “right” answer, and one nonlinear regression gave the “wrong” answer, that was because when we estimate a nonlinear regression we have to get our error structure right.

Conclusions

In the worked examples I’ve set out, I was modelling numbers of parties, something that doesn’t vary over more than one order of magnitude.

The issues with fixed exponent relationships of the form are likely to be more severe when dealing with data like conflict deaths which do range over orders of magnitude.

As always, we need to think carefully about our errors, and use the appropriate tools for the job. Generally, social scientists don’t think enough about variance (Braumoeller 2006), but that’s something that we’re forced to do when we step into nonlinear modelling.

Taagepera (2007)

A derivation of the approximation is given in Taagepera (2007), pp. 163–164. I reproduce this here, discussing the upper limit and lower limit separately.

lb <- function(s1) {
    suppressPackageStartupMessages(require(dplyr))
    case_when(s1 > 1/2 ~ 1 / (1 - 2 * s1 + 2 * s1^2),
              s1 > 1/3 ~ 1 / (1 - 4 * s1 + 6 * s1^2),
              s1 > 1/4 ~ 1 / (1 - 6 * s1 + 12 * s1^2),
              s1 > 1/5 ~ 1 / (1 - 8 * s1 + 20 * s1^2),
              s1 > 0 ~ 1  / s1)
}
ub <- function(s1) {
    (1 + s1) / (2 * s1 ^ 2)
}

par(bg = "#0f2537",
    col.lab = "#ebebeb",
    col.axis = "#ebebeb",
    fg = "#ebebeb")

plot(c(0, 1), c(1, 20), type = "n",
     xlab = expression(s[1]),
     ylab = expression(N[S]),
     log = "y",
     col = "#ebebeb")

inx <- seq(0.1, 1, length.out = 100)
my_lb <- sapply(inx, lb)
my_ub <- sapply(inx, ub)
my_gm <- sqrt(my_lb * my_ub)

lines(inx, my_lb, lty = 2, col = "#5cb85c")
lines(inx, my_ub, lty = 2, col = "#5cb85c")
lines(inx, my_gm, lty = 1, col = "white")

Gress and Rosenberg (2024)

Gress and Rosenberg (2024) set out expressions for lower and upper bounds on the Hill numbers (=effective numbers) of a composition with largest element and number of elements . If we adopt our notation instead, then their expressions are as follows:

and

I find it very difficult to look at this formula, and the formulae above, and see whether they’re any different. I’m therefore going to take the dullard’s way, calculating things and then plotting them. I’ll assume .

lb2 <- function(s1) {
    (floor(1/s1) * s1 ^2 + (1 - floor(1/s1) * s1)^2)^(-1)
}
ub2 <- function(s1) {
    N_S0 <- 1 / s1^2
    (s1^2 + (N_S0 - 1) * ((1 - s1)/(N_S0 - 1))^2) ^ (-1)
}

par(bg = "#0f2537",
    col.lab = "#ebebeb",
    col.axis = "#ebebeb",
    fg = "#ebebeb")

plot(c(0, 1), c(1, 20), type = "n",
     xlab = expression(s[1]),
     ylab = expression(N[S]),
     log = "y")

inx <- seq(0.1, 1, length.out = 100)
my_lb2 <- sapply(inx, lb2)
my_ub2 <- sapply(inx, ub2)
my_gm2 <- sqrt(my_lb2 * my_ub2)

lines(inx, my_lb, lty = 2, col = "#5cb85c")
lines(inx, my_ub, lty = 2, col = "#5cb85c")
lines(inx, my_gm, lty = 1, col = "grey")

lines(inx, my_lb2, lty = 2, col = "#2b3e50")
lines(inx, my_ub2, lty = 2, col = "#2b3e50")
lines(inx, my_gm2, lty = 1, col = "white")

Here the two sets of bounds overlap perfectly, and so whilst the expression found in Gress and Rosenberg (2024) is neater, it doesn’t help us advance on Taagepera (2007).

Conclusion

Sometimes I am easily impressed by papers which look complicated, but which don’t yield advances in knowledge.

Staggered elections

In just over a quarter of English councils, councillors are elected in staggered elections where (usually) one third of the council is elected every year, with the final year in a four year cycle seeing no elections.

Over the past twenty years, a number of councils have moved from staggered election by thirds to whole council or “all-out” elections. As far as I am aware, only one council has ever moved from whole council elections to election by thirds: in 2004, Castle Point in Essex made the switch, only to switch back this year.

Castle Point Council, in a consultation document inviting local residents’ views on switching to whole council elections, provided a list of reasons for and against the switch. One of the reasons in favour of switching was that

“The results from whole-council elections are simpler and more easily understood by the electorate. This may increase turn-out at local elections.” (emphasis added)

Governments and politicians don’t always internalize the lessons of political science research, but this view is supported by research on the frequency of elections and turnout.

• Filip Kostelka, Eva Krejcova, Nicolas Sauger and Alexander Wuttke) have shown that turnout in the general election of 2017 was lower in areas which had held local elections five weeks previously. This finding backs up their main finding (based on cross-national panel data) that the number of elections in a five year window is negatively associated with turnout.
• Colin Rallings, Michael Thrasher and Galina Borisyuk have shown that turnout in local authority by-elections is lower the less time has elapsed since the main council elections

Rallings, Thrasher and Borisyuk in particular treat the frequency of irregular by-elections as an illustration of a general point:

“One piece of evidence from British elections is the relative level of electoral turnout in the 32 London boroughs, which have whole council elections every four years, compared with that in the 36 metropolitan boroughs covering the major English conurbations outside London, which elect a third of their council membership annually. The two types of authority are responsible for the same local services, but the level of turnout is consistently higher in London. However, when general election turnout is examined the reverse position applies, with participation consistently higher in the metropolitan boroughs… A crucial factor in explaining these differences is likely to be that electors in the London boroughs are determining the composition of their council whereas their counterparts in the metropolitan boroughs will often be unable to effect a change in control however they cast their votes. Nonetheless, we should not ignore the possibility that the greater frequency with which electors in the metropolitan boroughs are expected to vote may also be a contributory factor to the differences in turnout” (emphasis added)

This paragraph nicely illustrates the two ways in which staggered elections can cause changes in turnout: by reducing the stakes (only one third of the council is at play) and by increasing election frequency. Of course, when term length is fixed, then increasing election frequency necessarily means reducing the stakes, and so some questions about mechanisms are moot in the English local government context.

Staggered elections and unitarisation

Staggered elections are found in a variety of English councils. Where there are staggered elections, councillors are usually elected in thirds, but a couple of weirdo councils (hello Oxford!) elect by halves.

Staggered elections aren’t only found in district councils. In fact, it’s metropolitan boroughs which are the most frequent users of this election type.

However when district councils have unitarised, the resulting unitary authorities tend to adopt all-out elections. For example: although two former councils in the North Yorkshire unitary council area (Craven and Harrogate) were elected by thirds, the new unitary council is elected in an all-out election. Similarly in Cumberland, although Carlisle elected by thirds the unitary council is elected in all-out elections.

Unitarisation might therefore reduce the number of councils electing by thirds, because no one choosing a new electoral system from scratch now seems to choose election by thirds.² Although the government’s broader plans might include incentives for metropolitan boroughs to move away from election by thirds, it is far easier to choose a system for a new institution than change a system for an existing institution. Unitarisation won’t eliminate election by thirds, but it will likely make it less common.

All-out elections and turnout

So far all I’ve shown you is that some district councils use staggered elections, and that there are good reasons from other political science studies to think that this damages turnout. Now I’m going to go one step beyond, and try and estimate how much moving to all-out elections could boost turnout, using data from English local elections.

The data I’m using mostly comes from successive editions of the Rallings/Thrasher Local Elections Handbook; more recent data comes from the Electoral Commission or the handbooks jointly produced by Rallings/Thrasher and the House of Commons library. The turnout figures here are “ballot box turnout figures”, which includes ballots which were rejected at the count. The data covers from the period from 1991 to 2023; I’m not aware of a good complete source for turnout in the 2024 local elections. You can download the data in CSV format here.

The data includes information on the type of election: whole, thirds, or halves. This applies to the current election, rather than the general scheme of elections used by that council. As a result, councils can have “whole council” elections for two reasons:

• they generally have whole council elections;
• they generally have staggered elections, but have had to hold whole council elections as a result of boundary changes

Whole council elections which were held just because of boundary changes are marked in the column whole_bc_redistr.

Figure 1 shows my attempt to plot the raw data. You’ll see peaks in election years (1997, 2001, 2005, 2010, and 2015). I’ve also overlaid three trend lines for the different election frequencies. These trend lines all share the same pattern over time, but they’re shifted up or down to best fit the pattern for the different election frequencies. The trend line for “whole council” elections is around four percentage points higher than the trend line for election by thirds. However, the higher turnout in these elections might be the result of the type of places which have these kinds of elections. What we really need are changes within councils.

We can estimate changes within councils by fitting a two-way fixed effects regression model. In TWFE models, we allow each year to affect turnout in its own unique way, and each unit (here, councils) to affect turnout in its own unique way. These year and unit fixed effects are nuisance parameters that we don’t really care about. What we do care about are the effects of other variables – in our case, the effects of election frequency.

If councils only ever held elections at the one frequency, then we wouldn’t be able to disentangle the council-specific effects from the effects of election frequency – but enough councils have changed election frequency over time to allow us to estimate some effects.

Any other things which might effect turnout – the presence of police and crime commissioner elections, concurrent European Parliament elections, postal vote trials, the changing quality of governance in the council – are either ignored or wrapped up into the year fixed effects.

Table 1 shows the results of four different regression models of turnout in English local elections. The zero-th model includes no control variables at all, and just shows what was previously shown in Figure 1: that councils which hold all-out elections have turnout which is slightly moer than four points higher than councils which hold elections by-thirds.

Model (1) then goes on to include controls for year and the council. Under certain assumptions,³ we can now start speaking of the “effect” of all-out elections, compared to election by thirds. This first estimate of the effect suggests that moving to all out elections increases turnout by around 1.3 percentage points. This is quite a big effect. It’s around one quarter of the within-year standard deviation in turnout between councils. It’s also around the same size as the negative effect of general election turnout as a result of the new voter identification laws. However, since local election turnout is lower than general election turnout, the effect is proportionately bigger.

It’s true that the effect does drop a little bit when we drop some observations which we might think, for various reasons, to be problematic. Model (2) drops cases where a council held all-out elections required by redistricting. Our best estimate of the effect is now less than a percentage point. However, the difference between these two estimates isn’t itself significant: the confidence interval now runs between 0.2 percentage points and 1.5 percentage points, and so includes the point estimate from Model (1). When we drop turnout levels in 2004, which might have been affected by all-postal voting trials, the estimate goes back up a little, but by such a small amount relative to the degree of uncertainty that it’s really not worth reporting.

This suggests to me that moving to all-out elections would boost turnout, compared to election by thirds – and that if unitarisation involves getting rid of election by thirds then that might be something to be said in its favour.

Conclusion

Even people who think about local government reform probably don’t think first of its effects on turnout. However, someone probably should. For the last four years for which there is consistent data (2019 to 2023), turnout has been below fifty percent in every local authority area. I don’t know whether mission-led government is the same as government by target – but it would be nice to think of a mission to boost turnout. What would it take to bring turnout in local elections over fifty percent, what would that cost, and what trade-offs might be involved?

Victorian number generator

I was thinking about Plato thanks to a recent trip to Terra Australis Australia, where the Parliament of Victora is reviewing electoral arrangements for its upper chamber, the Legislative Council.

Two parts of the review concern the size of the Legislative Council and the number of electoral districts. At present, the Legislative Council has 40 members, or roughly four-ninths the size of the Legislative Assembly, which has 88 members. The forty members of the Legislative Council are elected in eight districts, each with a district magnitude of five.

Considering first the size of the Legislative Council, the ratio between the Council and the Assembly is very slightly greater than the ratio between most upper chambers and their lower chambers. In general, upper chambers are around 40% the size of the corresponding lower chamber.

You can see that in Figure 1, which uses data from the International Parliamentary Union showing lower chamber size on the horizontal axis against upper chamber size on the vertical axis. The dotted line gives the best-fitting line according to ordinary least squares; the solid line shows the line where the size of the upper chamber is equal to the size of the lower chamber. The Legislative Council is slightly above the trend line, but still well within the confidence interval of the associated linear regression.

Code

ipu_download <- "compare--statutory_members_number--2024--export--241118-011132--EN.csv"
if (file.exists(ipu_download)) {
    dat <- read.csv(ipu_download) |>
        filter(Status != "Suspended") |>
        dplyr::select(iso2c = ISO.Code,
                      Members = Statutory.number.of.members,
                      Chamber.type) |>
        mutate(Chamber.type = ifelse(Chamber.type == "-",
                                     "Lower chamber",
                                     as.character(Chamber.type))) |>
        pivot_wider(names_from = Chamber.type,
                    values_from = Members) |>
        as.data.frame()
} else {
    
dat <- structure(list(iso2c = c("AL", "DZ", "AD", "AO", "AG", "AR",
"AM", "AU", "AT", "AZ", "BS", "BH", "BD", "BB", "BY", "BE", "BZ",
"BJ", "BT", "BO", "BA", "BW", "BR", "BN", "BG", "BF", "BI", "CV",
"KH", "CM", "CA", "CF", "TD", "CL", "CN", "CO", "KM", "CG", "CR",
"CI", "HR", "CU", "CY", "CZ", "KP", "CD", "DK", "DJ", "DM", "DO",
"EC", "EG", "SV", "GQ", "ER", "EE", "SZ", "ET", "FJ", "FI", "FR",
"GA", "GM", "GE", "DE", "GH", "GR", "GD", "GT", "GN", "GW", "GY",
"HT", "HN", "HU", "IS", "IN", "ID", "IR", "IQ", "IE", "IL", "IT",
"JM", "JP", "JO", "KZ", "KE", "KI", "KW", "KG", "LA", "LV", "LB",
"LS", "LR", "LY", "LI", "LT", "LU", "MG", "MW", "MY", "MV", "ML",
"MT", "MH", "MR", "MU", "MX", "FM", "MC", "MN", "ME", "MA", "MZ", "NA",
"NR", "NP", "NL", "NZ", "NI", "NE", "NG", "MK", "NO", "OM", "PK",
"PW", "PA", "PG", "PY", "PE", "PH", "PL", "PT", "QA", "KR", "MD",
"RO", "RU", "RW", "KN", "LC", "VC", "WS", "SM", "ST", "SA", "SN",
"RS", "SC", "SL", "SG", "SK", "SI", "SB", "SO", "ZA", "SS", "ES",
"LK", "SR", "SE", "CH", "SY", "TJ", "TH", "TL", "TG", "TO", "TT",
"TN", "TR", "TM", "TV", "UG", "UA", "AE", "GB", "TZ", "US", "UY",
"UZ", "VU", "VE", "VN", "YE", "ZM", "ZW"), `Lower chamber` = c(140L,
407L, 28L, 220L, 18L, 257L, 107L, 151L, 183L, 125L, 39L, 40L, 350L,
30L, 110L, 150L, 32L, 109L, 47L, 130L, 42L, 69L, 513L, 45L, 240L, 71L,
123L, 72L, 125L, 180L, 338L, 140L, 203L, 155L, 3000L, 187L, 24L, 151L,
57L, 255L, 151L, 470L, 80L, 200L, 687L, 500L, 179L, 65L, 32L, 190L,
137L, 596L, 60L, 100L, 150L, 101L, 74L, 547L, 55L, 200L, 577L, 98L,
58L, 150L, 598L, 275L, 300L, 15L, 160L, 81L, 102L, 69L, 119L, 128L,
199L, 63L, 545L, 580L, 290L, 329L, 160L, 120L, 400L, 63L, 465L, 138L,
98L, 350L, 45L, 65L, 90L, 164L, 100L, 128L, 122L, 73L, 200L, 25L,
141L, 60L, 163L, 193L, 223L, 93L, 147L, 65L, 33L, 176L, 66L, 500L,
14L, 24L, 126L, 81L, 395L, 250L, 104L, 19L, 275L, 150L, 120L, 91L,
171L, 360L, 123L, 169L, 90L, 336L, 16L, 71L, 118L, 80L, 130L, 316L,
460L, 230L, 45L, 300L, 101L, 330L, 450L, 80L, 16L, 18L, 22L, 51L, 60L,
55L, 151L, 165L, 250L, 35L, 149L, 104L, 150L, 90L, 50L, 275L, 400L,
550L, 350L, 225L, 51L, 349L, 200L, 250L, 63L, 500L, 65L, 113L, 30L,
42L, 161L, 600L, 125L, 16L, 529L, 450L, 40L, 650L, 393L, 435L, 99L,
150L, 52L, 167L, 500L, 301L, 167L, 280L), `Upper chamber` = c(NA,
174L, NA, NA, 17L, 72L, NA, 76L, 60L, NA, 16L, 40L, NA, 21L, 65L, 60L,
13L, NA, 25L, 36L, 15L, NA, 81L, NA, NA, NA, 39L, NA, 62L, 100L, 105L,
NA, NA, 50L, NA, 108L, NA, 72L, NA, 99L, NA, NA, NA, 81L, NA, 109L,
NA, NA, NA, 32L, NA, 300L, NA, 70L, NA, NA, 30L, 153L, NA, NA, 348L,
70L, NA, NA, 69L, NA, NA, 13L, NA, NA, NA, NA, 30L, NA, NA, NA, 245L,
NA, NA, NA, 60L, NA, 205L, 21L, 248L, 69L, 50L, 68L, NA, NA, NA, NA,
NA, NA, 33L, 30L, NA, NA, NA, NA, 18L, NA, 70L, NA, NA, NA, NA, NA,
NA, 128L, NA, NA, NA, NA, 120L, NA, 42L, NA, 59L, 75L, NA, NA, NA,
109L, NA, NA, 87L, 96L, 15L, NA, NA, 45L, NA, 24L, 100L, NA, NA, NA,
NA, 136L, 170L, 26L, NA, 11L, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, 40L, NA, 54L, 90L, 100L, 265L, NA, NA, NA, 46L, NA, 33L, 200L, NA,
NA, NA, 32L, 77L, NA, NA, NA, NA, NA, NA, 800L, NA, 100L, 31L, 100L,
NA, NA, NA, 111L, NA, 80L)),
row.names = c(NA, -190L),
class = "data.frame")
}

### Let's drop the 3,000 member chamber
dat <- dat |>
    filter(`Lower chamber` < 3000)

ggplot(dat,
       aes(x = `Lower chamber`,
           y = `Upper chamber`)) +
    geom_point(colour = "#ebebeb") +
    geom_smooth(formula = y ~ x,
                method = "lm",
                linetype = 2,
                color = "#df691a") +
    scale_x_continuous(limits = c(0, 700),
                       expand = expansion(c(0, 0.05))) +
    scale_y_continuous(limits = c(0, 800),
                       expand = expansion(c(0, 0.05))) +
    ### geom_abline(color = "#df691a") +
    geom_textabline(color = "#df691a",
                    label = "Upper chamber size equals Lower chamber size") +
    annotate(geom = "point",
             size = 1.5,
             x = 88,
             y = 44,
             colour = "#5cb85c") +
    annotate(geom = "text",
             x = 88,
             y = 144,
             hjust = 1,
             vjust = 0,
             label = "Victoria\nLeg. Council",
             colour = "#5cb85c") +
    annotate(geom = "curve",
             x = 48,
             y = 140,
             xend = 88 + 1,
             yend = 44 - 1,
             hjust = 1,
             vjust = 0,
             arrow = arrow(length = unit(0.01, "npc")),
             colour = "#5cb85c") + 
    theme_dark() +
    theme(plot.background = element_rect(fill = "#0f2537",
                                         colour = NA),
          panel.background = element_rect(fill = "#2b3e50"),
          text = element_text(colour = "#ebebeb"),
          axis.text = element_text(colour = "#ebebeb"))

How does the size of the legislative council interact with the size of the districts? If you want equal sized districts (and that’s a big if), then the total size of the Legislative Council sets limits on how many regions you can have. If you have 40 members, then you can have eight districts each of five members, or five districts of eight members, but you can’t have six evenly sized districts or seven evenly sized districts. If you had the twin primes (39 or 41 members), you wouldn’t be able to have evenly sized districts at all.

(Having evenly sized districts isn’t a necessary criterion, but the discussion paper which accompanies the review is mostly illustrated with even-sized districts, and the impact of variation in district magnitude upon system properties is in my view still an open question.)

Since you can change districts more easily than you can change the number of members of a legislature, it seems like we should choose the size of the legislature so it’s appropriate (is smaller than the lower chamber by some commonly found ratio) and permits lots of possible divisions.

In the case of the Victorian Legislative Council, a chamber with 36 members would more closely approximate the ratio found amongst national upper chambers (36/88 40%), and is highly divisible, allowing evenly-sized districts with 1, 2, 3, 4, 6, 9, 12, or 18 members, or eight different options. Indeed 36 is not only highly divisible in the sense of having a lot of divisors for a number its size, it’s highly composite. This means that a assembly size of 36 allows more flexibility than a legislative council of 40, which allows evenly sized districts with 1, 2, 4 5, 8, 10 and 20 members, or seven different options.

It’s all a bit odd

Perhaps this property of divisibility strikes you as verging into numerology. I’m certainly not aware that anyone has analysed how easily legislative assemblies can be divided. The only remotely similar property I’ve heard discussed is oddness and evenness, where there is an argument for having odd-sized chambers.

The argument goes like this: that democracy is about majority rule, and in an odd-sized chamber there will always be a majority if votes are binary votes and if abstentions are not allowed. By contrast, in an even-sized chamber there might be a tie. There are therefore some circumstances in which even-sized chambers fail to identify a majority either in favour or in opposition to some proposal. If a tie leads to failure of a motion, then this creates an anti-democratic bias towards the status quo, with the size of the bias depending on how big the chamber is and how often controversial issues will be finely balanced.

At least one parliament was designed explicitly to have an odd number of members and thereby prevent a tie. In the 1973 election to the Swedish Riksdag, government and opposition forces both won 175 of 350 seats. The number of seats was changed to 349 for the following 1976 election.²

Unfortunately, it seems that no one is paying attention to the Swedish experience. Most legislative chambers have an even number of members, as Figure 2 shows. This figure uses the same IPU data used to plot Figure 1, except that this time I include all chambers, whether or not the system is unicameral or bicameral, and I drop the Chinese National People’s Congress, because it’s absolutely huge.

Code

all_chambers <- c(dat$`Lower chamber`, dat$`Upper chamber`)
plot_df <- tribble(~comparison, ~parity, ~proportion,
        "All chambers", "Even", mean(all_chambers %% 2 == 0, na.rm = TRUE),
        "All chambers", "Odd", mean(all_chambers %% 2 == 1, na.rm = TRUE),
        "Lower chambers", "Even", mean(dat$`Lower chamber` %% 2 == 0),
        "Lower chambers", "Odd", mean(dat$`Lower chamber` %% 2 == 1),
        "Upper chambers", "Even", mean(dat$`Upper chamber` %% 2 == 0, na.rm = TRUE),
        "Upper chambers", "Odd", mean(dat$`Upper chamber` %% 2 == 1, na.rm = TRUE))

ggplot(plot_df,
       aes(x = parity,
           y = proportion,
           fill = parity)) +
    geom_col() +
    scale_x_discrete("Parity") +
    scale_y_continuous("Proportion of legislatures",
                       labels = scales::percent) +
    scale_fill_manual("", values = c("#df691a",
                                     colorspace::lighten("#df691a", 0.5)),
                      guide = "none") + 
    facet_wrap(~comparison, nrow = 1) +
    geom_hline(yintercept = 0.5,
               linetype = 2) +
    theme_dark() +
    theme(plot.background = element_rect(fill = "#0f2537",
                                         colour = NA),
          panel.background = element_rect(fill = "#2b3e50"),
          text = element_text(colour = "#ebebeb"),
          axis.text = element_text(colour = "#ebebeb"))    

The preference for even-sized assemblies is not overwhelming, but it is a bit more pronounced amongst upper chambers, where almost three out of every five chambers have an even number of members. Although having an odd number of members might help prevent ties, having an even number of members does help with Plato’s concern of divisibility. I’ll look at divisibility now.

A house divided

When we looked at whether odd- or even-sized chambers are preferred, we could calculate whether each individual chamber has an odd or even number of members, and then compare that to the expected proportion of even and odd numbers. That proportion is constant, at fifty percent.

When we look at divisibility, it’s not so clear what we’re supposed to compare to. We can calculate the number of divisors for each number, but it’s not clear what benchmark we should use for this.

Here’s an R function, taken from Stack Overflow, which will calculate the divisors for a given assembly size:

divisors <- function(x){
  #  Vector of numbers to test against
  y <- seq_len(x)
  #  Modulo division. If remainder is 0 that number is a divisor of x so return it
  y[ x%%y == 0 ]
}

We can list the divisors for the current size of the Victorian Legislative Council:

divisors(40)

[1]  1  2  4  5  8 10 20 40

and calculate the number of divisors (including the number itself):

n_divisors <- function(x) {
    sapply(x, function(x)length(divisors(x)))
}
n_divisors(40)

[1] 8

We can go on to calculate this for all chambers in the IPU data.

all_chambers <- c(dat$`Lower chamber`, dat$`Upper chamber`)
all_chambers <- na.omit(all_chambers)
stat <- mean(n_divisors(all_chambers))
stat

[1] 7.279851

The average legislative chamber therefore has just over seven divisors. We can calculate this, but we’re forced back to two classic questions, “is that a lot or a little?” and “a lot or little compared to what?”

Getting to the kernel of the comparison

Ideally we would like to compare the average divisibility of the chambers that actually exist to the average divisibility of numbers that are similar to the numbers we have in relevant respects (some bigger, some smaller) but which can be dissimilar in terms of their divisibility.

One way of doing this is to estimate a smoothed distribution which matches the numbers we see. Kernel density estimators are a good way of doing this. It’s not necessary to understand how a kernel density is constructed, but it is necessary to understand the output of the procedure.

Code

all_chambers <- all_chambers[all_chambers < 1000]
plot_df <- data.frame(x = all_chambers)

ggplot(plot_df, aes(x = x)) +
    geom_histogram(aes(y= after_stat(density)),
                   fill = "white", binwidth = 10) +
    geom_density(colour = "#df691a", linewidth = 1.5) +
    scale_x_continuous("Chamber size") +
    scale_y_continuous("Density") + 
    theme_dark() + 
    theme(plot.background = element_rect(fill = "#0f2537",
                                         colour = NA),
          panel.background = element_rect(fill = "#2b3e50"),
          text = element_text(colour = "#ebebeb"),
          axis.text = element_text(colour = "#ebebeb"))    

Figure 3 shows a histogram of chamber sizes. The orange line is the kernel density estimate. It reproduces the shape of the histogram – greater density at lower values, a long tail out towards the seven hundreds – but without reproducing exactly the spikes at certain values. The smoothness of the shape depends on a bandwidth parameter: here I’ve used the ggplot default, which is also the R default.

What we can therefore do is:

• estimate a kernel density using the actual sizes of legislative chambers;
• randomly draw from that estimated distribution
• calculate the average number of divisors of those “fake” chamber sizes
• compare the actual average number of divisors to the average on the fake data.

Here’s some code that will do just that. Note that I’ve taken logs so that we don’t get any negative numbers from our kernel density estimate.

set.seed(448)
z <- density(log(all_chambers),
             kernel = "gaussian")

rdens <- function(n,
                  z,
                  data) {
  width <- z$bw
  rkernel <- function(n) rnorm(n, sd = width)
  sample(data, n, replace=TRUE) + rkernel(n)
}

repl_func <- function(x) {
    ### Generate the density on the log scale
    newx <- rdens(length(x), z, x)
    ### Exponentiate and round to get back on the integer scale
    newx <- round(exp(newx))
    newstat <- mean(n_divisors(newx))
    return(newstat)
}

res <- replicate(1000, repl_func(log(all_chambers)))

We now have one thousand numbers giving the average number of divisors of fake chamber sizes in one thousand fake sets of data. We can plot that distribution, and plot the real world figure next to it.

Code

plot_df <- data.frame(x = res)

ggplot(plot_df, aes(x = x)) +
    geom_density(colour = "#df691a",
                 fill = colorspace::lighten("#df691a", 0.2),
                 linewidth = 1.5) +
    geom_vline(xintercept = stat,
               colour = "#5cb85c") + 
    scale_x_continuous("Average number of divisors") +
    scale_y_continuous("Density") + 
    theme_dark() +
    annotate(geom = "text",
             label = "Simulated data",
             x = 6.25,
             y = 1.6,
             hjust = 0,
             colour = "#df691a") +
    annotate(geom = "segment",
             x = 6.25,
             y = 1.6,
             yend = 1.6,
             xend = 5.95,
             arrow = arrow(length = unit(0.02, "npc")),
             colour = "#df691a") + 
    annotate(geom = "text",
             label = "Real world data",
             x = 7,
             y = 1.25,
             hjust = 1,
             colour = "#5cb85c") +
    annotate(geom = "segment",
             x = 7,
             y = 1.25,
             yend = 1.25,
             xend = stat,
             arrow = arrow(length = unit(0.02, "npc")),
             colour = "#5cb85c") + 
    theme(plot.background = element_rect(fill = "#0f2537",
                                         colour = NA),
          panel.background = element_rect(fill = "#2b3e50"),
          text = element_text(colour = "#ebebeb"),
          axis.text = element_text(colour = "#ebebeb"))    

The green line, representing data from the real world, is way out to the right of the distribution resulting from our fake data. This means that real chamber sizes are much more divisible than fake data which try to match the overall characteristics of chamber sizes.

A simpler way to the same conclusion

Although kernel density estimates are the most obvious way of mimicking the distribution of a sample, we don’t have to resort to such a complicated technique. Here’s an alternative way of generating something that “looks like” chamber sizes, but can have different divisibility:

• take each actual chamber size
• draw as many random numbers from [-1, 0, +1] as there are actual chamber sizes
• add on these increments to create some fake data
• calculate the average number of divisors for the fake data

This code does just that, and stores the average number of divisors.

repl_func2 <- function(x) {
### Generate the density on the log scale
    increments <- sample(c(-1, 0, 1), length(x), replace = TRUE)
    newx <- x + increments
    newstat <- mean(n_divisors(newx))
    return(newstat)
}

res2 <- replicate(1000, repl_func2(all_chambers))

As before, we can plot the distribution of the fake data and overlay the actual figure.

Code

plot_df <- data.frame(x = res2)

ggplot(plot_df, aes(x = x)) +
    geom_density(colour = "#df691a",
                 fill = colorspace::lighten("#df691a", 0.2),
                 linewidth = 1.5) +
    geom_vline(xintercept = stat,
               colour = "#5cb85c") + 
    scale_x_continuous("Average number of divisors") +
    scale_y_continuous("Density") + 
    theme_dark() +
        annotate(geom = "text",
             label = "Simulated data",
             x = 6.25,
             y = 1.6,
             hjust = 0,
             colour = "#df691a") +
    annotate(geom = "segment",
             x = 6.25,
             y = 1.6,
             yend = 1.6,
             xend = 6.05,
             arrow = arrow(length = unit(0.02, "npc")),
             colour = "#df691a") + 
    annotate(geom = "text",
             label = "Real world data",
             x = 7,
             y = 1.25,
             hjust = 1,
             colour = "#5cb85c") +
    annotate(geom = "segment",
             x = 7,
             y = 1.25,
             yend = 1.25,
             xend = stat,
             arrow = arrow(length = unit(0.02, "npc")),
             colour = "#5cb85c") + 
    theme(plot.background = element_rect(fill = "#0f2537",
                                         colour = NA),
          panel.background = element_rect(fill = "#2b3e50"),
          text = element_text(colour = "#ebebeb"),
          axis.text = element_text(colour = "#ebebeb"))    

Just as before, the real world legislative chambers we see are more divisible than the fake data we’ve generated by adding or subtracting a single seat. Whether we use a fancy technique (kernel density estimation) or a simple technique (adding or subtracting one), we reach the same conclusion: legislative chamber sizes are more divisible than they might otherwise be.

Why do we get this pattern?

It would be nice to think that constitutional framers are thinking carefully about the divisibility of their legislatures, and choosing that number “most likely to be useful to all [polities]”. Unfortunately, I don’t think that constitutional framers think like that, and that some of these patterns have to do with how much we like particular numbers.

Our like or dislike of certain numbers might seem a pretty slender basis for choosing the size of a legislature. After all, some likes and dislikes are culturally constructed. My house number is 44, which is just about the worst number you can have in Chinese, where 44 sounds a lot like “death death”. Many hotels have no thirteenth floor, but the association between thirteen and bad luck probably only holds in cultures that have some Christian roots.

Some numbers, however, do seem to be liked more than others because they’re easily accessible. Dan King and Chris Janiszewski claim that people like small numbers, and numbers which are the result of common additions and multiplications. Other authors reach similar conclusions.

To return to the numbers used by the Victorian Legislative Council: I might like the number forty more than I like the number thirty-nine or the number forty-one because I spent time as a child learning my times tables, and because forty appears in those times tables as the answer to five times eight and eight times five.

The likeability of numbers is therefore connected to their divisibility, but likeability probabily trumps divisibility. In King and Janiszewski’s data, numbers ending in five are liked more than other numbers. That probably has something to do with the fact that most humans have five fingers on each hand, and use hands to count with. But numbers ending in five aren’t always very divisible. Twenty-five has far fewer divisors (3) than twenty-four (8), but people apparently like twenty-five more than they like twenty-four. Even in the histogram shown in Figure 3 you can see the spikes at 25, 50, 75 and 100. We like the numbers that we like, and divisibility is a happy accident.

This of course is an explanation of why the distribution of chamber sizes is what it is, rather than a justification. For my part, I’ll be writing to suggest that the Victorian Legislative Council have a highly divisible, highly composite number of members (36) rather than the 40 it has now.

Acknowledgements

Thanks to @jameschalmers.bsky.social, @urbaneprofessor.bsky.social and @paulfscott.bsky.social, all of whom sent me copies of Bort’s article “The Numbers Game”.

Replicating what was done

I’ll begin by loading the libraries I need, and the replication data. I load tidyverse for general data cleaning, mclogit for modelling. I use modelsummary and hrbrthemes for making stuff look pretty.

set.seed(1891)
suppressPackageStartupMessages({
    library(tidyverse)
    library(mclogit)
    library(modelsummary)
    library(hrbrthemes)
})

dat <- read.delim("ShortlistedCands2019Anonymous.csv",
                  sep = ";",
                  na.strings = c("", "-", "NA"))

I’m going to filter to cases where we know the outcome. I’m also going to create a unique contest identifier, which is created by interacting party and constituency.

dat <- dat |>
    filter(!is.na(success)) |>
    filter(!is.na(BAME)) |>
    mutate(contest = as.numeric(interaction(party, constituency)))

I’ll also remove contests where there was only one candidate listed.

dat <- dat |>
    group_by(contest) |>
    mutate(n_cands = n(),
           pos_within_contest = 1:n()) |>
    ungroup() |>
    filter(n_cands > 1)

I’m also going to drop variables I don’t need:

dat <- dat |>
    dplyr::select(success, contest, gender, BAME, local,
                  past_selection, elected_cllr, occ_party)

Using this, I’ll estimate a model using the same predictors used by the authors to create their Figure 2: being female, being from an ethnic minority, living locally, having stood previously, being an elected councillor, and having worked for the party.

m <- mclogit(cbind(success, contest) ~ gender + BAME + local +
                 past_selection + elected_cllr + 
                 occ_party,
             data = dat)

Iteration 1 - deviance = 265.8233 - criterion = 0.4664284
Iteration 2 - deviance = 265.7618 - criterion = 0.0002312444
Iteration 3 - deviance = 265.7618 - criterion = 1.594636e-08
Iteration 4 - deviance = 265.7618 - criterion = 2.138082e-16
converged

I can then report this model.

my_cm <- c('gender' = 'Female',
        'BAME' = 'Ethnic minority',
        'local' = 'Lives locally',
        'past_selection' = 'Stood before',
        'elected_cllr' = 'Elected councillor',
        'occ_party' = 'Worked for party')
modelsummary(m,
             coef_map = my_cm,
             statistic = "conf.int") 

Now the coefficient values are there, and you can check how big they are, but it’s quite hard to relate those coefficients – log odds ratios of being chosen – to anything understandable by humans.

We will therefore need to calculate quantities we can understand – like changes in probabilities – under some specific scenarios. Ordinarily, when we calculate human-readable quantities of interest, we calculate average marginal effects, or the effect of changing a focal variable by one unit for each observation, and averaging this over all observations.

A targeted counterfactual

In cases like this, it doesn’t make sense to change the value of the focal variable for one unit for all observations. If the change of one unit affects all observations, then it doesn’t alter the probabilities of success.

We need to therefore consider a more targeted intervention – specifically, what happens if we change the value of the focal variable for a randomly chosen individuals within each contest. We can then average over each of these focal individuals, and calculate the change in the probability for them.

Here’s a function which will take in a data frame, and a variable name contained in a string var and restrict the analysis to contests where there was some variable in the focal variable. It will then randomly choose an individual with the minimum value of the focal variable, alter that individual’s value of the focal variable to the maximum value, and tag that individual as “focal”. This altered variable is stored under the name cfval.

create_counterfactual <- function(data, var) {
    ### Only look at those situations where there is more than one unique value of the variable
    cf <- data |>
        dplyr::ungroup() |>
        dplyr::group_by(contest) |>
        dplyr::mutate(n_distinct = n_distinct(get(var))) |>
        dplyr::ungroup() |>
        dplyr::filter(n_distinct > 1)
    ## set one minimum value to the maximum value
    drop_first <- function(x) {
        x[-1]
    }
    cf <- cf |>
        mutate(randno = rnorm(n())) |>
        dplyr::group_by(contest) |>
        dplyr::arrange(.data[[var]], randno) |>
        dplyr::mutate(cfval = c(max(get(var)), drop_first(get(var))),
                      focal = c(1, rep(0, n() - 1))) |>
        dplyr::select(-randno)
    return(cf)
}

Now that we can create altered sets of data, we need to be able to calculate predicted probabilities. That can be done using the calc_pps function below, which calculates the linear predictor mu, exponentiates it, and divides by the sum of exponentiated predicted probabilities for each contest.

calc_pps <- function(b, X, grp) {
    mu <- X %*% b
    emu <- exp(mu)[,1]
    grpsums <- ave(emu, grp, FUN = sum)
    pr <- emu / grpsums
    return(pr)
}

Finally, we can put this in a overall function, which creates the counterfactuals, does some messing around with model matrices, before drawing some coefficients and calculating averages.

calc_fx <- function(mod, df, var, n_sims = 1000) {
    cf <- create_counterfactual(df, var)

    ### Create the two model matrices
    base_mat <- model.matrix(formula(mod), cf)[,-1]
    newf <- as.character(formula(mod))
    newf <- sub(as.character(var), "cfval", newf)
    newf <- as.formula(paste(newf[[2]], newf[[3]], sep = " ~ "))
    cf_mat <- model.matrix(newf, cf)[,-1]

### Get some betas
    betas <- MASS::mvrnorm(n = n_sims,
                           mu = coef(mod),
                           Sigma = vcov(mod))
    pp <- apply(betas, 1, function(b) calc_pps(b, base_mat, cf$contest))
    cf_pp <- apply(betas, 1, function(b) calc_pps(b, cf_mat, cf$contest))
    delta <- cf_pp - pp
### Just get those comparisons we're interested in
    delta <- delta[which(cf$focal == 1), ]
### Average over cases in the data
    delta <- apply(delta, 2, mean)
### Return the distribution of average effects
    return(delta)
    
}

res <- calc_fx(m, dat, "BAME")

The return value of this function is a vector of n_sims average effect sizes. We can plot the distribution of values like this:

xlab <- str_wrap("Effect upon probability of success of making a random white candidate within each contest come instead from an ethnic minority")
ggplot(data.frame(value = res),
       aes(x = res)) +
    geom_histogram() +
    scale_x_continuous(xlab,
                       labels = scales::percent) +
    scale_y_continuous("Density") +
    geom_vline(xintercept = 0, linetype = 2) +
    geom_vline(aes(xintercept = mean(res)),
               colour = "darkred") + 
    theme_ipsum_rc()

We can see that across multiple simulated batches of coefficients, the average change in probability was -13.1 percentage points. That seems to me to be a pretty major penalty.

We can repeat this analysis for women candidates. The gender variable is coded so that values of one are equal to women.

res2 <- calc_fx(m, dat, "gender")

xlab <- str_wrap("Effect upon probability of success of making a random male candidate within each contest be female instead")
ggplot(data.frame(value = res2),
       aes(x = res2)) +
    geom_histogram() +
    scale_x_continuous(xlab,
                       labels = scales::percent) +
    scale_y_continuous("Density") +
    geom_vline(xintercept = 0, linetype = 2) +
    geom_vline(aes(xintercept = mean(res2)),
               colour = "darkred") + 
    theme_ipsum_rc()

`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

Thus, in addition to reporting the coefficient values in their article, the authors might have said:

“changing the gender of a randomly chosen male candidate in a contest to be female is associated with a change in the probability of selection of -9.5 percentage points (95 percent confidence interval: -18.5 to 0.3 percentage points”

and similarly

“making a randomly chosen white candidate in a contest come instead from an ethnic minority is associated with a change in the probability of selection of -13.1 percentage points (95 percent confidence interval: -24.6 to 0.1 percentage points”

both of which are smaller effects than the effects of being local, where they might have said:

res3 <- calc_fx(m, dat, "local")

“making a randomly chosen non-local candidate in a contest come instead from the local area is associated with a change in the probability of selection of 17.3 percentage points (95 percent confidence interval: 9.8 to 23.8 percentage points”

In UK candidate selection, the key really is “location, location, location”…

Why the king’s rooms run hot and cold

A long time ago, in a far away country, there lived two inventors.

These two inventors wished to measure heat.

One inventor measured heat by placing brandy in a thin column of glass.

As the level of the brandy rose, so the heat in the room was judged greater.

The other inventor did the same, but with quicksilver instead of brandy.

The devices made by these men sometimes agreed, but sometimes differed.

People in the mountains preferred the measure based on brandy; people in the plains preferred the measure based on quicksilver.

The two men brought their inventions to the king of that land.

They showed him their inventions, and asked him to judge which was best.

The king, wishing to offend neither man, said he could not decide, but offered to install devices from both men in the rooms in his palace.

Shortly after, the king’s steward noted disagreements between these devices.

The cellar to the west of the palace was judged by brandy to be the coldest room in the palace.

By quicksilver, though, it was the cellar to the east of the palace that was judged coldest.

The two inventors, hearing of the steward’s observations, called on the king again.

“Sire”, they said, “you said you could not decide between us on general grounds. Here you need only decide on particulars. Go into the west cellar, and then into the east cellar, and say which is coldest. Then shall you know which measure is best”.

The king went to the west cellar, and then to the east cellar, and then back to the two inventors. He told them that just as before, he could not judge which of the rooms was colder, and thus could not say which measure was better.

The two inventors thanked the king and left disappointed. The king too was disappointed, for he had no wish to spend any time in the cellars of his palace, and treated his steward rudely thereafter.

Time passed, and the king grew old, and took to complaining of the temperature in his rooms.

He asked his steward to make the temperature in his water closet more equal to the temperature in his bed chamber.

His steward, much put upon, declined.

“No sire, I shall not. For I mark the difference between the measure made in your bed-chamber, and the measure made in your water closet, and these two measures are closer together than the difference between the measures in the west cellar and the measures in the east cellar, which you thought so finely matched. If you tell me now there is a difference, then I must go back to those two men who troubled you earlier, and have them trouble you again”.

The king, not wishing to see the two inventors again, and knowing that his steward had the better of him, accept the man’s argument. And that is why the king’s rooms blow hot and cold.

Back to the social sciences

Though that fable was exceeding subtle, I hope you see some analogies to the social sciences. We have different measures of things, and if we really want to say that we’re indifferent between them, and indifferent to cases where they disagree, we have to say that some differences (in terms of one of those measures) aren’t important.

The fable simplifies things, because by construction there’s only one relevant case.

Now let’s start again from first principles, and only then deal with the complications posed by multiple cases.

Let’s suppose that there is some social scientist who is indifferent between two different measures of the same concept.

(This individual may have written articles which use one measure, and different articles which use the rival measure. Maybe they’ve even written articles which use both measures. After all, doesn’t everyone love robustness to different measurement strategies?)

Let’s suppose that these two measures are continuous. This means that we can always make a pairwise comparison between two cases and identify one case which has “more” of the underlying concept on this measure. Figuratively: the measures never put their hands up and say, “too close to call”.

Because our two measures are different, then with enough comparisons they will on occasion disagree. If they always agreed, they wouldn’t be different.¹ One measure will say that the example A in the comparison has “more”; the other measure will say that it’s example B that has more.

Let’s suppose that there is precisely one pairwise comparison where there is disagreement. This is, of course, the situation given in the fable above.

In this case, the social scientist either thinks

• “example A has more [of this concept]”, or
• they are indifferent between the example, or they think
• “example B has more of this concept”.

I would argue that they must be indifferent, because if they thought that either example A or example B had more of this concept, they would have a good reason to prefer the measure that they sided with, and couldn’t therefore be indifferent between the two measures as we initially supposed they were.

Now let’s consider the more typical situation where there are lots of cases of pairwise disagreement between measures.

We can’t take any specific instance and expect our social scientist to be indifferent. Maybe that instance of disagreement results from very particular circumstances. Maybe in that case the researcher thinks that one example has more, thus giving them a reason to prefer one index, but there are other offsetting cases which suggest the rival index.

What I’ll suggest is that our researcher has to be indifferent in the average or median case of disagreement. They have to be indifferent in the case of this case of disagreement because if they weren’t, it would strongly imply that they were not in fact indifferent between measures.

This argument is informal and not fully worked out. In theory if we allow no restrictions on the social scientist’s intuitive judgements, we could pair off freak results and find that the social scientist only has to be indifferent in cases involving the smallest disagreement.

A worked example with disproportionality

In this section, I’ll look at how we might establish a minimal important difference for votes-seats disproportionality, given indifference between measures. Votes-seats disproportionality is a good example because most of the time votes and seats are measured without error. As a result we can’t calculate a minimal important difference based on the minimal detectable change. I’ll assume that there is a social scientist who is indifferent between the Gallagher index and the Sainte-Laguë index, two commonly used indices of disproportionalit. I’ll calculate the values of these indices for modern parliamentary elections covered by ParlGov, and generate all pairs of elections.

suppressPackageStartupMessages(library(tidyverse))
dat <- read.csv("view_election.csv")
dat <- dat |>
    filter(election_type == "parliament") |>
    mutate(election_date = as.Date(election_date)) |>
    filter(election_date > as.Date("1945-12-31")) |>
    mutate(vote_share = vote_share / 100,
           seat_share = seats / seats_total)

gallagher <- function(v, s) {
    sqrt(1/2 * sum((v - s)^2, na.rm = TRUE))
}
sainte_lague <- function(v, s) {
    sum((v - s)^2 / v, na.rm = TRUE)
}
dat <- dat |>
    group_by(country_name_short, election_date, election_id) |>
    summarize(d_gall = gallagher(vote_share, seat_share),
              d_sl = sainte_lague (vote_share, seat_share),
              .groups = "drop")

### Generate all pairs
pairs <- expand.grid(A = unique(dat$election_id),
                     B = unique(dat$election_id)) |>
    filter(A < B) |>
    left_join(dat, by = join_by(A == election_id)) |>
    left_join(dat, by = join_by(B == election_id),
              suffix = c(".A", ".B"))

Having generated all pairs of elections, I focus on those pairs where there is disagreement concerning which example has “more” disproportionality. As might be expected, only a minority of pairings throw up some disagreement, but that minority, at roughly one in eight, isn’t neglible.

nrow(pairs)

[1] 212878

pairs <- pairs |>
    mutate(gall_pref = sign(d_gall.B - d_gall.A),
           sl_pref = sign(d_sl.B - d_sl.A)) |>
    filter(gall_pref != sl_pref)
nrow(pairs)

[1] 26250

Now let’s examine the absolute differences between values of the Sainte-Laguë index.

pairs <- pairs |>
    mutate(delta_sl = d_sl.B - d_sl.A)

summary(abs(pairs$delta_sl))

    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.000000 0.005039 0.013049 0.022868 0.028760 0.599469 

In this case, we wouldn’t want to force anyone to say that a difference in the Sainte-Laguë index of 0.59 units(!) was not important just because it arose in the context of a disagreement between indices. But it does seem plausible to suggest that the median absolute value of 0.013 units might be a minimal important difference. For what it’s worth, if we compare that value to the standard deviation of values of the index in our original data, we have a minimal important difference of roughly 0.125 standard deviations.

We could, of course, have started from the other index. Let’s examine the absolute difference between values of the Gallagher index.

pairs <- pairs |>
    mutate(delta_gall = d_gall.B - d_gall.A)

summary(abs(pairs$delta_gall))

     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
1.600e-07 3.848e-03 9.325e-03 1.339e-02 1.863e-02 1.091e-01 

In this case, we get a pretty small median absolute difference of just under 0.01 units – but of course the standard deviation of the Gallagher index is much smaller, at 0.048 units. Expressed in terms of standard deviations, we have a minimal important difference of roughly 0.2 standard deviations.

We therefore have a minimal important difference for one index, and a minimal important difference for another index. This means that some differences will be important when we use one index, and not important when we use another index. That might seem unsatisfactory, but “disagreement over indices” should surely have consequences for what is substantively important.

Back to measures of democracy

We can apply this same logic to democracy. Let’s suppose there is a researcher who is indifferent between two measures of electoral democracy:

• the V-Dem project’s measure v2x_polyarchy
• Polity V scores

Thankfully these two measures are both available in the V-Dem data.

library(vdemdata)
data("vdem")
### Just select the variables we're interested in
dat <- vdem |>
    dplyr::select(country_text_id, year,
                  v2x_polyarchy,
                  e_polity2)
### Fix some Polity codes
dat <- dat |>
    mutate(e_polity2 = na_if(e_polity2, -88),
           e_polity2 = na_if(e_polity2, -66))

Generating all possible pairwise comparisons of all country years across all of the modern period is memory-intensive, so I’ll focus on years since 1945 and countries covered by both projects.

### Restrict it to the post-war period
dat <- dat |>
    filter(year >= 1945)

### Restrict it to common cases
dat <- dat |>
    filter(!is.na(e_polity2)) |>
    filter(!is.na(v2x_polyarchy))

### Create a unique label
dat <- dat |>
    mutate(label = paste0(country_text_id, year))

### Generate all pairwise combinations of country years.
### Note that there are around 4,600 country years
### so we have 4,600 * (4,600 - 1) pairings
### but we can restrict it to cases order is alphabetical
### taking it down to 10 million
pairs <- expand.grid(A = unique(dat$label),
                     B = unique(dat$label),
                     stringsAsFactors = FALSE) |>
    filter(A < B)

### Start merging on the indicators
pairs <- left_join(pairs,
                   dat |> dplyr::select(label, v2x_polyarchy, e_polity2),
                   by = join_by(A == label))

pairs <- left_join(pairs,
                   dat |> dplyr::select(label, v2x_polyarchy, e_polity2),
                   by = join_by(B == label),
                   suffix = c(".A", ".B"))

As before, we focus just on cases of disagreement. Here, I’ll look at cases of “strict” disagreement.

nrow(pairs)

[1] 48960460

pairs <- pairs |>
    mutate(vdem_pref = sign(v2x_polyarchy.B - v2x_polyarchy.A),
           polity_pref = sign(e_polity2.B - e_polity2.A)) |>
    filter((vdem_pref == 1 & polity_pref == -1) |
           (vdem_pref == -1 & polity_pref == 1))
nrow(pairs)

[1] 4961321

Here, around 10% of pairings of country years see a disagreement between the two indices. Let’s work out what the median absolute difference in these cases of disagreement is.

First, let’s do v2x_polyarchy:

pairs <- pairs |>
    mutate(delta_vdem = v2x_polyarchy.B - v2x_polyarchy.A)

summary(abs(pairs$delta_vdem))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.00100 0.02800 0.06500 0.08811 0.11900 0.78000 

Our candidate for the minimal important difference is of 0.065 units. That’s around 0.25 standard deviations when looking at the V-Dem data as a whole, and bigger than 0.05, the minimal important difference implied by the minimal detectable change.

and now e_polity2:

pairs <- pairs |>
    mutate(delta_polity = e_polity2.B - e_polity2.A)

summary(abs(pairs$delta_polity))

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.000   1.000   2.000   3.295   4.000  20.000 

Because the Polity score is a fine-grained ordinal variable rather than a continuous variable, the median absolute difference in cases of disagreement is exactly two units.

Conclusions

In this post I’ve suggested that if you are indifferent between two measures, this can help us establish a minimal important difference. This way of establishing a minimal important difference is valuable in cases where, like electoral disproportionality, we can’t establish a minimal important difference by way of measurement error.

I don’t know how common it is for researchers to be genuinely indifferent between measures. I also don’t know how persistent that indifference is. Indeed, one possible way of reacting to these claims about minimal important differences is to quickly develop a more fine grained preference between measures based on cases of disagreement. Obviously it’s fanciful to imagine anyone going through 4 million pairs of country years and determining whether they side more with Polity or V-Dem, but sharpening our preferences by looking at cases of disagreement might still be a worthwhile way to spend some time.

The minimal detectable change

Measurement – at least for continuous quantities – is never exact. This is true for ordinary observable quantities as well as for the social scientist’s latent constructs. In the recent summer Olympics, times in swimming competitions were recorded to the hundredth of a second, rather than to the thousandth of a second.

The reason is rather prosaic: although it’s possible to record elapsed time to the thousandth of a second, it’s not possible to construct swimming pools with tolerances of less than a millimetre, and so any difference of less than a hundredth of a second may be due to minute differences in the length of the lanes.

There is therefore some sense in which a difference of less than a hundredth of a second is not important. If it were important, we’d start constructing swimming pools the same way we construct particle accelerators.

Social scientists tend not to talk about tolerances, but about the standard error of measurement. This standard error of measurement comes up in different guises:

• Maybe you have a psychometric test estimated under the assumptions of classical test theory, and you have a known standard error of measurement.
• Maybe you have some rates of intercoder reliability and have converted a reliability statistic to a standard error of measurement.
• Maybe you have a Bayesian measurement model and approximate the standard error of measurement by the posterior standard deviation of some parameter.

If you are in the first two categories; you have a single standard error of measurement; if you are in the last category; you have as many standard errors of measurement as you have measured parameters. You will therefore want to replace references to the standard error of measurement with the smallest standard error of measurement.

If you have a standard error of measurement, you can work out the minimum detectable change, or “the amount of change needed to exceed measurement error for a specific measure based on a predetermined confidence threshold”. We may suppose that any two measures we care to compare are draws from independent normal distributions which have different means (the true scores we want to estimate but can never know) but identical standard deviations (the standard error of measurement). The difference between two normal distributions is distributed normally with standard deviation equal to the square root of the sum of the two variances. If we want to make claims about differences at the 95% level of significance,

where SEM is the standard error of measurement. If you’re a Bayesian, and if you have access to the posterior distributions needed, you might want to ignore this useful equation and calculate the probability that a difference is greater than zero directly.

The MDC for electoral democracy

Let’s use as an example the minimal detectable change for electoral democracy. For this I’ll use data from the V-Dem project, and values of their variable v2x_polyarchy for the post-war period. I’ll also use the posterior standard deviation (v2x_polyarchy_sd) as an equivalent of the standard error of measurement.

library(vdemdata)
data("vdem")
vdem <- vdem |>
    subset(year >= 1945,
           select = c(country_text_id, year, v2x_polyarchy, v2x_polyarchy_sd)) |>
    subset(!is.na(v2x_polyarchy))

smallest_posterior_sd <- min(vdem$v2x_polyarchy_sd)

Specifically, I’ll use the smallest posterior standard deviation. In part because of the way the V-Dem scores are transformed to the range [0-1], the standard deviations are smallest at low values. The V-Dem project as a whole is most confident about the levels of electoral democracy in Eritrea in 2007. (Spoiler: they’re pretty confident there wasn’t much). The posterior standard deviation for this country year is 0.002 units on a 0-1 scale.

If we multiply this quantity by 1.96 and againt by the square root of two, we get a figure of 0.0055 units. To put it on a different metric, that’s roughly 0.02 standard deviations.

This MDC gives a rationale for only reporting V-Dem figures to two decimal places, because any information in that third decimal place is smaller than the minimum detectable change.

It is possible to give a different minimum detectable change depending on the research question. This alternative figure can be larger or smaller.

conditional_posterior_sd <- vdem |>
    subset(v2x_polyarchy > 0.4)

conditional_posterior_sd <- conditional_posterior_sd$v2x_polyarchy_sd |>
    min()

Here’s an example of a larger MDC. Perhaps you are interested in democratic back-sliding, and want to focus on countries which score above 0.4 on this 0-1 scale. Recall that the V-Dem project tends to be more confident about low scores on their democracy measures. When we restrict ourselves to cases which score above 0.4, the smallest posterior standard deviation is 0.019. This in turn means that the MDC is around 0.05 units. This is an order of magnitude greater than what we had before. For comparison: levels of electoral democracy in the UK since 2000 have never ranged by more than 0.04 units (from 0.84 to 0.88).

It’s also possible to argue for a smaller MDC by shifting the unit of analysis. Suppose that we were not interested in country-years, but in regional averages. For independent measures, the standard error of measurement decreases with the square root of the number of observations. If our regions have thirty-six countries in them, the MDC is smaller by a factor of 6. If we really formed our ideas of what was detectable (and, a fortiori, important) by thinking first about regional averages, this might be a good MDC to use. I tend not to think “region-first”, but maybe you do.

Is the MDC for democracy a MID?

I don’t think anything in the previous section is particularly controversial. It’s as good a guide to the MDC as you can get without having access to the full posterior distribution for the V-Dem variable in question.³

What is more controversial is the idea that the minimum detectable change should set a lower limit on the minimal important difference.

There is a good pragmatic argument, which says that for any putative change in one direction, you should not hold it out to be important if later more accurate measurement would show that it was in fact a change in the opposite direction.

This pragmatic argument, however, only works in the context of the measures we have. It is perfectly consistent to say that the minimal important difference is in fact smaller than the minimum detectable change, and for that reason we should improve our measurement techniques.

I think the strongest context for this argument comes from conflict deaths. One of my colleagues at Royal Holloway works a lot on the measurement of conflict deaths. His paper on survey measurement of conflict deaths in Iraq shows that estimates of the number of conflict deaths in Iraq differ by one and a half orders of magnitude, from 26,000 to 1.0 million. If the differences between estimates represented the standard error of measurement rather than poor research conduct, we would be placed in a position where the minimum detectable change involved several thousand deaths. That seems wrong. Maybe we should just get better at measuring conflict casualties – a sort of “git gud” approach to substantive significance.

The pragmatic argument does, however, seem to work for measures of electoral democracy. It would in principle to be possible to improve the V-Dem project by involving more experts per country, and to increase its scope by extending the analysis not just forward (at a rate of one year per year) but backwards. The increase in the number of country experts would improve the accuracy; the increase in the number of country-years can only make the smallest posterior standard deviation smaller.

In practice, however, the V-Dem project is one of the best funded social science projects in history, and the project has literally unrivalled temporal and geographic coverage. In what follows, I’ll look at other big projects which have near complete coverage of the units of analysis.

MDCs for other variables

mq <- read.csv("http://mqscores.wustl.edu/media/2022/justices.csv")

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.5
✔ forcats   1.0.0     ✔ stringr   1.5.1
✔ ggplot2   3.5.2     ✔ tibble    3.2.1
✔ lubridate 1.9.4     ✔ tidyr     1.3.1
✔ purrr     1.0.4     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(IRRsim)
set.seed(26943)
tmp <- IRRsim::simulateRatingMatrix(nLevels = 3, 
                             k = 100, 
                             k_per_event = 100, 
                             agree = 0.93, 
                             nEvents = 100) |>
    as.data.frame() |>
    mutate(across(everything(), function(x) as.numeric(x > 1)))
sem <- sd(colMeans(tmp))
mdc <- qnorm(.975) * sqrt(2) * sem

Conclusions

In this post I’ve suggested one way that we can established minimal important differences for outcomes in the social sciences, at least for those outcomes which are measured on a continuous scale with some reported standard error of measurement (or something analogous).

I’ve applied this to four measures – levels of democracy, left-right positions of political parties, left-right positions of judges on SCOTUS, and rates of pledge fulfillment.

I’ve noted that the MDC depends on how the unit of analysis is characterised. Some measures – like seat-share weighted cabinet positions – are measured on the same scale as constituent units, but we can be more precise when dealing with aggregates. The situation would be more complicated with measurements which depend on two noisy inputs.

In the next blog, I’ll look at how we might instead establish minimal important differences in the case where we are indifferent between two rival measures. This can be applied to cases where both measures are measured ostensibly without error, and generally gives larger MIDs.

Somalia 1969

Between independence in 1960 and Siad Barre’s coup in 1969, Somalia held two elections which were genuine multiparty contests, and which might have been free and fair if you don’t look too closely at some of the reported vote totals. In both of these elections, the Somali Youth League won a majority of seats. In the 1964 election, it won a majority of seats with a majority of votes. Five years later, it won a 73/127 = 57% of seats on a third of the vote (33.24%). It was only able to do this due to the remarkable fragmentation of the vote: per Wikipedia, 26 parties won election to the legislature.

Norway 1915

The 1915 election was the penultimate Norwegian election held under first-past-the-post, and looking at the result it’s not hard to see why. The Liberal party won 74/123 = 60% of seats on a third of the vote (33.07%). Following the election

“all parties acknowledged that the situation was such that this could no longer continue, and a better electoral rule should be found, a more just electoral rule” (Gunnard Knudsen, quoted in Cox, Fiva and Smith )

In fact it would take the perverse results of the 1918 election, where Labour came first in terms of votes but third in terms of seats, before electoral reform could be approved.

Grenada 1995

I don’t know a lot about Grenada. I would have expected that a small micro-state with an assembly size of just 15 would have a two-party system. But here comes the New National Party, winning 8/15 = 53% of seats on 32.4% of the vote, with the two remaining parties splitting most of the remainder of the vote between them, whilst still leaving space for the “Good Old Democratic Party”, who surely must be the least successful political party to have its own Wikipedia page.

Honourable mentions

The Fijian election of 1999 would have made the list, but for the use of the Alternative Vote in constituencies of two types, meaning that the vote share of the Fiji Labour party (33.26%) is based on first preferences in the “open constituencies” rather than communal constituencies.

The South African election of 1910 would have made the list if I were included contested elections in non-democratic systems; if it were included, it would be at the top of the list, with Botha’s National Party winning a majority despite winning fewer votes than the Unionist Party (28.5% versus 37.7%).

Historical UK and comparative reference points

One reference point is to look at past historical elections where there was a large gap between the first and second-placed, and first and third placed parties, and see what share of seats the first-placed party won. I focus on the gap between first and second because under first past the post it’s not enough to do well in an absolute sense: you also have to be ahead of your opponent. A vote share of 48% can win you almost all of the seats if your nearest opponent is on 24%; it can lose you almost all of the seats if your nearest opponent is on 52%.

In post-war UK electoral history, the two elections with a big lead are the elections of 1997 and 1983. In 1997, Labour polled 12.8 percentage points ahead of the Conservatives in Great Britain, and won 418 seats, albeit in a slightly larger House of Commons of 659. At present, Labour’s share of the vote is lower than it was in 1997, but its lead over the Conservative party is greater. If the Labour lead is greater than 1997, then maybe the Labour seat count will be bigger too.

In 1983, the Conservatives polled almost sixteen percentage points ahead of Labour in Great Britain, and won 397 seats. At present, Labour’s share of the vote is lower than the Conservative share was in 1983, but their lead is once again greater. This example is less favourable to Labour, in that it implies that you can rack up a sixteen point lead and still not break four hundred, but sixteen points is less than what we see at present.

More generally we can look at all post-war democratic elections and examine those elections which were held under first-past-the-post, and plot the gap to second place against the leading party’s seat share.

I’ve plotted that using data from ElectionsGlobal combined with information from Democratic Electoral Systems in order to restrict my analysis to elections held under first past the post. I’ve also added a polynomial trend line to show what we might expect at different leads, if we knew nothing other than the gap between the first- and second-placed party. The trend line fits the two UK elections just mentioned quite well.

The suggestion from the trend line is that if you have a lead of around 19%, you might expect to win 68% of seats, or (0.68 * 650) = 442 seats.

Comparative evidence therefore suggests that if you knew nothing about the election except the Labour lead over the Conservatives and the fact that the election is being contested under first-past-the-post, then you would predict a very large seat share. This comparative evidence is in line with two UK historical examples, which would predict that the Labour seat share in 2024 should be greater than the Labour seat share in 1997, or the Conservative seat share in 1983, because the Labour lead is bigger than either party’s lead was in those two elections.

Uniform and proportional swing

At this point it might reasonably be objected that we do in fact know more than simply the Labour lead over the Conservatives and the fact that the election is being held under first-past-the-post. We know past support for each party in each seat from notional election results. This can help us predict seat counts under an assumption of uniform national swing, such that if Labour is up so many percentage points, we can add that to their performance last time, and by repeating this for all parties, calculate who is in the lead in each seat.

John Curtice has helpfully calculated the seat count under uniform national swing using an average of vote shares from different MRP models. Under UNS, you get a Labour total of 370, with the SNP on 34. This calculation precedes the most recent Survation update.

Uniform national swing does have a tremendous record in predicting seat outcomes. It is very useful, but it does have some limitations. The main limitation is that it can create negative values where a party’s national vote share goes down by more than it previously won in a seat. (The converse, where adding an increase to a party’s national share would take its local share to greater than 100%, is rarely if ever seen). Since parties cannot win negative votes, then any national change which implies negative votes in some places means that the change must be less than uniform (i.e., less steep) in some places, and correspondingly more than uniform (i.e., steeper) in some other places. Sometimes the label used more this type of swing is proportional, since the amount of swing is proportional to how well the party did previously.

Usually this limitation of uniform national swing is not a serious limitation, and is very rarely a serious problem for one of the two largest parties. However, the likely change in the Conservative vote is so large than applying uniform national swing creates some problems. If we suppose a change in the Conservative vote share of twenty points, then there are almost ninety seats where uniform national swing would create negative Conservative vote share. If we sum up the negative votes predicted, that’s almost 350,000 extra votes which have to come from other seats, or roughly one percent of the national vote share. This means that the Conservatives would need to lose one percentage point more than their nation-wide loss in all the other seats just to make the sums add up. If we suppose that the Conservative vote never reaches exactly zero, but falls to one or two percent, then the figures become even greater.

Uniform national swing will therefore be placed under severe pressure in this election, and there is no golden rule saying that when UNS is under pressure, it will emerge triumphant. Steve Fisher and Jake Dibden have helpfully analysed cases where parties have suffered large national swings, and examined the proportionality of their swing in these cases. The conclusion is that proportional swing can well happen when there are large national swings.

What I take from these two analyses is that uniform national swing sets a plausible lower bound on the Labour seat tally, and that we should therefore expect a figure greater than 370. “How much greater” is, of course, a matter of further analysis, and the point of MRP modelling is to take the analysis to the level of the voter and make post-stratified predictions. Our election models in 2017 and 2019, conducted for private clients in the financial sector, were reasonably accurate in terms of the headline seat counts, and for that reason we opted to retain the model we had used for 2019, without making any adjustments to account for attenuation.

Getting the headline figures wrong

The discussion of comparative reference points and models of swing suggest that even if our mapping between national vote share and seat level outcomes is wrong in the specific sense of being too favourable to Labour, then we would still expect a very big Labour figure based on Labour’s lead over the Conservative party and its change in the share of the vote since last time.

The second failure mode of MRP models is that they get the national shares wrong. Because the national shares are just aggregates of what we predict at local level, if we get the national shares wrong it means we’ve also gotten the local results wrong.

We might get it wrong because MRP relies on polling, and because the errors in modern polling are not the kinds of errors that result from simple random sampling. Opinion pollsters rely on people who opt-in to internet panels or who pick up the phone to queries from polling companies. These people are different from the general population. Although we can adjust for some of these differences – and in particular, we adjust for age, sex, highest level of qualifications, past vote, and constituency – we cannot adjust for characteristics for which we have no national reference point, like “political interest” or “agreeableness”, neither of which feature on the Census (though maybe they should).

It is therefore possible that although the samples used for MRP are very large, they could be very wrong. If there is a polling miss of the same direction and magnitude as there was in 1992 or 2015, our seat forecasts would be very wrong. Most big polling misses are collective misses, but here there is a canary in the mine: polls like the most recent poll from Verian, a company which I greatly respect, suggest a very different picture, with a 15 point Labour lead over the Conservatives (36 over 21).

The question to ask, therefore, is: could the polls be wrong? This question was extensively discussed at a recent British Polling Council event on the 5th June. The general conclusion from that event was that there was very little evidence from Westminster by-elections or local elections to suggest that they are very wrong. In particular, I took heart from one random probability sample by Verian, which at the time was giving very similar national vote shares to our MRP model at the time. Random probability samples, which involve drawing people at random from a list and repeatedly going back to them until they answer, are an expensive gold standard.

My view is therefore that whilst error in the overall vote share is the single greatest point of failure in our model, there is not much evidence to suggest that failure is likely, or more likely than normal. The pattern of failure would have to be quite specific, and (given the random probability sample I mentioned earlier) would likely have to have emerged during the course of the election, either because of opinion change or because of a compositional change which emerges when you weight by turnout likelihood.

Vibes check

One source of information which I think can be discarded are reports by Labour canvassers who say that they are not detecting the same kinds of enthusiasm which they detected in 1997, and that therefore Labour cannot be on course for a similarly large victory. “Enthusiasm” in some absolute sense does not matter in elections. Our MRP model is not forecasting that Labour will win more votes than they did in 1997, and might for that reason be more popular or the subject of greater enthusiasm. Indeed, we’re only just confident that they’ll poll a greater vote share than they did in 2017. What matters is not enthusiasm for Labour, but a general weariness of the Conservatives. That sentiment does not seem to be in short supply.

Conclusions

This is the kind of forecast that you want if you don’t want to see herding: people who are following an existing model, who trust in the process, and don’t make adjustments to reflect an external view of what seems most likely. Unfortunately it is also the kind of forecast that means that on Friday morning there is a chance that we will have egg on our faces. Right now I am dwelling more on that chance that the chance we will look like seers.

What did the parties talk most about?

Our first pass at the data has to involve the different Manifesto Project categories themselves. If we can’t understand the distribution of text across topic codes, we can’t understand anything built on those codes.

With that in mind, here’s the proportion of sentences (or things recognized as sentences by the tidytext package) belonging to the top ten topic codes, in the Labour manifesto.

The top entry – welfare state expansion – is perhaps unsurprising to see in the manifesto of a nominally left-wing party. What might be more surprising are that “law and order: positive” and “political authority” are second and third. These two codes are often taken to be (culturally or generally) “right-wing” codes. Let’s see some sample sentences coded as “law and order: positive” (more accurately: sentences for which “law and order: positive” had the highest probability of membership):

Labour will restore neighbourhood policing with thousands of extra officers, and we will equip officers with the powers they need.
 - Labour will end the practice of empty warnings by ensuring knife carrying triggers rapid intervention and tough consequences.
 - Labour will stop the chaos that lets too many criminals act with impunity, turn the page with stronger policing, and rebuild our criminal justice system.
 - We aim to halve knife crime in a decade.
 - We will make it easier for high- performing police forces to charge domestic abuse suspects to speed up the process.
 - Labour will introduce a new Neighbourhood Policing Guarantee, restoring patrols to our town centres by recruiting thousands of new police officers, police and community support officers, and special constables.

Now let’s see some sentences which are coded as “political authority”:

I am absolutely convinced that Keir Starmer is the leader to deliver the change this country needs.
 - We switched and voted Labour at the local elections.
 - Only Labour can turn the page.
 - Because whether it is crashing the pound to give tax cuts to the richest 1%; degrading public services because of a mess made by the banks; or the failure to invest in clean British energy that left us exposed when Putin invaded Ukraine – so much of what Britain has been through in the past 14 years is explained by a Conservative failure to face the future.
 - In contrast, Labour has been transformed from a party of protest to one that always puts the interests of the country first.
 - I have changed my party.

These sentences seem to be divided between sentences positioning Labour as a party which has authority, and sentences criticising the Conservatives for eroding the authority of the state.

If Labour wrote most about welfare state expansion, law and order, and political authority, what did the Conservatives write about?

The table is surprisingly similar. Seven issues appear in both top ten lists. Just to show you that the classifier is not a stopped clock which always spits out the same probabilities of membership in these categories, here’s the top ten list for the Green Party of England and Wales:

It is reassuring to see environmental protection as the most common code, and surprising (but not for that less accurate) to see anti-growth or degrowth feature in there.

How left- or right-wing are they?

Learning about topic codes is important, but part of the reason we come up with lists of codes is in order to reduce complexity. One way of reducing complexity is to code topics as either “left” or “right-wing” based on a salience theory of communication, where parties concerned with equality talk about things they want to use to address inequality (market regulation, nationalisation), rather than engaging in a detailed discussion of why unfettered markets go wrong. That is, parties follow the maxim: “say what you’re for, rather than what you’re against”.

The Manifesto Project’s RILE list of left- and right-wing codes has the following topics as left-wing:

103 - Anti-Imperialism
 - 105 - Military: Negative
 - 106 - Peace
 - 107 - Internationalism: Positive
 - 202 - Democracy
 - 403 - Market Regulation
 - 404 - Economic Planning
 - 406 - Protectionism: Positive
 - 412 - Controlled Economy
 - 413 - Nationalisation
 - 504 - Welfare State Expansion
 - 506 - Education Expansion
 - 701 - Labour Groups: Positive

and the following topics as right-wing:

104 - Military: Positive
 - 201 - Freedom and Human Rights
 - 203 - Constitutionalism: Positive
 - 305 - Political Authority
 - 401 - Free Market Economy
 - 402 - Incentives
 - 407 - Protectionism: Negative
 - 414 - Economic Orthodoxy
 - 505 - Welfare State Limitation
 - 601 - National Way of Life: Positive
 - 603 - Traditional Morality: Positive
 - 605 - Law and Order: Positive
 - 606 - Civic Mindedness: Positive

The Manifesto Project constructs a measure, RILE, which is the proportion of right-coded sentences, minus the proportion of left-coded sentences. How do the parties line up when we follow this procedure?

This ordering makes some sense. Reform is to the right of the Conservative party, which in turn is to the right of the Labour Party. The Green Party, together with the Scottish National Party and Plaid Cymru, are to the left of both the Labour Party and the Liberal Democrats. There may, however, be some questions about the relative positioning of Labour and the Liberal Democrats: whilst the party has been very clearly to the left of Labour on some occasions (most notably under Charlie Kennedy’s leadership), the Liberal Democrats are still(!) led by a contributor to the Orange Book.

How have the parties changed since 2019?

One natural question is: to what extent have the parties’ positions changed since 2019? Here I rely on estimates produced by my colleague Nick Allen, together with Judith Bara, with coding by Michelle Springfield. The following plot shows the parties’ 2024 scores on the horizontal axis, against their 2019 scores. The dotted line shows a 1:1 correspondence.

Five of six plotted points land close to the dotted line, with Labour the big exception. The fact that Labour is in the lower right quadrant means that the party has moved substantially to the right.

This finding is somewhat sensitive to the coding decisions made. Here’s a similar plot, but plotting both the hand-coded 2019 scores and an automated read of the 2019 Labour and Conservative manifestos, using the same process I have used above for 2024.

The automated reading of the 2019 manifestos would have Labour roughly as left-wing as the other left-wing parties, rather than being out to their flank, and the Conservatives more left-wing than a manual reading of the 2019 manifesto suggests. An apples-to-apples, LLM-to-LLM comparison would therefore suggest that both parties have moved to the right: the Conservatives a little bit, Labour a lot.

Conclusions

As you might expect from my other work, I believe that large language models can help researchers code large volumes of text quickly and reliably. In writing this blog post, I’ve experienced more unease defending the RILE schema than I have defending the coding decisions made by the large language model. Any systematic process will, when applied to enough data, inevitably generate one or two decisions that seem implausible, but I think that for existing schema with large volumes of training data, LLMs reduce that number to the bare minimum.