esoteriic.com

The parent thread for this topic is here: Political Compass: the road towards an invariant expression of opinion

The political compass is something that I genuinely like as an idea but feel it could do with being far more rigorous. I previously wrote a an article which is a precursor for this one back in 2009, it was something of a teaser and was supposed to be followed up soon after. Unfortunately, I spent years debating with myself about how to present the ideas and also finding the courage to share them. I've told you that many times now and I'm likely to state it again. So in this thread I will do my best to enumerate the major flaws I've found although I doubt the list will ever be exhaustive. I'll start with some of the obvious ones which are more surface level considerations (not necessarily flaws), the deeper structural concerns are more important but less obvious. Also, as I've said before, this analysis can be applied to personality tests or the various other tests you are bombarded with on facebook.

Questions

The relevancy of the questions to what you are trying to probe is one of the first things you notice with the political compass. Some questions feel like they are misleading you, or that you have to answer in a particular way to get the result you desire. Some of the questions conflated different concepts that should be measured in separate questions rather than in the same question. Essentially the answerer will be trying to guess what the questioner is asking and may make a wrong assumption and hence lead to an incorrect answer. An incorrect answer being one that does not faithfully represent a person's views.

eg. "A genuine free market requires restrictions on the ability of predator multinationals to create monopolies."

Naturally, people won't agree on what defines a free market. Some will argue that the existence of a monopoly is just part of what a free market allows. While others will disagree. That's fine. However, this question (really a statement) implicitly assumes that you accept the notion of free markets being good; however, those that don't support free markets will have to answer the question based upon what they think a free market is defined to be, not whether they support such a concept. Bob despises free markets but believes the textbook definition of a free market to be one where monopolies don't exist. This is a problem as the answer will affect Bob's position on the political compass.

There are other questions that potentially lead to problems where belief / definitions and interpretation are muddled up. Getting this part right is very important although admittedly not easy.

Furthmore, the categories that the questions are pulled from must be relevant to what you are trying to measure. In the case of the political compass I would say that they get this right. They are measuring people's opinions on liberty: i.e. do you want more or less civil liberties? Although never quite so bluntly stated this is an example of what they are trying to achieve. Providing 2 dimensions is a vast improvement over the usual 1 dimensional vernacular representation. That said the question on astrology seemed quite far out of left field, is based on a correlation they have found between civil liberties and belief in astrology? It doesn't seem relevant.

Answers

Almost as important as the questions. The answers are formed in one's head as they read the question, as already pointed out the answerer may try to answer the questions by guessing what the questioner is means. Although there is more to it than that. The political compass provides 4 outcomes: you can agree/ disagree, or strongly agree/disagree. There is no neutral answer option. I think that is an oversight and should be included. If a question is not understood or the answerer is somewhat ignorant on the topic of discussion then they can't state a preference of agree or disagree. Therefore, the answer they provide is literally wrong.

Building on from that suggestion it could also be wise to provide more options for some questions: e.g. strongly agree, mostly agree, partly agree, neutral, etc. Ultimately it should improve the accuracy of representing people's opinions on a graph. If we return to the problem of beliefs/ definitions then we can see a further problem in that you might agree with a statement if the definition is X but may disagree if the intended definition is Y. This means that people are undecided and the chance of them putting the correct answer is (naively) 50%. Moreover, the current system of questions rules out the possibility of the following type of beliefs: I agree that X is fine if it applies to me but I disagree with X if it applies to other people. It might be hypocritical but it is still a popular point of view.

Positioning

Now here comes an interesting though: is strongly agree +2 points or +3 points? Or perhaps it should be +5, while agree is merely +1. How should the answers be matched with the numerical answers and hence the position on the final graph? Does each question hold the same weight (ie all agree statements = +1 point of movement (or, -1), or should they hold different weights? It isn't entirely obvious how this is done with the political compass. Can a single question affect both of axes? Ie a agreeing with a statement gives +1 on the x-axis and +1 on the y-axis.

It is natural to allow questions to vary in terms of weight, some are more important than others, it also makes sense to allow questions to affect more than one axis. This then leads to a question of calibration, how can we be sure that the questions and answers fairly match up and provide a faithful representation of the person answering the questions? One simple solution to this would be to suggest that thousands of people should answer the questions and see if there unexpected correlations (or hotspots). It could be hard to determine if a correlation truly exists or if the correlation is forced because of the nature of the questions. Re-asking the same question in many different ways might straighten out this issue but I'm naturally cautious.

A very subtle point is that the positioning is defining. What I mean here is that while a +1 movement in the x-direction seems fair when you are constructing the first question, you won't be able to appreciate what the maximum possible x-value is until all the questions are written and assigned values. This may lead to a re-calibration of the movements assigned to the questions. It isn't obvious if the political compass creators have noticed this and thought about how to deal with it.

This also leads to a problem of defining the edges of the graph. The further away reachable point is determined by the questions asked; this should probably be symmetric about the origin and for both (all) axes. This means that there are less possible paths you can take to get to a particular edge of the graph. This partly makes sense from a human perspective in that centrists have a less well defined political stance while "hardcore" supporters / activists (or whatever) will have a very defined and focussed set of beliefs (hence less paths to get to that edge). This is an important paragraph that highlights an idea that I will return to many times, it is one of the core features (problems) of such a system of representation.

Error bars

It would be useful to see an estimate of the error associated with each position. As the position is subject to many possible flaws it would be wise to attach some error to the final result. This is an important part of science and is no less important here.

As there are less paths to the outer edges of the graph so the error should be lower there. Conversely, people at the centre of the graph will have a higher error attached to their position as there are many more possible paths that can lead a person to that position.

Axes

What axes should be used? Obviously this depends on what you are hoping to measure. I think the political compass here equates (forces) one's views of civil liberties to be the same as that of political liberties. Which is not necessarily true.