The Worthlessness of Psychological Testing

So, the Myers-Briggs psychological profile test is worthless.  You’ve probably encountered this before: take a bunch of profiling questions and it’ll spit out whether you’re Introverted/Extroverted, Intuitive/Sensing, Thinking/Feeling, and Perceiving/Judging (16 total profiles).  It’s plagued by measurement issues, including the fact that the constituent categories are based on nothing at all and the fact that measurement error is remarkably high – as many as 50% of people get different results when they take the test multiple times.  Furthermore, and this is the real kicker, it has zero predictive power in predicting people’s happiness, situational comfort, job success, or any other tangible outcomes.

So measuring latent variables is hard, and it’s actually particularly difficult in psychology.  A latent variable is one that can’t be observed, but can be inferred from other things.  A simple example is generosity – you can’t exactly measure how charitable a person or society is in their heart of hearts, but you can measure how much they volunteer or donate to charity and use that to make inferences about their level of generosity.   Measuring these latent variables is a key social science problem, and one on which people spend a great deal of time. 

There are two reason why psych latent variable measurement is particularly tricky – we don’t really know what the variables mean, and we don’t really know what the key variables are.  The first is simply that it’s difficult to cleanly define introversion/extroversion in a way that doesn’t rely heavily on pre-existing notions that emerge from…where?  Probably from pre-existing notions, which is indeed where Jung derived his categories.  This is troublesome, because it means we’re to some degree testing for things defined however we want and introduces a degree of circular logic.  The second concern is more diffuse – how do we know that introversion/extroversion is a key component of personality?  How specifically do we know that it’s more important than, say, general degree of anxiety or like/dislike of peanut butter?  It may seem more important, but…um…why?  Even if Jung’s four axes were scientifically derived and correctly measured, there’s no obvious reason to believe these four axes are the central components of personality.

The problems of deriving measurements for psychology suggest that it might be a better fit for different techniques.  Psych testing is a classic example of “supervised learning” – we define outcomes, see how people match up to them, and use that info to derive predictions about how new people will match up to them.  That in turn drives the test.  But the problems with that are large, as I detailed above – unsupervised learning might be a better fit.  This would include techniques such as clustering, wherein you give people a bunch of questions and use an algorithm to see whether there are natural lines of division in the data rather than specify beforehand what questions are important.  That in turn would allow you to infer what exactly are the crucial components of personality – though it wouldn’t necessarily help with defining what exactly it is you’re measuring, it is a clear step forward.

A lot of social science problems are not obviously well-suited for unsupervised learning, but this one seems to be.

Tags: , , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: