Data-Driven Discrimination is Coming Whether We Like It or Not
The availability of data makes discrimination easier. The example the authors start with, of bus schedules, is a pretty trivial example, but it gets real pretty quickly. Health insurance is the most famous example of this, as it is a business governed solely by adverse selection. The ideal business model for health insures is to provide exactly zero healthcare reimbursement, and they would achieve this by denying coverage to anyone who might need it. This type of concern is gradually permeating most of the spheres characterized by information asymmetry – those where individuals are dealing with businesses or individuals who would like to know more about them than the individuals would like to put forward. There’s a spectrum of reasonableness here – at the top is a criminal record, which is why employers conduct background checks. At the bottom is political or religious beliefs, things that are immaterial to almost any purpose other than discrimination. But much of what’s in between is characterized by a fair amount of dispute over what might be considered a reasonable subject of interest, and that ambiguity should concern those of us interested in preventing unfair discrimination.
Some of these categories might well have valid applications. For example, let us imagine a world where the vast majority of white-collar criminals are middle-aged straight white males. Suspend your disbelief for a moment, please. If that’s the case, presumably an employer would be justly cautious when hiring a middle-aged straight white male and would wish for further information to assuage their confidence. They could use the vast amount of publicly available data to look for high-risk predictors of white-collar crime – past convictions, criminal head shapes, degrees from Harvard Business School, etcetera. The argument can be made that the innocent have nothing to hide, and indeed that’s true – in fact, because of the same logic, poor black men actually benefit on net from mandatory drug testing by employers. More information only helps the innocent. And yet it is a hop, skip, and a jump from that logic to the logic of redlining. It’s not even a particularly slippery slope, but rather an application of the same logic that underlies price discrimination in many of the areas where “Big Data” is applied today.
I think the political economy of this is unsustainable. The use of personal data in ways that today seem highly inappropriate will only continue and will become even more personal and intrusive over time. It’s not just bus schedules – it’s car insurers trawling Facebook to see whether you’re high-risk, and even worse, once Facebook figures out the algorithm for high-risk driving they will be happy to sell that data to whoever has the cash. People resist the intrusion of the market into their lives, and the monetization of people’s being will eventually trigger intense political backlash and a desire to declare certain uses of data out of bounds. “Privacy” is just the tip of the iceberg – people will demand protection from the precise measurement and metering of their souls that Silicon Valley is out to create.
I wish I had the answers for what policy might serve these goals, but I suspect policy responses will mostly be inadequate to the scale of this societal change.