Weapons of math destruction: how big data increases inequality and threatens democracy

by Cathy O'Neil

Posted on
mathematics ethics
thumbnail

In fact, I saw all kinds of parallels between finance and Big Data. Both industries gobble up the same pool of talent, much of it from elite universities like MIT, Princeton, or Stanford. These new hires are ravenous for success and have been focused on external metrics–like SAT scores and college admissions–their entire lives. Whether in finance or tech, the message they’ve received is that they will be rich, that they will run the world. Their productivity indicates that they’re on the right track, and it translates into dollars. This leads to the fallacious conclusion that whatever they’re doing to bring in more money is good. It “adds value.” Otherwise, why would the market reward it?

^ This is the problem for me if I’m not careful.

I should tell my story, so I can understand the biases and opportunities I’ve had.

A prison system can either:

  • try to determine recitivism by running an analysis of people’s background, or
  • try to reduce recitivism by testing different factors like sunlight, sports, food, literacy, etc.

Use models to improve the system, not to perpetuate it. Another example of this is using data to help people with extra challenges (learning English, mothers) to succeed rather than cutting them out based on those attributes.

You cannot make decisions based on a feedback-less model, without extra evidence supporting the decision.

Give people freedom to choose their lifestyle.

Stop a program if it doesn’t work.

Good intentions + transparency + positive feedback loops

Like the industrial revolution, we need to act to overcome the effects.

  • Government regulation and oversight
  • A desire to be moral – A sort of data science “Hippocratic Oath”
  • Choose to accomplish good rather than automate decisions already being made