Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O’Neil
My rating: 3 of 5 stars
Important topic; how can new online tools and algorithms influence our behaviour? These new tools allow good salesmen to leverage their impact. In the old days, a good car salesman only reached hundereds of potential customers, a good Facebook-marketeer might reach a billion people.
These tools when used for evil are dubbed Weapons of Math Destruction (WMD) by Cathy O’Neill. Good, sticky name. And she rightly warns of the perils. The book, however, is a bit too superficial. It provides examples from different domains (finance, education, job-market), but feels repetitive and shallow.
I docked a star on my rating because of two elementary mistakes: on page 136 a teacher scored 6 out of 100 one year and 96 the next year. O’Neil writes “The 90 percent difference in socres”, which should be precentage-points (nit-picking, but in a book about math you’d expect that to be correct). And onpage 176 she botches the Body Mass Index formula; “The BMI is a person’s weight in kilograms divided by their height in centimeters”. In fact it is height in meters squared.
Quotes & interesting links
p7 In WMDs, many poisonous assumptions are camouflaged by math and go largely untested and unquestioned.
p12 For many of the businesses running these rogue algorithms [WMDs], the money pouring in seems to prove that their models are working (…) The software is doing its job. The trouble is that profits end up serving as a stand-in, or proxy, for truth. We’ll see this dangerous confusion crop up again and again.
p17 Moneyball: The Art of Winning an Unfair Game is now shorthand for any statistical approach in domains long ruled by the gut. But baseball represents a healthy case study (…) models are fair, in part, because they’re transparant. Everyone has access to the stats (…) the number of home runs and strikeouts are there for everyone to see. (…) The folks building WMDs routinely lack data for the behavors they’re most interested in. So they substitute stand-in data, or proxies.
p27 [on crime and recivism] the model itself contributes to a toxic cycle and helps to sustain it. That’s a signature quality of a WMD.
p31 So to sum up, these are the three elements of a WMD: Opacity, Scale, and Damage.
p67 The number of “graduates employed nine months adter graduation” can be gamed (…) Some schools hired their own graduates for hourly temp jobs just as the crucial nine-month period approached. Others sent out surveys to recent alumni and counted all those that didn’t respond as “employed” See http://www.nytimes.com/2011/01/09/bus…
p103 But that objective [uphold community standards] has been steamrolled by models that equate arrests with safety (…) From a mathematical point of view, however, trusts is hard to quantify. That’s a challenge to people building models. Sadly, it’s far simpler to keep counting arrests, to build models that assume we’re birds of a feather and treat us as such.
p108 Frank Schmidt (Univ Iowa) analyzed century of workplace productivity data to measure the predictive value of various selection processes. Personality tests ranked low on th escale – they were only one-third as predictive as cognitive exams, and also far below reference checks.
p109 The primary purpose of the [personality] test,” said Roland Behm, “is not to find the best employee. It’s to exclude as many people as possible as cheaply as possible.”
p137 (on the example of a teacher who scored 6% one year and 96% the next year); the problem was that the administrators lost track of accuracy in their quest to be fair (correcting for too many factors).
More on the Value Added Model; https://garyrubinstein.wordpress.com/…
p158 from https://pando.com/2013/07/31/flush-wi…
ZestFinance, at its core, is a math company – analytical models run in parallel to interpret more than 10,000 data points per credit applicant to arrive at more than 70,000 signals – all in under five seconds. Compare this to the 10 to 15 pieces of data that traditional lenders use and it’s unsurprising that the company is better able to assess consumer credit risk. (…)
the way a consumer types their name in the credit application – using all lowercase, all uppercase, or correct case – can be a predictor of credit risk. Other seemingly trivial data points include whether an applicant has read a letter on the company’s website and whether the applicant has a pre-paid or post-paid cell phone.
p165 In Florida, adults with clean driving records and poor credit scores paid an average of $1,552 more than the same drivers with excellent credit and a drunk driving conviction (…) I cannot imagine a more meaningful piece of data for auto insurers than a drunk riving record.
p197 With political messaging, as with most WMDs, the heart of the problem is almost always the objective. Change the objective from leeching off people to helping them, and a WMD is disarmed – and can even become a force for good.
p200 Being poor in a world of WMDs is getting more and more dangerous and expensive
p202 Fairness, in most cases, was a by-product
p204 We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.
p205 The Financial Modelers’ Manifesto (oath) http://www.emanuelderman.com/writing/…
p210 Take a look at the inputs [of models]
p210 Princeton Web Transparency & Accountability Project https://webtap.princeton.edu/
p218 Thes [WMD] models are constructed not just from data but from the choices we make about which data to pay attention to – and which to leave out. Those choices are not just about logistics, profits, and efficiency. They are fundamentally moral.
View all my reviews