TED2017
Cathy O'Neil: The era of blind faith in big data must end
Keti O'Nil (Cathy O'Neil): Doba slepe vere u masovne podatke mora se okončati
Filmed:
Readability: 4.4
1,391,460 views
Algoritmi odlučuju ko dobija kredit, ko će biti pozvan na razgovor za posao, ko dobija osiguranje i još mnogo toga, ali nisu automatski pravični. Keti O'Nil, matematičarka i naučnica za podatke, skovala je frazu za tajne, važne i štetne algoritme - „oružja matematičkog uništenja“. Saznajte više o tajnim planovima koji se kriju iza formula.
Cathy O'Neil - Mathematician, data scientist
Data skeptic Cathy O’Neil uncovers the dark secrets of big data, showing how our "objective" algorithms could in fact reinforce human bias. Full bio
Data skeptic Cathy O’Neil uncovers the dark secrets of big data, showing how our "objective" algorithms could in fact reinforce human bias. Full bio
Double-click the English transcript below to play the video.
Algoritmi su svuda.
00:12
Algorithms are everywhere.
0
975
1596
00:16
They sort and separate
the winners from the losers.
the winners from the losers.
1
4111
3125
Filtriraju i odvajaju
pobednike od gubitnika.
pobednike od gubitnika.
Pobednici dobijaju posao
00:20
The winners get the job
2
8019
2264
ili dobru ponudu kreditne kartice.
00:22
or a good credit card offer.
3
10307
1743
Za gubitnike nema ni intervjua
00:24
The losers don't even get an interview
4
12074
2651
ili plaćaju više za osiguranje.
00:27
or they pay more for insurance.
5
15590
1777
Ocenjuju nas tajne formule
koje ne razumemo,
koje ne razumemo,
00:30
We're being scored with secret formulas
that we don't understand
that we don't understand
6
18197
3549
za koje često ne postoje
sistemi za podnošenje žalbe.
sistemi za podnošenje žalbe.
00:34
that often don't have systems of appeal.
7
22675
3217
To nas tera se zapitamo:
00:39
That begs the question:
8
27240
1296
„Šta ako algoritmi greše?“
00:40
What if the algorithms are wrong?
9
28560
2913
Da napravite algoritam
potrebne su vam dve stvari:
potrebne su vam dve stvari:
00:45
To build an algorithm you need two things:
10
33100
2040
00:47
you need data, what happened in the past,
11
35164
1981
podaci, šta je bilo u prošlosti,
i definicija uspeha,
00:49
and a definition of success,
12
37169
1561
koji tražite i kojem se često nadate.
00:50
the thing you're looking for
and often hoping for.
and often hoping for.
13
38754
2457
Osposobljavate algoritam
posmatranjem i zaključivanjem.
posmatranjem i zaključivanjem.
00:53
You train an algorithm
by looking, figuring out.
by looking, figuring out.
14
41235
5037
Algoritam pronalazi vezu sa uspehom.
00:58
The algorithm figures out
what is associated with success.
what is associated with success.
15
46296
3419
Koja situacija vodi ka uspehu?
01:01
What situation leads to success?
16
49739
2463
Zapravo, svi koriste algoritme,
01:04
Actually, everyone uses algorithms.
17
52881
1762
ali ih ne formalizuju u pisani kôd.
01:06
They just don't formalize them
in written code.
in written code.
18
54667
2718
Evo vam primera.
01:09
Let me give you an example.
19
57409
1348
Svaki dan koristim algoritam
da napravim porodici doručak.
da napravim porodici doručak.
01:10
I use an algorithm every day
to make a meal for my family.
to make a meal for my family.
20
58781
3316
Podaci koje koristim su:
01:14
The data I use
21
62121
1476
namirnice u kuhinji,
01:16
is the ingredients in my kitchen,
22
64394
1659
vreme koje imam na raspolaganju,
01:18
the time I have,
23
66077
1527
ambicija koju imam,
01:19
the ambition I have,
24
67628
1233
i te podatke obrađujem.
01:20
and I curate that data.
25
68885
1709
U hranu ne računam
ona mala pakovanja instant špageta.
ona mala pakovanja instant špageta.
01:22
I don't count those little packages
of ramen noodles as food.
of ramen noodles as food.
26
70618
4251
(Smeh)
01:26
(Laughter)
27
74893
1869
Moja definicija uspeha je -
01:28
My definition of success is:
28
76786
1845
obrok je uspeo ako moja deca jedu povrće.
01:30
a meal is successful
if my kids eat vegetables.
if my kids eat vegetables.
29
78655
2659
To bi izgledalo mnogo drugačije
da se moj sin pita.
da se moj sin pita.
01:34
It's very different
from if my youngest son were in charge.
from if my youngest son were in charge.
30
82181
2854
Njemu bi uspeh bio da se najede nutele.
01:37
He'd say success is if
he gets to eat lots of Nutella.
he gets to eat lots of Nutella.
31
85059
2788
Ali, ja biram uspeh.
01:41
But I get to choose success.
32
89179
2226
Ja sam zadužena za to.
Moje mišljenje je važno.
Moje mišljenje je važno.
01:43
I am in charge. My opinion matters.
33
91429
2707
To je prvo pravilo algoritama.
01:46
That's the first rule of algorithms.
34
94160
2675
Algoritmi su mišljenja ugrađena u kôd.
01:48
Algorithms are opinions embedded in code.
35
96859
3180
To se veoma razlikuje od onoga što mislite
da većina ljudi misli o algoritmima.
da većina ljudi misli o algoritmima.
01:53
It's really different from what you think
most people think of algorithms.
most people think of algorithms.
36
101562
3663
01:57
They think algorithms are objective
and true and scientific.
and true and scientific.
37
105249
4504
Ljudi misle da su algoritmi
objektivni, istiniti i naučni.
objektivni, istiniti i naučni.
To je marketinški trik.
02:02
That's a marketing trick.
38
110387
1699
02:05
It's also a marketing trick
39
113269
2125
Marketinški trik jeste
02:07
to intimidate you with algorithms,
40
115418
3154
i kada vas plaše algoritmima,
kada vas teraju da verujete
i da se plašite algoritama
i da se plašite algoritama
02:10
to make you trust and fear algorithms
41
118596
3661
jer verujete matematici i plašite je se.
02:14
because you trust and fear mathematics.
42
122281
2018
Mnogo grešaka može se desiti
kada slepo verujemo u masovne podatke.
kada slepo verujemo u masovne podatke.
02:17
A lot can go wrong when we put
blind faith in big data.
blind faith in big data.
43
125567
4830
Ovo je Kiri Soares,
direktor srednje škole u Bruklinu.
direktor srednje škole u Bruklinu.
02:23
This is Kiri Soares.
She's a high school principal in Brooklyn.
She's a high school principal in Brooklyn.
44
131684
3373
Godine 2011. rekla mi je
da njen kolektiv ocenjuju
da njen kolektiv ocenjuju
02:27
In 2011, she told me
her teachers were being scored
her teachers were being scored
45
135081
2586
složenim, tajnim algoritmom
02:29
with a complex, secret algorithm
46
137691
2727
koji se zove „model dodatne vrednosti“.
02:32
called the "value-added model."
47
140442
1489
Rekla sam joj: „Saznaj
koja je formula i pokaži mi je.
koja je formula i pokaži mi je.
02:34
I told her, "Well, figure out
what the formula is, show it to me.
what the formula is, show it to me.
48
142505
3092
Objasniću ti je.“
02:37
I'm going to explain it to you."
49
145621
1541
Rekla je: „Pokušala sam dobiti formulu,
02:39
She said, "Well, I tried
to get the formula,
to get the formula,
50
147186
2141
ali osoba iz Ministarstva obrazovanja
mi je rekla da je to matematika
mi je rekla da je to matematika
02:41
but my Department of Education contact
told me it was math
told me it was math
51
149351
2772
i da je neću razumeti.“
02:44
and I wouldn't understand it."
52
152147
1546
Postaje sve gore.
02:47
It gets worse.
53
155266
1338
Njujork Post je podneo zahtev
na osnovu zakona o slobodi informacija,
na osnovu zakona o slobodi informacija,
02:48
The New York Post filed
a Freedom of Information Act request,
a Freedom of Information Act request,
54
156628
3530
dobio imena i ocene svih nastavnika
02:52
got all the teachers' names
and all their scores
and all their scores
55
160182
2959
i onda ih objavio kao čin
sramoćenja nastavnika.
sramoćenja nastavnika.
02:55
and they published them
as an act of teacher-shaming.
as an act of teacher-shaming.
56
163165
2782
Kada sam probala istim putem doći
do formula, do izvornog kôda,
do formula, do izvornog kôda,
02:59
When I tried to get the formulas,
the source code, through the same means,
the source code, through the same means,
57
167084
3860
rečeno mi je da ne može.
03:02
I was told I couldn't.
58
170968
2149
Odbijena sam.
03:05
I was denied.
59
173141
1236
Posle sam saznala
03:06
I later found out
60
174401
1174
da niko u Njujorku
nema podatke o toj formuli.
nema podatke o toj formuli.
03:07
that nobody in New York City
had access to that formula.
had access to that formula.
61
175599
2866
Niko je nije razumeo.
03:10
No one understood it.
62
178489
1305
Onda se uključio neko
veoma bistar, Geri Rubenstajn.
veoma bistar, Geri Rubenstajn.
03:13
Then someone really smart
got involved, Gary Rubenstein.
got involved, Gary Rubenstein.
63
181929
3224
Pronašao je 665 nastavnika
iz onog članka u Njujork Postu
iz onog članka u Njujork Postu
03:17
He found 665 teachers
from that New York Post data
from that New York Post data
64
185177
3621
koji zapravo imaju dva rezultata.
03:20
that actually had two scores.
65
188822
1866
Ovo se moglo desiti jer su predavali
matematiku u sedmom i osmom razredu.
matematiku u sedmom i osmom razredu.
03:22
That could happen if they were teaching
66
190712
1881
03:24
seventh grade math and eighth grade math.
67
192617
2439
Odlučio je da ih ubaci u grafikon.
03:27
He decided to plot them.
68
195080
1538
Svaka tačka je nastavnica ili nastavnik.
03:28
Each dot represents a teacher.
69
196642
1993
(Smeh)
03:31
(Laughter)
70
199104
2379
Šta je to?
03:33
What is that?
71
201507
1521
(Smeh)
03:35
(Laughter)
72
203052
1277
To se nikako nije trebalo koristiti
za individualne procene.
za individualne procene.
03:36
That should never have been used
for individual assessment.
for individual assessment.
73
204353
3446
Ovo je kao generator nasumičnih brojeva.
03:39
It's almost a random number generator.
74
207823
1926
(Aplauz)
03:41
(Applause)
75
209773
2946
Ali, korišćeno je.
03:44
But it was.
76
212743
1162
Ovo je Sara Visoki.
03:45
This is Sarah Wysocki.
77
213929
1176
Otpuštena je kad i 205 drugih nastavnika
03:47
She got fired, along
with 205 other teachers,
with 205 other teachers,
78
215129
2175
iz škola vašingtonskog okruga
03:49
from the Washington, DC school district,
79
217328
2662
iako je imala odlučne preporuke direktora
03:52
even though she had great
recommendations from her principal
recommendations from her principal
80
220014
2909
i roditelja učenika.
03:54
and the parents of her kids.
81
222947
1428
Znam šta mnogi od vas ovde sada misle,
03:57
I know what a lot
of you guys are thinking,
of you guys are thinking,
82
225390
2032
posebno naučnici za podatke,
stručnjaci za veštačku inteligenciju.
stručnjaci za veštačku inteligenciju.
03:59
especially the data scientists,
the AI experts here.
the AI experts here.
83
227446
2487
04:01
You're thinking, "Well, I would never make
an algorithm that inconsistent."
an algorithm that inconsistent."
84
229957
4226
Mislite: „Pa, nikada ne bismo napravili
tako nedosledan algoritam.“
tako nedosledan algoritam.“
Ali, algoritmi mogu pogrešiti,
04:06
But algorithms can go wrong,
85
234853
1683
čak imati i duboko destruktivno dejstvo
sa dobrom namerama.
sa dobrom namerama.
04:08
even have deeply destructive effects
with good intentions.
with good intentions.
86
236560
4598
Dok loše napravljen avion
04:14
And whereas an airplane
that's designed badly
that's designed badly
87
242531
2379
padne na tlo i svi to vide,
04:16
crashes to the earth and everyone sees it,
88
244934
2001
loše osmišljen algoritam
04:18
an algorithm designed badly
89
246959
1850
može trajati dugo i potajno
i tiho praviti ogromnu štetu.
i tiho praviti ogromnu štetu.
04:22
can go on for a long time,
silently wreaking havoc.
silently wreaking havoc.
90
250245
3865
Ovo je Rodžer Ejls.
04:27
This is Roger Ailes.
91
255748
1570
(Smeh)
04:29
(Laughter)
92
257342
2000
Osnovao je Foks Njuz 1996. godine.
04:32
He founded Fox News in 1996.
93
260524
2388
Preko 20 žena žalilo se
na seksualno uznemiravanje.
na seksualno uznemiravanje.
04:35
More than 20 women complained
about sexual harassment.
about sexual harassment.
94
263436
2581
Rekle su da im u Foks Njuzu
nije dozvoljen uspeh.
nije dozvoljen uspeh.
04:38
They said they weren't allowed
to succeed at Fox News.
to succeed at Fox News.
95
266041
3235
Izbačen je prošle godine,
04:41
He was ousted last year,
but we've seen recently
but we've seen recently
96
269300
2520
ali nedavno smo videli
da problemi još nisu rešeni.
da problemi još nisu rešeni.
04:43
that the problems have persisted.
97
271844
2670
To zahteva da se postavi pitanje
04:47
That begs the question:
98
275654
1400
šta Foks Njuz treba da uradi
da okrene novi list.
da okrene novi list.
04:49
What should Fox News do
to turn over another leaf?
to turn over another leaf?
99
277078
2884
Šta bi se desilo da proces zapošljavanja
04:53
Well, what if they replaced
their hiring process
their hiring process
100
281245
3041
zamene mašinskim algoritmom koji uči?
04:56
with a machine-learning algorithm?
101
284310
1654
Zvuči dobro, zar ne?
04:57
That sounds good, right?
102
285988
1595
Razmislite o tome.
04:59
Think about it.
103
287607
1300
Podaci. Šta bi bi bili podaci?
05:00
The data, what would the data be?
104
288931
2105
Ima smisla izabrati prijave za Foks Njuz
tokom poslednjih 21 godina.
tokom poslednjih 21 godina.
05:03
A reasonable choice would be the last
21 years of applications to Fox News.
21 years of applications to Fox News.
105
291060
4947
Ima smisla.
05:08
Reasonable.
106
296031
1502
A definicija uspeha?
05:09
What about the definition of success?
107
297557
1938
Razuman izbor bio bi, valjda,
05:11
Reasonable choice would be,
108
299921
1324
neko ko je uspešan u Foks Njuzu?
05:13
well, who is successful at Fox News?
109
301269
1778
Recimo, osoba koja je tamo
bila četiri godine
bila četiri godine
05:15
I guess someone who, say,
stayed there for four years
stayed there for four years
110
303071
3580
i dobila unapređenje makar jednom.
05:18
and was promoted at least once.
111
306675
1654
Ima smisla.
05:20
Sounds reasonable.
112
308816
1561
Onda bismo osposobljavali algoritam.
05:22
And then the algorithm would be trained.
113
310401
2354
Osposobili bismo ga da traži ljude,
da uči šta je vodilo ka uspehu,
da uči šta je vodilo ka uspehu,
05:24
It would be trained to look for people
to learn what led to success,
to learn what led to success,
114
312779
3877
kakve vrste prijava su vremenom
vodile ka uspehu
vodile ka uspehu
05:29
what kind of applications
historically led to success
historically led to success
115
317219
4318
u skladu sa tom definicijom.
05:33
by that definition.
116
321561
1294
Razmislite sada šta bi se desilo
05:36
Now think about what would happen
117
324200
1775
kada bismo to primenili
na trenutne kandidate.
na trenutne kandidate.
05:37
if we applied that
to a current pool of applicants.
to a current pool of applicants.
118
325999
2555
Izbacilo bi žene
05:41
It would filter out women
119
329119
1629
jer ne deluju kao osobe
koje su bile uspešne u prošlosti.
koje su bile uspešne u prošlosti.
05:43
because they do not look like people
who were successful in the past.
who were successful in the past.
120
331663
3930
Algoritmi ne popravljaju stvari
05:51
Algorithms don't make things fair
121
339752
2537
ako ih samo nonšalantno
i slepo primenjujete.
i slepo primenjujete.
05:54
if you just blithely,
blindly apply algorithms.
blindly apply algorithms.
122
342313
2694
Ne popravljaju stvari.
05:57
They don't make things fair.
123
345031
1482
Ponavljaju našu praksu iz prošlosti,
05:58
They repeat our past practices,
124
346537
2128
naše šablone.
06:00
our patterns.
125
348689
1183
Automatizuju status kvo.
06:01
They automate the status quo.
126
349896
1939
Da živimo u savršenom
svetu to bi bilo sjajno,
svetu to bi bilo sjajno,
06:04
That would be great
if we had a perfect world,
if we had a perfect world,
127
352718
2389
ali ne živimo.
06:07
but we don't.
128
355905
1312
Dodaću da većina firmi
nema sramne parnice,
nema sramne parnice,
06:09
And I'll add that most companies
don't have embarrassing lawsuits,
don't have embarrassing lawsuits,
129
357241
4102
ali naučnicima za podatke
u tim kompanijama
u tim kompanijama
06:14
but the data scientists in those companies
130
362446
2588
rečeno je idu tragom podataka,
06:17
are told to follow the data,
131
365058
2189
da paze na tačnost.
06:19
to focus on accuracy.
132
367271
2143
Razmislite šta to znači.
06:22
Think about what that means.
133
370273
1381
Pošto smo svi pristrasni, to znači
da će možda kodifikovati seksizam
da će možda kodifikovati seksizam
06:23
Because we all have bias,
it means they could be codifying sexism
it means they could be codifying sexism
134
371678
4027
ili drugu netrpeljivost.
06:27
or any other kind of bigotry.
135
375729
1836
Misaoni eksperiment,
06:31
Thought experiment,
136
379488
1421
jer ih volim:
06:32
because I like them:
137
380933
1509
jedno društvo, skroz podeljeno -
06:35
an entirely segregated society --
138
383574
2975
na osnovu rase, svi gradovi, sve opštine -
06:40
racially segregated, all towns,
all neighborhoods
all neighborhoods
139
388247
3328
a policiju šaljemo
samo u delove gde živi manjina
samo u delove gde živi manjina
06:43
and where we send the police
only to the minority neighborhoods
only to the minority neighborhoods
140
391599
3037
u potrazi za kriminalom.
06:46
to look for crime.
141
394660
1193
Podaci o hapšenjima
bili bi veoma pristrasni.
bili bi veoma pristrasni.
06:48
The arrest data would be very biased.
142
396451
2219
Šta ako bismo, povrh svega,
pronašli naučnike za podatke
pronašli naučnike za podatke
06:51
What if, on top of that,
we found the data scientists
we found the data scientists
143
399851
2575
06:54
and paid the data scientists to predict
where the next crime would occur?
where the next crime would occur?
144
402450
4161
i platili tim naučnicima
da predvide mesto sledećeg zločina?
da predvide mesto sledećeg zločina?
Opštine sa manjinama.
06:59
Minority neighborhood.
145
407275
1487
Ili da predvide
ko će sledeći biti kriminalac?
ko će sledeći biti kriminalac?
07:01
Or to predict who the next
criminal would be?
criminal would be?
146
409285
3125
Neko iz manjine.
07:04
A minority.
147
412888
1395
Naučnici bi se hvalisali
svojim sjajnim i tačnim modelom,
svojim sjajnim i tačnim modelom,
07:07
The data scientists would brag
about how great and how accurate
about how great and how accurate
148
415949
3541
07:11
their model would be,
149
419514
1297
i bili bi u pravu.
07:12
and they'd be right.
150
420835
1299
Realnost nije tako drastična,
ali postoje ozbiljne podele
ali postoje ozbiljne podele
07:15
Now, reality isn't that drastic,
but we do have severe segregations
but we do have severe segregations
151
423951
4615
u mnogim malim i velikom gradovima,
07:20
in many cities and towns,
152
428590
1287
07:21
and we have plenty of evidence
153
429901
1893
i imamo mnoštvo dokaza
o pristrasnim podacima
u sistemu policije i pravosuđa.
u sistemu policije i pravosuđa.
07:23
of biased policing
and justice system data.
and justice system data.
154
431818
2688
Mi zapravo predviđamo krizna mesta,
07:27
And we actually do predict hotspots,
155
435632
2815
mesta gde će se desiti nasilje.
07:30
places where crimes will occur.
156
438471
1530
I predviđamo, zapravo,
pojedinačni kriminalitet,
pojedinačni kriminalitet,
07:32
And we do predict, in fact,
the individual criminality,
the individual criminality,
157
440401
3866
kriminalitet pojedinaca.
07:36
the criminality of individuals.
158
444291
1770
Novinska organizacija Propablika
nedavno je proverila
nedavno je proverila
07:38
The news organization ProPublica
recently looked into
recently looked into
159
446972
3963
jedan „algoritam ugrožen recidivizmom“
07:42
one of those "recidivism risk" algorithms,
160
450959
2024
kako ih zovu,
07:45
as they're called,
161
453007
1163
koje sudije koriste
u presudama na Floridi.
u presudama na Floridi.
07:46
being used in Florida
during sentencing by judges.
during sentencing by judges.
162
454194
3194
Bernard, levo, crnac,
dobio je 10 od 10 poena.
dobio je 10 od 10 poena.
07:50
Bernard, on the left, the black man,
was scored a 10 out of 10.
was scored a 10 out of 10.
163
458411
3585
Dilan, desno, 3 od 10.
07:55
Dylan, on the right, 3 out of 10.
164
463179
2007
Deset od deset, visok rizik.
Tri od deset, nizak rizik.
Tri od deset, nizak rizik.
07:57
10 out of 10, high risk.
3 out of 10, low risk.
3 out of 10, low risk.
165
465210
2501
Obojica su privedeni
zbog posedovanja droge.
zbog posedovanja droge.
08:00
They were both brought in
for drug possession.
for drug possession.
166
468598
2385
Obojica su imali dosije,
08:03
They both had records,
167
471007
1154
ali Dilan je imao krivično delo,
08:04
but Dylan had a felony
168
472185
2806
a Bernard nije.
08:07
but Bernard didn't.
169
475015
1176
Ovo je bitno jer što su ti veći poeni,
08:09
This matters, because
the higher score you are,
the higher score you are,
170
477818
3066
veće su šanse da dobiješ dužu kaznu.
08:12
the more likely you're being given
a longer sentence.
a longer sentence.
171
480908
3473
O čemu se ovde radi?
08:18
What's going on?
172
486294
1294
Pranje podataka.
08:20
Data laundering.
173
488526
1332
Proces kojim tehnolozi
sakrivaju ružnu istinu
sakrivaju ružnu istinu
08:22
It's a process by which
technologists hide ugly truths
technologists hide ugly truths
174
490930
4427
u crne kutije algoritama
08:27
inside black box algorithms
175
495381
1821
i nazivaju ih objektivnima;
08:29
and call them objective;
176
497226
1290
nazivaju ih meritokratskim.
08:31
call them meritocratic.
177
499320
1568
Za tajne, važne i destruktivne algoritme
08:35
When they're secret,
important and destructive,
important and destructive,
178
503118
2385
sam skovala frazu
„oružje za matematičko uništenje“.
„oružje za matematičko uništenje“.
08:37
I've coined a term for these algorithms:
179
505527
2487
08:40
"weapons of math destruction."
180
508038
1999
(Smeh)
08:42
(Laughter)
181
510061
1564
(Aplauz)
08:43
(Applause)
182
511649
3054
Oni su svuda i to nije greška.
08:46
They're everywhere,
and it's not a mistake.
and it's not a mistake.
183
514727
2354
To su privatne kompanije
koje prave privatne algoritme
koje prave privatne algoritme
08:49
These are private companies
building private algorithms
building private algorithms
184
517695
3723
za privatne ciljeve.
08:53
for private ends.
185
521442
1392
Čak i one već spomenute,
za nastavnike i policiju,
za nastavnike i policiju,
08:55
Even the ones I talked about
for teachers and the public police,
for teachers and the public police,
186
523214
3214
napravile su privatne kompanije
08:58
those were built by private companies
187
526452
1869
i zatim ih prodale vladinim telima.
09:00
and sold to the government institutions.
188
528345
2231
Zovu ih „tajnim umakom“;
09:02
They call it their "secret sauce" --
189
530600
1873
zato nam ništa ne mogu reći o tome.
09:04
that's why they can't tell us about it.
190
532497
2128
To je i privatna moć.
09:06
It's also private power.
191
534649
2220
Zarađuju na korišćenju autoriteta
koji se ne može proveriti.
koji se ne može proveriti.
09:09
They are profiting for wielding
the authority of the inscrutable.
the authority of the inscrutable.
192
537924
4695
Možda ste pomislili da,
pošto je ovo privatno,
pošto je ovo privatno,
09:17
Now you might think,
since all this stuff is private
since all this stuff is private
193
545114
2934
postoji konkurencija;
09:20
and there's competition,
194
548072
1158
možda će slobodno tržište rešiti problem.
09:21
maybe the free market
will solve this problem.
will solve this problem.
195
549254
2306
Neće.
09:23
It won't.
196
551584
1249
Mnogo se novca može napraviti nepravdom.
09:24
There's a lot of money
to be made in unfairness.
to be made in unfairness.
197
552857
3120
Uz to, mi nismo
ekonomski racionalni činioci.
ekonomski racionalni činioci.
09:29
Also, we're not economic rational agents.
198
557127
3369
Svi smo pristrasni.
09:33
We all are biased.
199
561031
1292
Svi smo rasisti i netrpeljivi
onako kako ne želimo biti
onako kako ne želimo biti
09:34
We're all racist and bigoted
in ways that we wish we weren't,
in ways that we wish we weren't,
200
562960
3377
u oblicima koje i ne poznajemo.
09:38
in ways that we don't even know.
201
566361
2019
Ipak, znamo da je to kolektivno
09:41
We know this, though, in aggregate,
202
569352
3081
jer to sociolozi dosledno dokazuju
09:44
because sociologists
have consistently demonstrated this
have consistently demonstrated this
203
572457
3220
eksperimentima koje osmišljavaju,
09:47
with these experiments they build,
204
575701
1665
kada pošalju gomilu prijava za posao,
09:49
where they send a bunch
of applications to jobs out,
of applications to jobs out,
205
577390
2568
podjednako dobrih, ali neke imaju
imena koja zvuče belački
imena koja zvuče belački
09:51
equally qualified but some
have white-sounding names
have white-sounding names
206
579982
2501
a neke koje zvuče kao crnački,
09:54
and some have black-sounding names,
207
582507
1706
i uvek su razočaravajući rezultati; uvek.
09:56
and it's always disappointing,
the results -- always.
the results -- always.
208
584237
2694
Tako, mi smo pristrasni
09:59
So we are the ones that are biased,
209
587510
1771
i mi u algoritme ubacujemo pristrasnost
10:01
and we are injecting those biases
into the algorithms
into the algorithms
210
589305
3429
izborom podataka za prikupljanje,
10:04
by choosing what data to collect,
211
592758
1812
kao kada sam odlučila
da ne mislim o instant-špagetama;
da ne mislim o instant-špagetama;
10:06
like I chose not to think
about ramen noodles --
about ramen noodles --
212
594594
2743
odlučila sam da su nebitne.
10:09
I decided it was irrelevant.
213
597361
1625
Ako verujemo podacima
koji otkrivaju praksu iz prošlosti
koji otkrivaju praksu iz prošlosti
10:11
But by trusting the data that's actually
picking up on past practices
picking up on past practices
214
599010
5684
i biramo definiciju uspeha,
10:16
and by choosing the definition of success,
215
604718
2014
kako onda očekujemo
da algoritmi ostanu neoštećeni?
da algoritmi ostanu neoštećeni?
10:18
how can we expect the algorithms
to emerge unscathed?
to emerge unscathed?
216
606756
3983
Ne možemo. Moramo ih proveriti.
10:22
We can't. We have to check them.
217
610763
2356
Moramo proveriti da li su pravični.
10:26
We have to check them for fairness.
218
614165
1709
Dobra vest jeste da možemo
proveriti jesu li pravični.
proveriti jesu li pravični.
10:27
The good news is,
we can check them for fairness.
we can check them for fairness.
219
615898
2711
Algoritme možemo ispitati
10:30
Algorithms can be interrogated,
220
618633
3352
i reći će nam istinu svaki put.
10:34
and they will tell us
the truth every time.
the truth every time.
221
622009
2034
I možemo ih popraviti.
Možemo ih poboljšati.
Možemo ih poboljšati.
10:36
And we can fix them.
We can make them better.
We can make them better.
222
624067
2493
To zovem revizijom algoritma
10:38
I call this an algorithmic audit,
223
626584
2375
i ukratko ću vam je objasniti.
10:40
and I'll walk you through it.
224
628983
1679
Prvo, provera integriteta podataka.
10:42
First, data integrity check.
225
630686
2196
Zbog algoritma rizika od recidivizma
o kojem sam govorila,
o kojem sam govorila,
10:46
For the recidivism risk
algorithm I talked about,
algorithm I talked about,
226
634132
2657
provera integriteta podataka
značila bi prihvatanje činjenice
značila bi prihvatanje činjenice
10:49
a data integrity check would mean
we'd have to come to terms with the fact
we'd have to come to terms with the fact
227
637582
3573
da u SAD crnci i belci
podjednako puše travu
podjednako puše travu
10:53
that in the US, whites and blacks
smoke pot at the same rate
smoke pot at the same rate
228
641179
3526
ali crnci imaju mnogo više
šanse da budu uhapšeni -
šanse da budu uhapšeni -
10:56
but blacks are far more likely
to be arrested --
to be arrested --
229
644729
2485
četiri ili pet puta, zavisi od kraja.
10:59
four or five times more likely,
depending on the area.
depending on the area.
230
647238
3184
Kako ta pristrasnost izgleda
u drugim kriminalnim oblastima,
u drugim kriminalnim oblastima,
11:03
What is that bias looking like
in other crime categories,
in other crime categories,
231
651317
2826
i kako je uzimamo u obzir?
11:06
and how do we account for it?
232
654167
1451
Drugo, treba da razmislimo
o definiciji uspeha,
o definiciji uspeha,
11:08
Second, we should think about
the definition of success,
the definition of success,
233
656162
3039
da je revidiramo.
11:11
audit that.
234
659225
1381
Setite se algoritma za zapošljavanje
koji smo spomenuli.
koji smo spomenuli.
11:12
Remember -- with the hiring
algorithm? We talked about it.
algorithm? We talked about it.
235
660630
2752
Osoba koja je tu četiri godine
i unapređena je jednom?
i unapređena je jednom?
11:15
Someone who stays for four years
and is promoted once?
and is promoted once?
236
663406
3165
Pa, to je uspešan zaposleni,
11:18
Well, that is a successful employee,
237
666595
1769
ali je takođe i zaposleni
u skladu sa njihovom kulturom.
u skladu sa njihovom kulturom.
11:20
but it's also an employee
that is supported by their culture.
that is supported by their culture.
238
668388
3079
Tako i to može biti pristrasno.
11:24
That said, also it can be quite biased.
239
672089
1926
Moramo razdvojiti te dve stvari.
11:26
We need to separate those two things.
240
674039
2065
Treba da uzmemo
slepe audicije za orkestar kao primer.
slepe audicije za orkestar kao primer.
11:28
We should look to
the blind orchestra audition
the blind orchestra audition
241
676128
2426
11:30
as an example.
242
678578
1196
Tamo ljudi konkurišu su iza zastora.
11:31
That's where the people auditioning
are behind a sheet.
are behind a sheet.
243
679798
2756
Što je meni bitno jeste
11:34
What I want to think about there
244
682946
1931
da ljudi koji slušaju
odlučuju šta je bitno
odlučuju šta je bitno
11:36
is the people who are listening
have decided what's important
have decided what's important
245
684901
3417
i odlučuju su šta nije bitno,
11:40
and they've decided what's not important,
246
688342
2029
tako da im to ne odvlači pažnju.
11:42
and they're not getting
distracted by that.
distracted by that.
247
690395
2059
Otkad su počele
slepe audicije za orkestre,
slepe audicije za orkestre,
11:44
When the blind orchestra
auditions started,
auditions started,
248
692961
2749
broj žena u orkestrima
povećao se pet puta.
povećao se pet puta.
11:47
the number of women in orchestras
went up by a factor of five.
went up by a factor of five.
249
695734
3444
Zatim, moramo razmotriti tačnost.
11:52
Next, we have to consider accuracy.
250
700253
2015
Tada bi se model dodatne vrednosti
za nastavnike odmah raspao.
za nastavnike odmah raspao.
11:55
This is where the value-added model
for teachers would fail immediately.
for teachers would fail immediately.
251
703233
3734
Nema savršenog algoritma, naravno,
11:59
No algorithm is perfect, of course,
252
707578
2162
pa moramo razmotriti
greške svakog algoritma.
greške svakog algoritma.
12:02
so we have to consider
the errors of every algorithm.
the errors of every algorithm.
253
710620
3605
Koliko su greške česte
i za koga ovaj model ne funkcioniše?
i za koga ovaj model ne funkcioniše?
12:06
How often are there errors,
and for whom does this model fail?
and for whom does this model fail?
254
714836
4359
Koja je cena te nefunkcionalnosti?
12:11
What is the cost of that failure?
255
719850
1718
Na kraju, moramo razmotriti
12:14
And finally, we have to consider
256
722434
2207
dugoročne efekte algoritama,
12:17
the long-term effects of algorithms,
257
725973
2186
njihove povratne kružne sprege
koje se stvaraju.
koje se stvaraju.
12:20
the feedback loops that are engendering.
258
728866
2207
Ovo zvuči apstraktno,
12:23
That sounds abstract,
259
731586
1236
ali zamislite da su Fejsbukovi inženjeri
to uzeli u obzir
to uzeli u obzir
12:24
but imagine if Facebook engineers
had considered that
had considered that
260
732846
2664
pre odluke da nam prikažu
samo postove naših prijatelja.
samo postove naših prijatelja.
12:28
before they decided to show us
only things that our friends had posted.
only things that our friends had posted.
261
736270
4855
Imam još dve poruke,
jednu za naučnike koji se bave podacima.
jednu za naučnike koji se bave podacima.
12:33
I have two more messages,
one for the data scientists out there.
one for the data scientists out there.
262
741761
3234
Naučnici za podatke - ne treba
da budemo sudije istine.
da budemo sudije istine.
12:37
Data scientists: we should
not be the arbiters of truth.
not be the arbiters of truth.
263
745450
3409
Treba da budemo
prevodioci etičkih rasprava
prevodioci etičkih rasprava
12:41
We should be translators
of ethical discussions that happen
of ethical discussions that happen
264
749520
3783
koje se odvijaju u širem društvu.
12:45
in larger society.
265
753327
1294
(Aplauz)
12:47
(Applause)
266
755579
2133
A za vas ostale,
12:49
And the rest of you,
267
757736
1556
koji niste naučnici za podatke:
12:52
the non-data scientists:
268
760011
1396
12:53
this is not a math test.
269
761431
1498
ovo nije test iz matematike.
Ovo je politička borba.
12:55
This is a political fight.
270
763632
1348
Od naših algoritamskih vladara
moramo zahtevati odgovornost.
moramo zahtevati odgovornost.
12:58
We need to demand accountability
for our algorithmic overlords.
for our algorithmic overlords.
271
766587
3907
(Aplauz)
13:04
(Applause)
272
772118
1499
Doba slepe vere u masovne podatke
mora se okončati.
mora se okončati.
13:05
The era of blind faith
in big data must end.
in big data must end.
273
773641
4225
Hvala vam mnogo.
13:09
Thank you very much.
274
777890
1167
(Aplauz)
13:11
(Applause)
275
779081
5303
ABOUT THE SPEAKER
Cathy O'Neil - Mathematician, data scientistData skeptic Cathy O’Neil uncovers the dark secrets of big data, showing how our "objective" algorithms could in fact reinforce human bias.
Why you should listen
In 2008, as a hedge-fund quant, mathematician Cathy O’Neil saw firsthand how really really bad math could lead to financial disaster. Disillusioned, O’Neil became a data scientist and eventually joined Occupy Wall Street’s Alternative Banking Group.
With her popular blog mathbabe.org, O’Neil emerged as an investigative journalist. Her acclaimed book Weapons of Math Destruction details how opaque, black-box algorithms rely on biased historical data to do everything from sentence defendants to hire workers. In 2017, O’Neil founded consulting firm ORCAA to audit algorithms for racial, gender and economic inequality.
Cathy O'Neil | Speaker | TED.com