TED2017
Noriko Arai: Can a robot pass a university entrance exam?
Filmed:
Readability: 4.4
1,550,497 views
Meet Todai Robot, an AI project that performed in the top 20 percent of students on the entrance exam for the University of Tokyo -- without actually understanding a thing. While it's not matriculating anytime soon, Todai Robot's success raises alarming questions for the future of human education. How can we help kids excel at the things that humans will always do better than AI?
Noriko Arai - AI expert
Could an AI pass the entrance exam for the University of Tokyo? Noriko Arai oversees a project that wants to find out. Full bio
Could an AI pass the entrance exam for the University of Tokyo? Noriko Arai oversees a project that wants to find out. Full bio
Double-click the English transcript below to play the video.
00:13
Today, I'm going to talk about AI and us.
0
1014
3660
00:18
AI researchers have always said
1
6206
2143
00:20
that we humans do not need to worry,
2
8373
2594
00:22
because only menial jobs
will be taken over by machines.
will be taken over by machines.
3
10991
3580
00:27
Is that really true?
4
15274
1603
00:30
They have also said
that AI will create new jobs,
that AI will create new jobs,
5
18365
3827
00:34
so those who lose their jobs
will find a new one.
will find a new one.
6
22216
3411
00:38
Of course.
7
26264
1355
00:39
But the real question is:
8
27643
2172
00:41
How many of those
who may lose their jobs to AI
who may lose their jobs to AI
9
29839
4105
00:45
will be able to land a new one,
10
33968
2489
00:48
especially when AI is smart enough
to learn better than most of us?
to learn better than most of us?
11
36481
5850
00:55
Let me ask you a question:
12
43397
2185
00:58
How many of you think
13
46666
1798
01:00
that AI will pass the entrance examination
of a top university by 2020?
of a top university by 2020?
14
48488
6094
01:07
Oh, so many. OK.
15
55836
2457
01:10
So some of you may say, "Of course, yes!"
16
58317
4358
01:15
Now singularity is the issue.
17
63369
2134
01:18
And some others may say, "Maybe,
18
66590
3095
01:21
because AI already won
against a top Go player."
against a top Go player."
19
69709
4508
01:27
And others may say, "No, never. Uh-uh."
20
75213
3681
01:32
That means we do not know
the answer yet, right?
the answer yet, right?
21
80195
3593
01:36
So that was the reason why
I started Todai Robot Project,
I started Todai Robot Project,
22
84268
4960
01:41
making an AI which passes
the entrance examination
the entrance examination
23
89252
3872
01:45
of the University of Tokyo,
24
93148
2589
01:47
the top university in Japan.
25
95761
2537
01:51
This is our Todai Robot.
26
99464
2548
01:56
And, of course, the brain of the robot
is working in the remote server.
is working in the remote server.
27
104131
5810
02:02
It is now writing a 600-word essay
28
110747
4222
02:06
on maritime trade in the 17th century.
29
114993
4119
02:11
How does that sound?
30
119136
1765
02:14
Why did I take the entrance exam
as its benchmark?
as its benchmark?
31
122113
4104
02:19
Because I thought we had to study
the performance of AI
the performance of AI
32
127098
4741
02:23
in comparison to humans,
33
131863
2114
02:26
especially on the scales and expertise
34
134001
2860
02:28
which are believed
to be acquired only by humans
to be acquired only by humans
35
136885
4088
02:32
and only through education.
36
140997
2335
02:35
To enter Todai, the University of Tokyo,
37
143782
4043
02:39
you have to pass
two different types of exams.
two different types of exams.
38
147849
4421
02:44
The first one is
a national standardized test
a national standardized test
39
152294
3760
02:48
in multiple-choice style.
40
156078
2403
02:50
You have to take seven subjects
41
158505
2455
02:52
and achieve a high score --
42
160984
1955
02:54
I would say like an 84 percent
or more accuracy rate --
or more accuracy rate --
43
162963
4772
02:59
to be allowed to take
the second stage written test
the second stage written test
44
167759
4087
03:03
prepared by Todai.
45
171870
2159
03:06
So let me first explain
how modern AI works,
how modern AI works,
46
174994
5317
03:12
taking the "Jeopardy!" challenge
as an example.
as an example.
47
180335
3069
03:17
Here is a typical "Jeopardy!" question:
48
185539
3079
03:20
"Mozart's last symphony
shares its name with this planet."
shares its name with this planet."
49
188642
4461
03:26
Interestingly, a "Jeopardy!"
question always asks,
question always asks,
50
194195
4013
03:30
always ends with "this" something:
51
198232
3328
03:33
"this" planet, "this" country,
52
201584
2827
03:36
"this" rock musician, and so on.
53
204435
2608
03:39
In other words, "Jeopardy!" doesn't ask
many different types of questions,
many different types of questions,
54
207067
4299
03:43
but a single type,
55
211390
1837
03:45
which we call "factoid questions."
56
213251
2536
03:48
By the way, do you know the answer?
57
216975
2167
03:53
If you do not know the answer
and if you want to know the answer,
and if you want to know the answer,
58
221980
4055
03:58
what would you do?
59
226059
1287
04:00
You Google, right? Of course.
60
228160
3132
04:03
Why not?
61
231316
1480
04:04
But you have to pick appropriate keywords
62
232820
3592
04:08
like "Mozart," "last"
and "symphony" to search.
and "symphony" to search.
63
236436
4364
04:13
The machine basically does the same.
64
241462
2400
04:16
Then this Wikipedia page
will be ranked top.
will be ranked top.
65
244457
4660
04:21
Then the machine reads the page.
66
249840
1908
04:23
No, uh-uh.
67
251772
1171
04:25
Unfortunately, none of the modern AIs,
68
253470
3462
04:28
including Watson, Siri and Todai Robot,
69
256956
3968
04:32
is able to read.
70
260948
1661
04:35
But they are very good
at searching and optimizing.
at searching and optimizing.
71
263437
3800
04:40
It will recognize
72
268158
2023
04:42
that the keywords "Mozart,"
"last" and "symphony"
"last" and "symphony"
73
270866
2935
04:45
are appearing heavily around here.
74
273825
2903
04:49
So if it can find a word which is a planet
75
277790
4375
04:54
and which is co-occurring
with these keywords,
with these keywords,
76
282189
3648
04:57
that must be the answer.
77
285861
1989
05:00
This is how Watson finds
the answer "Jupiter," in this case.
the answer "Jupiter," in this case.
78
288762
5186
05:08
Our Todai Robot works similarly,
but a bit smarter
but a bit smarter
79
296433
4049
05:12
in answering history yes-no questions,
80
300506
3239
05:16
like, "'Charlemagne repelled the Magyars.'
Is this sentence true or false?"
Is this sentence true or false?"
81
304560
5663
05:23
Our robot starts producing
a factoid question,
a factoid question,
82
311181
4073
05:27
like: "Charlemagne repelled
[this person type]" by itself.
[this person type]" by itself.
83
315278
4899
05:32
Then, "Avars" but not
"Magyars" is ranked top.
"Magyars" is ranked top.
84
320995
4732
05:38
This sentence is likely to be false.
85
326357
3049
05:42
Our robot does not read,
does not understand,
does not understand,
86
330772
4860
05:48
but it is statistically
correct in many cases.
correct in many cases.
87
336335
4144
05:54
For the second stage written test,
88
342147
2508
05:56
it is required to write
a 600-word essay like this one:
a 600-word essay like this one:
89
344679
5106
06:01
[Discuss the rise and fall
of the maritime trade
of the maritime trade
90
349809
2278
06:04
in East and Southeast Asia
in the 17th century ...]
in the 17th century ...]
91
352111
2422
06:06
and as I have shown earlier,
92
354557
1387
06:07
our robot took the sentences
from the textbooks and Wikipedia,
from the textbooks and Wikipedia,
93
355968
4194
06:12
combined them together,
94
360186
1961
06:14
and optimized it to produce an essay
95
362171
3619
06:17
without understanding a thing.
96
365814
2207
06:20
(Laughter)
97
368045
1737
06:21
But surprisingly, it wrote a better essay
98
369806
4895
06:26
than most of the students.
99
374725
1561
06:28
(Laughter)
100
376310
2391
06:30
How about mathematics?
101
378725
1529
06:33
A fully automatic math-solving machine
102
381354
3158
06:36
has been a dream
103
384536
1631
06:38
since the birth of the word
"artificial intelligence,"
"artificial intelligence,"
104
386191
4679
06:43
but it has stayed at the level
of arithmetic for a long, long time.
of arithmetic for a long, long time.
105
391785
6007
06:51
Last year, we finally succeeded
in developing a system
in developing a system
106
399530
5350
06:56
which solved pre-university-level
problems from end to end,
problems from end to end,
107
404904
5173
07:02
like this one.
108
410101
1262
07:05
This is the original problem
written in Japanese,
written in Japanese,
109
413648
4002
07:09
and we had to teach it
2,000 mathematical axioms
2,000 mathematical axioms
110
417674
4397
07:14
and 8,000 Japanese words
111
422095
2774
07:16
to make it accept the problems
written in natural language.
written in natural language.
112
424893
4558
07:22
And it is now translating
the original problems
the original problems
113
430234
3542
07:25
into machine-readable formulas.
114
433800
3139
07:30
Weird, but it is now ready
to solve it, I think.
to solve it, I think.
115
438578
6099
07:36
Go and solve it.
116
444701
1411
07:38
Yes! It is now executing
symbolic computation.
symbolic computation.
117
446818
4284
07:44
Even more weird,
118
452030
1580
07:45
but probably this is the most
fun part for the machine.
fun part for the machine.
119
453634
4825
07:50
(Laughter)
120
458483
2351
07:52
Now it outputs a perfect answer,
121
460858
2815
07:55
though its proof is impossible to read,
even for mathematicians.
even for mathematicians.
122
463697
4707
08:02
Anyway, last year our robot
was among the top one percent
was among the top one percent
123
470773
6961
08:10
in the second stage written
exam in mathematics.
exam in mathematics.
124
478199
3633
08:14
(Applause)
125
482652
3210
08:18
Thank you.
126
486412
1311
08:19
So, did it enter Todai?
127
487747
2471
08:22
No, not as I expected.
128
490981
3058
08:26
Why?
129
494783
1399
08:28
Because it doesn't understand any meaning.
130
496206
2639
08:32
Let me show you a typical error
it made in the English test.
it made in the English test.
131
500308
4079
08:36
[Nate: We're almost at the bookstore.
Just a few more minutes.
Just a few more minutes.
132
504411
2977
08:39
Sunil: Wait. ______ .
Nate: Thank you! That always happens ...]
Nate: Thank you! That always happens ...]
133
507412
3039
08:42
Two people are talking.
134
510475
1151
08:43
For us, who can understand
the situation --
the situation --
135
511650
2054
08:45
[1. "We walked for a long time."
2. "We're almost there."
2. "We're almost there."
136
513704
2773
08:48
3. "Your shoes look expensive."
4. "Your shoelace is untied."]
4. "Your shoelace is untied."]
137
516501
3032
08:51
it is obvious number four
is the correct answer, right?
is the correct answer, right?
138
519557
2873
08:54
But Todai Robot chose number two,
139
522454
2238
08:56
even after learning 15 billion
English sentences
English sentences
140
524716
5360
09:02
using deep learning technologies.
141
530100
2728
09:07
OK, so now you might
understand what I said:
understand what I said:
142
535600
4172
09:12
modern AIs do not read,
143
540399
2648
09:15
do not understand.
144
543071
1413
09:17
They only disguise as if they do.
145
545516
3169
09:24
This is the distribution graph
146
552867
2981
09:27
of half a million students
who took the same exam as Todai Robot.
who took the same exam as Todai Robot.
147
555872
5777
09:34
Now our Todai Robot
is among the top 20 percent,
is among the top 20 percent,
148
562558
5165
09:40
and it was capable to pass
149
568986
2415
09:43
more than 60 percent
of the universities in Japan --
of the universities in Japan --
150
571425
3941
09:47
but not Todai.
151
575390
1377
09:50
But see how it is beyond the volume zone
152
578116
4025
09:54
of to-be white-collar workers.
153
582165
2864
10:00
You might think I was delighted.
154
588060
2858
10:03
After all, my robot was surpassing
students everywhere.
students everywhere.
155
591939
3971
10:09
Instead, I was alarmed.
156
597022
2691
10:13
How on earth could this unintelligent
machine outperform students --
machine outperform students --
157
601086
5607
10:18
our children?
158
606717
1292
10:20
Right?
159
608033
1153
10:22
I decided to investigate
what was going on in the human world.
what was going on in the human world.
160
610101
4402
10:28
I took hundreds of sentences
from high school textbooks
from high school textbooks
161
616542
4729
10:33
and made easy multiple-choice quizzes,
162
621859
3313
10:37
and asked thousands
of high school students to answer.
of high school students to answer.
163
625196
4143
10:42
Here is an example:
164
630690
1176
10:43
[Buddhism spread to ... ,
Christianity to ... and Oceania,
Christianity to ... and Oceania,
165
631890
2818
10:46
and Islam to ...]
166
634732
1151
10:47
Of course, the original problems
are written in Japanese,
are written in Japanese,
167
635907
2740
10:50
their mother tongue.
168
638671
1155
10:51
[ ______ has spread to Oceania.
169
639850
1515
10:53
1. Hinduism 2. Christianity
3. Islam 4. Buddhism ]
3. Islam 4. Buddhism ]
170
641389
2417
10:55
Obviously, Christianity
is the answer, isn't it?
is the answer, isn't it?
171
643830
2299
10:58
It's written!
172
646153
1214
11:01
And Todai Robot chose
the correct answer, too.
the correct answer, too.
173
649482
4026
11:06
But one-third of junior
high school students
high school students
174
654758
4879
11:11
failed to answer this question.
175
659661
2612
11:16
Do you think it is only the case in Japan?
176
664456
3159
11:19
I do not think so,
177
667639
1976
11:21
because Japan is always ranked
among the top in OECD PISA tests,
among the top in OECD PISA tests,
178
669639
6371
11:28
measuring 15-year-old
students' performance in mathematics,
students' performance in mathematics,
179
676034
3927
11:31
science and reading
180
679985
1964
11:33
every three years.
181
681973
1636
11:39
We have been believing
182
687390
2053
11:41
that everybody can learn
183
689467
2043
11:43
and learn well,
184
691534
1905
11:45
as long as we provide
good learning materials
good learning materials
185
693463
3697
11:49
free on the web
186
697184
1455
11:50
so that they can access
through the internet.
through the internet.
187
698663
3069
11:53
But such wonderful materials
may benefit only those who can read well,
may benefit only those who can read well,
188
701756
5859
12:00
and the percentage
of those who can read well
of those who can read well
189
708534
3935
12:04
may be much less than we expected.
190
712493
3378
12:10
How we humans will coexist with AI
191
718040
4241
12:14
is something we have
to think about carefully,
to think about carefully,
192
722305
3522
12:17
based on solid evidence.
193
725851
2137
12:21
At the same time,
we have to think in a hurry
we have to think in a hurry
194
729063
3977
12:25
because time is running out.
195
733064
2402
12:28
Thank you.
196
736106
1162
12:29
(Applause)
197
737626
3933
12:34
Chris Anderson: Noriko, thank you.
198
742211
2080
12:36
Noriko Arai: Thank you.
199
744315
1765
12:38
CA: In your talk, you so beautifully
give us a sense of how AIs think,
give us a sense of how AIs think,
200
746104
5304
12:43
what they can do amazingly
201
751432
1564
12:45
and what they can't do.
202
753020
1695
12:46
But -- do I read you right,
203
754739
1494
12:48
that you think we really need
quite an urgent revolution in education
quite an urgent revolution in education
204
756257
5270
12:53
to help kids do the things
that humans can do better than AIs?
that humans can do better than AIs?
205
761551
4155
12:57
NA: Yes, yes, yes.
206
765730
1328
12:59
Because we humans
can understand the meaning.
can understand the meaning.
207
767082
4035
13:03
That is something
which is very, very lacking in AI.
which is very, very lacking in AI.
208
771141
4906
13:08
But most of the students
just pack the knowledge
just pack the knowledge
209
776071
4368
13:12
without understanding
the meaning of the knowledge,
the meaning of the knowledge,
210
780463
3903
13:16
so that is not knowledge,
that is just memorizing,
that is just memorizing,
211
784390
2809
13:19
and AI can do the same thing.
212
787223
2450
13:21
So we have to think about
a new type of education.
a new type of education.
213
789697
3631
13:25
CA: A shift from knowledge,
rote knowledge, to meaning.
rote knowledge, to meaning.
214
793352
3289
13:28
NA: Mm-hmm.
215
796665
1151
13:29
CA: Well, there's a challenge
for the educators. Thank you so much.
for the educators. Thank you so much.
216
797840
3240
13:33
NA: Thank you very much. Thank you.
217
801104
1698
13:34
(Applause)
218
802826
1185
ABOUT THE SPEAKER
Noriko Arai - AI expertCould an AI pass the entrance exam for the University of Tokyo? Noriko Arai oversees a project that wants to find out.
Why you should listen
Noriko Arai is the program director of an AI challenge, Todai Robot Project, which asks the question: Can AI get into the University of Tokyo? The project aims to visualize both the possibilities and the limitation of current AI by setting a concrete goal: a software system that can pass university entrance exams. In 2015 and 2016, Todai Robot achieved top 20 percent in the exams, and passed more than 70 percent of the universities in Japan.
The inventor of Reading Skill Test, in 2017 Arai conducted a large-scale survey on reading skills of high and junior high school students with Japan's Ministry of Education. The results revealed that more than half of junior high school students fail to comprehend sentences sampled from their textbooks. Arai founded the Research Institute of Science for Education to elucidate why so many students fail to read and how she can support them.
Noriko Arai | Speaker | TED.com