English-Video.net comment policy

The comment field is common to all languages

Let's write in your language and use "Google Translate" together

Please refer to informative community guidelines on TED.com

TED2012

Jared Ficklin: New ways to see music (with color! and fire!)

Filmed
Views 672,881

Designer Jared Ficklin creates wild visualizations that let us see music, using color and even fire (a first for the TED stage) to analyze how sound makes us feel. He takes a brief digression to analyze the sound of a skatepark -- and how audio can clue us in to developing creativity.

- Visualizer
In his day job, Jared Ficklin makes user interfaces at frog design. As a hobby, he explores what music looks like ... in light, in shapes, in fire. Full bio

My passions
00:16
are music, technology and making things.
00:17
And it's the combination of these things
00:21
that has led me to the hobby of sound visualization,
00:24
and, on occasion, has led me to play with fire.
00:26
This is a Rubens' tube. It's one of many I've made over the years,
00:31
and I have one here tonight.
00:33
It's about an 8-foot-long tube of metal,
00:35
it's got a hundred or so holes on top,
00:36
on that side is the speaker, and here
00:38
is some lab tubing, and it's connected to this tank
00:39
of propane.
00:41
So, let's fire it up and see what it does.
00:44
So let's play a 550-herz frequency
00:53
and watch what happens.
00:55
(Frequency)
00:57
Thank you. (Applause)
01:05
It's okay to applaud the laws of physics,
01:08
but essentially what's happening here
01:10
-- (Laughter) --
01:11
is the energy from the sound via the air and gas molecules
01:13
is influencing the combustion properties of propane,
01:17
creating a visible waveform,
01:20
and we can see the alternating regions of compression
01:22
and rarefaction that we call frequency,
01:24
and the height is showing us amplitude.
01:26
So let's change the frequency of the sound,
01:28
and watch what happens to the fire.
01:30
(Higher frequency)
01:31
So every time we hit a resonant frequency we get a standing wave
01:42
and that emergent sine curve of fire.
01:45
So let's turn that off. We're indoors.
01:47
Thank you. (Applause)
01:48
I also have with me a flame table.
01:54
It's very similar to a Rubens' tube, and it's also used
01:56
for visualizing the physical properties of sound,
01:58
such as eigenmodes, so let's fire it up
02:00
and see what it does.
02:02
Ooh. (Laughter)
02:08
Okay. Now, while the table comes up to pressure,
02:12
let me note here that the sound is not traveling
02:15
in perfect lines. It's actually traveling in all directions,
02:17
and the Rubens' tube's a little like bisecting those waves
02:20
with a line, and the flame table's a little like
02:23
bisecting those waves with a plane,
02:25
and it can show a little more subtle complexity, which is why
02:27
I like to use it to watch Geoff Farina play guitar.
02:31
(Music)
02:33
All right, so it's a delicate dance.
03:15
If you watch closely — (Applause)
03:17
If you watch closely, you may have seen
03:19
some of the eigenmodes, but also you may have seen
03:22
that jazz music is better with fire.
03:25
Actually, a lot of things are better with fire in my world,
03:29
but the fire's just a foundation.
03:31
It shows very well that eyes can hear,
03:34
and this is interesting to me because
03:35
technology allows us to present sound to the eyes
03:36
in ways that accentuate the strength of the eyes
03:39
for seeing sound, such as the removal of time.
03:42
So here, I'm using a rendering algorithm to paint
03:45
the frequencies of the song "Smells Like Teen Spirit"
03:48
in a way that the eyes can take them in
03:51
as a single visual impression, and the technique
03:53
will also show the strengths of the visual cortex
03:55
for pattern recognition.
03:57
So if I show you another song off this album,
03:58
and another, your eyes will easily pick out
04:00
the use of repetition by the band Nirvana,
04:04
and in the frequency distribution, the colors,
04:07
you can see the clean-dirty-clean sound
04:09
that they are famous for,
04:12
and here is the entire album as a single visual impression,
04:13
and I think this impression is pretty powerful.
04:17
At least, it's powerful enough that
04:19
if I show you these four songs,
04:20
and I remind you that this is "Smells Like Teen Spirit,"
04:22
you can probably correctly guess, without listening
04:24
to any music at all, that the song
04:27
a die hard Nirvana fan would enjoy is this song,
04:28
"I'll Stick Around" by the Foo Fighters,
04:30
whose lead singer is Dave Grohl,
04:33
who was the drummer in Nirvana.
04:35
The songs are a little similar, but mostly
04:38
I'm just interested in the idea that someday maybe
04:40
we'll buy a song because we like the way it looks.
04:41
All right, now for some more sound data.
04:46
This is data from a skate park,
04:47
and this is Mabel Davis skate park
04:49
in Austin, Texas. (Skateboard sounds)
04:51
And the sounds you're hearing came from eight
04:54
microphones attached to obstacles around the park,
04:55
and it sounds like chaos, but actually
04:57
all the tricks start with a very distinct slap,
04:59
but successful tricks end with a pop,
05:03
whereas unsuccessful tricks
05:04
more of a scratch and a tumble,
05:06
and tricks on the rail will ring out like a gong, and
05:08
voices occupy very unique frequencies in the skate park.
05:12
So if we were to render these sounds visually,
05:15
we might end up with something like this.
05:17
This is all 40 minutes of the recording,
05:18
and right away the algorithm tells us
05:21
a lot more tricks are missed than are made,
05:23
and also a trick on the rails is a lot more likely
05:25
to produce a cheer, and if you look really closely,
05:27
we can tease out traffic patterns.
05:30
You see the skaters often trick in this direction. The obstacles are easier.
05:32
And in the middle of the recording, the mics pick this up,
05:38
but later in the recording, this kid shows up,
05:39
and he starts using a line at the top of the park
05:42
to do some very advanced tricks on something
05:45
called the tall rail.
05:47
And it's fascinating. At this moment in time,
05:48
all the rest of the skaters turn their lines 90 degrees
05:50
to stay out of his way.
05:54
You see, there's a subtle etiquette in the skate park,
05:55
and it's led by key influencers,
05:58
and they tend to be the kids who can do the best tricks,
05:59
or wear red pants, and on this day the mics picked that up.
06:03
All right, from skate physics to theoretical physics.
06:05
I'm a big fan of Stephen Hawking,
06:09
and I wanted to use all eight hours
06:11
of his Cambridge lecture series to create an homage.
06:12
Now, in this series he's speaking with the aid of a computer,
06:15
which actually makes identifying the ends of sentences
06:18
fairly easy. So I wrote a steering algorithm.
06:21
It listens to the lecture, and then it uses
06:24
the amplitude of each word to move a point on the x-axis,
06:26
and it uses the inflection of sentences
06:29
to move a same point up and down on the y-axis.
06:31
And these trend lines, you can see, there's more questions
06:34
than answers in the laws of physics,
06:36
and when we reach the end of a sentence,
06:38
we place a star at that location.
06:40
So there's a lot of sentences, so a lot of stars,
06:43
and after rendering all of the audio, this is what we get.
06:45
This is Stephen Hawking's universe.
06:48
(Applause)
06:51
It's all eight hours of the Cambridge lecture series
06:58
taken in as a single visual impression,
07:00
and I really like this image,
07:02
but a lot of people think it's fake.
07:04
So I made a more interactive version,
07:06
and the way I did that is I used their position in time
07:08
in the lecture to place these stars into 3D space,
07:13
and with some custom software and a Kinect,
07:16
I can walk right into the lecture.
07:18
I'm going to wave through the Kinect here
07:21
and take control, and now I'm going to reach out
07:23
and I'm going to touch a star, and when I do,
07:24
it will play the sentence
07:28
that generated that star.
07:30
Stephen Hawking: There is one, and only one, arrangement
07:31
in which the pieces make a complete picture.
07:35
Jared Ficklin: Thank you. (Applause)
07:38
There are 1,400 stars.
07:42
It's a really fun way to explore the lecture,
07:45
and, I hope, a fitting homage.
07:47
All right. Let me close with a work in progress.
07:48
I think, after 30 years, the opportunity exists
07:54
to create an enhanced version of closed captioning.
07:57
Now, we've all seen a lot of TEDTalks online,
07:59
so let's watch one now with the sound turned off
08:01
and the closed captioning turned on.
08:04
There's no closed captioning for the TED theme song,
08:08
and we're missing it, but if you've watched enough of these,
08:10
you hear it in your mind's ear,
08:12
and then applause starts.
08:13
It usually begins here, and it grows and then it falls.
08:16
Sometimes you get a little star applause,
08:18
and then I think even Bill Gates takes a nervous breath,
08:20
and the talk begins.
08:23
All right, so let's watch this clip again.
08:25
This time, I'm not going to talk at all.
08:30
There's still going to be no audio,
08:32
but what I am going to do is I'm going to render the sound
08:33
visually in real time at the bottom of the screen.
08:35
So watch closely and see what your eyes can hear.
08:39
This is fairly amazing to me.
09:03
Even on the first view, your eyes will successfully
09:05
pick out patterns, but on repeated views,
09:08
your brain actually gets better
09:12
at turning these patterns into information.
09:13
You can get the tone and the timbre
09:15
and the pace of the speech,
09:17
things that you can't get out of closed captioning.
09:18
That famous scene in horror movies
09:20
where someone is walking up from behind
09:22
is something you can see,
09:25
and I believe this information would be something
09:27
that is useful at times when the audio is turned off
09:29
or not heard at all, and I speculate that deaf audiences
09:32
might actually even be better
09:35
at seeing sound than hearing audiences.
09:36
I don't know. It's a theory right now.
09:38
Actually, it's all just an idea.
09:40
And let me end by saying that sound moves in all directions,
09:41
and so do ideas.
09:45
Thank you. (Applause)
09:47
Translated by Joseph Geni
Reviewed by Morton Bast

▲Back to top

About the speaker:

Jared Ficklin - Visualizer
In his day job, Jared Ficklin makes user interfaces at frog design. As a hobby, he explores what music looks like ... in light, in shapes, in fire.

Why you should listen

Jared Ficklin is a Senior Principal Design Technologist at frog, where he builds user experiences for clients, playing with interactions including touch and multi-touch, and applying physics to enhance the user experience. A passion for music and making things introduced him to the hobby of sound visualization, which has led him on occasion to play with fire. (As Flash on the Beach puts it, "Jared Ficklin’s sonic experiments stood out for their individuality, drama and casual disregard for health and safety.") Every March in Austin, Texas, Ficklin organizes the frog party, a collective social experiment for a few thousand people attending SXSW Interactive. It's a form of playful R&D for social technology. And he has spent 10 years helping fund, design and  build quality free public skateparks for Austin as part of the Austin Public Skatepark Action Committee. 

More profile about the speaker
Jared Ficklin | Speaker | TED.com