The man behind Facebook’s artificial brain attempt (Wired UK)


Yann LeCun, the new head of artificial intelligence at Facebook

Josh Valcarcel/Wired


It’s good to be Yann LeCun.

Mark Zuckerberg recently handpicked the longtime NYU professor
to run
Facebook’s new artificial intelligence lab
. The IEEE
Computational Society just gave him its prestigious Neural Network Pioneer Award, in honour of his
work on deep learning, a form of artificial intelligence meant to
more closely mimic the human brain. And, perhaps most of all, deep
learning has suddenly spread across the commercial tech world, from
Google to Microsoft to Baidu to Twitter, just a few years after
most AI researchers openly scoffed at it.

All of these tech companies are now exploring a particular type
of deep learning called convolutional neural networks, aiming to
build web services that can do things like automatically understand
natural language and recognise images. At Google, “convnets” power
the voice recognition system available
on Android phones
. At China’s
Baidu
, they drive a new visual search engine. This kind of deep
learning has many fathers, but its success should resonate with
LeCun more than anyone. “Convolutional neural nets for vision –
that’s what he pushed more than anybody else,” says Microsoft’s
Leon Bottou, one of LeCun’s earliest collaborators.

He pushed it in the face of enormous skepticism. In the ’80s,
when LeCun first got behind the idea of convnets — an
approximation of the networks of neurones in the brain — the
powerful computers and enormous data sets needed to make them work
just didn’t exist. The very notion of a neural network had fallen
into disrepute after it failed to deliver on the promises of
scientists who first dreamed of artificial intelligence at the dawn
of the computer age. It was hard to publish anything related to
neural nets in the major academic journals, and this would remain
the case in the ’90s and on into the aughts.

But LeCun persisted. “He kind of carried the torch through the
dark ages,” says Geoffrey Hinton, the central
figure in the deep learning movement
. And eventually, computer
power caught up with the remarkable technology.

LeCun’s LeNets

More than two decades before joining Facebook, LeCun worked at
Bell Labs, perhaps the most famous of computer research labs, the
birthplace of the transistor, the Unix operating system, and the C programming
language. During his stint there, the French researcher
developed a system that could recognise written digits. He
called it LeNet.

Automatically reading bank checks, it marked the first time
convolutional neural nets were applied to practical problems.
“Convolutional nets were little toys, and Yann changed them into
things that worked on a large scale,” says Bottou. And thanks to
Larry Jackel, the chief of LeCun’s department who recorded a demo
in 1993, you can see the technology in action, with a giddy,
baby-faced LeCun test-driving the thing (video below).

Some of the concepts baked into LeNet date back to work
LeCun did in France at Bottou’s college pad and later at a lab
run by Hinton, and he was first inspired by two research
papers from by Kunihiko Fukushima, another neural-net great who, in
the ’70s and ’80s, had invented what were called the Cognitron and Neocognitron. These early neural nets could learn to pick
out patterns in data, on their own, without much human
prompting. But they were rather complicated, and researchers
couldn’t quite figure out how to make them work well. “What was
missing was the supervised learning algorithm,” says LeCun. “What
we call back prop now.”

“Back propagation” is a clever way to minimise error. To
understand it, you also have to understand how convolutional
neural nets work.

What the hell is a Convolutional Neural
Net?


Like other flavours of neural nets, convolutional networks are
software creations organised into interconnected layers, much like
the visual cortex, the part of the brain that process visual
information. What makes them different is that they
can reuse the same filters at multiple locations in an image.
That means that once the network has learned to recognise, say
a face, at one location, it can also automatically find
faces in others. (The same principle holds for sound waves and
written words.)

This allows artificial neural nets to be trained quickly, and
because they have a “small memory footprint, you don’t need to
separately store a filter for each location in the image…[making]
them well-suited to building much more scalable deep nets,” says Baidu’s Andrew
Ng
. It’s also what makes them so adept at recognising
patterns
.

When receiving an image — the input — the network
translates it into arrays of numbers that represent
features, and the “neurones” in each layer of the network are tuned
to recognise certain pattens in the numbers. Low-level neurones
recognise things like edges or basic shapes, while neurones in
higher layers can “see” objects — say, a dog or a
person. Each layer communicates with the one above it,
and as information travels up the network, some averaging takes
place. At the end, the network comes up with an output — a
guess at what’s in the image.

If the network makes a mistake, engineers can fine-tune the
connections between layers to get the right answer. But neural
nets work better if they can do this fine tuning on their own.
That’s where the back propagation algorithm comes in.

Convolutional Network Demo from 1993Yann LeCun

Hello, back prop

Back prop is all about computing the error and using that value to
update the strength — or weight — given to each of the layers in
a neural net. Hinton, with David Rumelhart and Ronald Williams,
came up with a version of back prop that calculated the error for
multiple inputs at once and then took the average. That value was
then back-propagated through the network, from the output to the
input layers. In a paper published in the journal Nature in 1986, they showed that
this approach improved learning.

Around the same time, LeCun was busy devising his own recipe for
back prop in Paris, based on research that dated back to the 1950s
but that others had more or less failed to apply to real-world
problems. Instead of averaging, LeCun’s version calculated the
error for every single example. It was more noisy, but it worked
well — and more quickly. “A lot of people didn’t believe
that you could do that,” recalls Bottou.

According to Bottou, the method was the result of the weak
machines they were using. “The computers we had in France were
less endowed.” They had to come up with a hack to
calculate error quickly while using as little computational
power as possible. But what seemed like a “fudge” at the time — to
borrow Hinton’s term — turned out to be spot on. Today, it’s
called stochastic gradient descent, and like convolutional neural
nets as a whole, it’s a staple of the artificial intelligence
toolbox.

LeCun’s LeNets would be widely licensed and used in
ATMs and banks across the world to read what was written on
checks. But skepticism remained. “Somehow, that wasn’t enough to
convince the computer vision community that convolutional neural
nets were worthy,” says LeCun. Part of it was that, although
they were powerful, nobody knew why they so powerful. The
inner-workings of this technology were a mystery.

The wager on the future of AI

There were many critics. Vladimir Vapnik, a mathematician and the
father of the support vector machine, one of the most widely used
AI models, was among them.

One March afternoon in 1995, Vapnik and Larry Jackel –
who’d recruited him and LeCun to Bell Labs — made a bet.
Jackel wagered that, by the year 2000, we’d have a
have a handle on how deep artificial neural nets worked. Vapnik
disagreed. He also thought that by 2005, “no one in his right
mind will use neural nets that are essentially like those used in
1995.” Fancy dinners were at stake, so they put the bet on paper
and signed it — in front of witnesses. LeCun served as the third
official signatory, Bottou as an unofficial observer.

Vapnik would win the first half of the wager. In 2000, the inner
workings of neural nets were still largely shrouded in mystery, and
even now, researchers can’t pinpoint mathematically exactly what
makes them work well. But, the second victory belonged to Jackel –
and, more importantly, to LeCun. By 2005, deep neural nets were
still being used in ATMs and banks, and they were very much
rooted in work dating back to LeCun’s work in the mid-1980s and
early ’90s.

“It’s rarely the case where a technology that has been around
for 20, 25 years — basically unchanged — turns out to be the
best,” says LeCun. “The speed at which people have embraced it
is nothing short of amazing. I’ve never seen anything like
this before.”

What’s to come

But this is just a start. The deep learning community — LeCun
included — are working to improve the technology.
Today’s most widely used convolutional neural nets rely almost
exclusively on supervised learning. Basically, that means that
if you want it to learn how to identify a particular object, you
have to label more than a few examples. Yet unsupervised learning — or learning from unlabelled
data — is closer to how real brains learn, and some deep learning
research is exploring this area.

“How this is done in the brain is pretty much completely
unknown. Synapses adjust themselves, but we don’t have a clear
picture for what the algorithm of the cortex is,” says LeCun. “We
know the ultimate answer is unsupervised learning, but we don’t
have the answer yet.”

It’s also unlikely that back prop — what neural-net expert Yoshua
Bengio
calls “the workhorse of most deep learning systems” –
mirrors the human brain, so researchers are developing
alternatives. Plus, the way that convolutional nets
“pool,” or average, data doesn’t sit well with some, so
there are efforts to improve this as well. “It loses information,”
says Hinton.

Say you’re looking at faces. The system learns to recognise
facial features, like eyes and lips. Based on these,
it’s good at identifying that there is a face in an image, but
much less tuned to picking out differences between faces. If you
want to know the precise location of the eyes in a face, for
example, it can’t do that very well. As tech companies and
governments want to build more detailed digital dossiers of their
customers and citizens, these kinds of limitations will become,
well, limiting.

The general ideas behind LeCun’s convnets may not be perfect,
but they’re the state of the art. “He turned out to be completely
right,” says Hinton, before adding with a quick laugh, “except for
the pooling.”


Yann LeCun with Rob Fergus, another member of the new Facebook AI Lab, at NYU in 2010

Katie Drummond/Wired


LeCun’s legacy

LeCun’s work extends well beyond neural nets. In the late ’90s, he
had a hand in building a seminal image compression system. The
idea was to scan documents and then put these up on the web for all
to see. The technology never quite panned out for LeCun, but
the concepts behind the technology impressed a young Larry
Page, the co-founder of Google, who heard a talk LeCun gave at
Stanford in 1998 when he was still a graduate student.

LeCun has also worked with robotics and AI hardware. He
recently founded NYU’s Center for Data Science. And he has mentored
a new generation of AI researchers, including Clement Fabaret,
whose image-indexing company Madbits was recently acquired by
Twitter
. In his spare time, he builds model planes.

With that pedigree, it’s no surprise that Zuckerberg asked
LeCun to help the company make sense
of all its data
. After all, the social network has been busy
acquiring companies — like the virtual-reality company Oculus,
solar-powered drone maker Ascenta, and WhatsApp — whose
products could benefit from the type of AI LeCun
pioneered.

LeCun is actively looking to hire more AI talent at the company
— Rob Fergus, with whom he collaborated with at NYU, is
already part of his team at Facebook — and he’s been tasked
with turning the AI lab into a world-class research outfit, a place
to compete with Google, Microsoft, IBM and Baidu, to be sure, but
also an operation that harkens back to Bell Labs, which served as a
breeding ground for innovation and the birthplace of many of
the technologies we take for granted today, including deep
learning.

A scientist at heart, he also wants to keep developing his own
ideas. “I’m not giving up on research, on the contrary. I’m opening
up a new venue for new research to take place,” he says. “It’s
a much more exciting situation to be in — when the
contribution you make is valued.”

This article originally appeared on Wired.com

If the article suppose to have a video or a photo gallery and it does not appear on your screen, please Click Here

15 August 2014 | 9:18 am – Source: wired.co.uk

Leave a Reply

Your email address will not be published.