Yann LeCun & Soumith Chintala | Fireside Chat at PyTorch Developer Day 2021

Introduction

In this section, Soumith Chintala introduces Yann Lecun as his mentor and one of the pioneers of deep learning. He also talks about how Yann's frameworks in the '80s and '90s are the foundations for many ideas in PyTorch.

Introducing Yann Lecun

• Soumith Chintala introduces Yann Lecun as a pioneer of deep learning.

• Yann Lecun is a Turing Award winner, chief scientist at Facebook, and professor at NYU.

• Yann's frameworks from the '80s and '90s are foundational to many ideas in PyTorch.

The Early Days of Deep Learning Frameworks

In this section, Yann Lecun talks about his early work on deep learning frameworks. He discusses his first neural net simulator called HLM, which was written in Pascal. He then talks about how he moved to Lisp because it was easy to write an interpreter for it.

Early Work on Deep Learning Frameworks

• In 1983, Yann wrote his first neural net simulator called HLM.

• HLM was written in Pascal and could do backpropagation.

• To improve HLM, Yann realized he needed loops and functions so he started studying front-end programming languages.

• Eventually settled on Lisp because it was easy to write an interpreter for it.

• In 1987, there was no Python or MATLAB available off the internet so you had to write your own language.

SN2: A Neural Net Simulator

In this section, Yann Lecun talks about SN2 which stands for "simulateur du neuronal". It was developed with Leon Bottou who is currently at Facebook. They wrote a Lisp interpreter and then hooked up a numerical library for neural nets in the back.

SN2: A Neural Net Simulator

• SN2 stands for "simulateur du neuronal".

• Yann developed SN2 with Leon Bottou who is currently at Facebook.

• They wrote a Lisp interpreter and hooked up a numerical library for neural nets in the back.

• SN2 was eventually open-sourced in the early 2000s under the name Lush.

Assembling Blocks

In this section, Yann Lecun talks about how he and Leon Bottou had an idea that the right way to build a deep learning system was to have functional modules that you assembled in graphs. This would allow for automatic differentiation.

Assembling Blocks

• The right way to build a deep learning system is to have functional modules that you assemble in graphs.

• This allows for automatic differentiation.

• You don't need neurons and weights, just functional blocks.

• Torch, Caffe1, Caffe2, TensorFlow, and PyTorch all use this model of assembling blocks.

Training Neural Nets

In this section, Yann Lecun talks about training neural nets on fixed-point versus floating-point numbers. He also mentions that they experimented with both but usually trained on floating point numbers.

Training Neural Nets

• HLN was floating point but very slow because machines couldn't do it well at the time.

• HLN had a mode where you could turn on fixed-point which was much faster.

• When experimenting with SN2, they used floating point numbers most of the time.

Building a Demo for Character Recognition

In this section, the speaker talks about his experience building a demo for character recognition using computational nets and DSP boards.

Building the Demo

• The speaker wrote a compiler that would take the computational net, prune it, suppress small weights, and generate self-contained C code.

• The demo was compiled on a DSP board because PCs were not powerful enough at the time. AT&T was building a DSP chip capable of 20 megaflops which was amazing at the time.

• The demo involved putting a list of numbers under a camera and hitting a key. It ran on the DSP and had a compiler that made computation cheap enough.

Breakthroughs in Research

In this section, the speaker talks about breakthroughs in research over the last few decades and what's top of mind right now.

Top of Mind Research Problem

• The speaker is interested in self-supervised learning as one of two or three main problems to solve to make true progress towards more intelligent systems.

• One problem is allowing machines to learn like humans and animals. Currently used learning paradigms are not nearly as efficient as what we can observe in humans and animals.

• Another problem is allowing machines to reason in ways compatible with deep learning and gradient-based learning more generally. For tasks that require planning or reasoning beyond perception, there are big conceptual battles about what approaches work best.

• A third problem is training machines to represent action plans so they can learn how to decompose tasks into subtasks using tokens learned through observation.

Solving Research Problems

• The first problem to solve is how to get machines to learn how the world works and accumulate background knowledge about it through observation.

• The second and third problems are somewhat related. We need to figure out how to get machines to reason in ways compatible with deep learning and gradient-based learning more generally, as well as train them to represent action plans so they can learn how to decompose tasks into subtasks using tokens learned through observation.

Deep Learning and Symbol Manipulation

In this section, the speakers discuss the possibility of combining deep learning with classical AI methods for symbol manipulation. They also explore the idea of using vectors instead of symbols for reasoning.

Combining Deep Learning with Symbol Manipulation

• It is possible that in the short to medium term, systems will combine deep learning with classical AI methods for symbol manipulation.

• There is evidence that even deep learning systems can do symbol manipulation to some extent without being explicitly driven to do so.

Using Vectors for Reasoning

• The speakers discuss the idea of using vectors instead of symbols for reasoning.

• Geoff Hinton has been advocating for this approach for 40 years.

• The solution preferred by one speaker is one in which the reasoning part is done with vectors or patterns of activities rather than symbols.

Applying Current Techniques in Machine Learning and AI

In this section, the speakers discuss how much we could apply current techniques in machine learning and AI and how it could have a measurable effect on productivity and economic growth.

• The amount of stuff we could apply using current techniques in machine learning and AI is mind-boggling. There are not enough people trying to do this right now to really do a good job at it.

• Over the next 10 years or so, there will be a dissemination of deep learning AI technology in all corners of the economy, according to economists' predictions. This could have a measurable effect on productivity and economic growth.

Upstream Concepts in Machine Learning

In this section, the speakers discuss some of the interesting concepts that have emerged in machine learning over the last few years.

• There are many interesting concepts that people have come up with over the last few years that have a lot of potential to change the way we do things.

• Some of these concepts include graph neural nets, graph computational nets, invariant and variant neural nets, reversible models, generative models, and self-supervised learning.

• Self-supervised learning used to be based on an auto-encoder paradigm but this doesn't work for images because there are many different ways to reconstruct an image.

Dr. Lim and Non-Contrasting Methods

In this section, the speakers discuss Dr. Lim and various algorithms of this type where two neural nets are trained to output the same image. They also talk about non-contrasting methods that people have come up with over the last year.

Dr. Lim and Neural Nets

• Two neural nets are trained to output the same image.

• The outputs are trained to be the same when two different versions of the same image are shown.

• There are contrasting methods and non-contrasting methods.

Non-Contrasting Methods

• People have come up with non-contrasting methods over the last year.

• The speakers find these methods exciting.

Future of Machine Learning Frameworks

In this section, the speakers discuss their thoughts on current machine learning frameworks and where they might go in the future.

Current State of Machine Learning Frameworks

• Python is what people want because it's convenient.

• Python is not a good choice for a front-end language for a numerical library because it's slow and not compatible.

Future of Machine Learning Frameworks

• Efforts are being made to compile more Python code in PyTorch framework and ecosystem.

• Attempts to create functional versions of PyTorch that look like Jax or Lisp may fix limitations of Python.

Advice for Scientists and Engineers

In this section, one speaker gives advice on what young scientists or engineers should learn early on in their careers.

Advice for Young Scientists and Engineers

• Learn things that are difficult to learn when you're young and impossible to learn when you're less young, such as physics and math.

• Learning physics is relevant to machine learning.

Machine Learning in the Sciences

In this section, the speaker discusses how machine learning is becoming increasingly important in scientific research and how having expertise in both machine learning and a specific field of science can make someone valuable.

Importance of Deep Learning in Science

• Machine learning is becoming more prevalent in scientific research.

• Deep learning is one of the main tools that will take us forward.

• Having expertise in both deep learning and a specific field of science makes someone unique and valuable.

Learn Things with Long Shelf Life

• Courses on quantum mechanics, linear algebra, and probability theory teach you to reason about certain things that otherwise you would have no idea about.

• Knowledge gained from these courses has a long shelf life compared to technology that may disappear within a few years.

Participate in Open Source Projects

• Contributing to an open-source project can help you get noticed.

• Your name will be on the Git repository, which can give you credit for your work.

Advice for Success

In this section, the speaker gives advice on how to make yourself stand out as a machine learning professional.

Make Yourself Unique

• Try to make yourself unique by combining skills that don't exist anywhere else.

• Having double expertise in machine learning and another field makes you valuable.

Participate in Open Source Projects

• Contributing to an open-source project can help you get noticed.

• Your name will be on the Git repository, which can give you credit for your work.

Learn Things with Long Shelf Life

• Take courses on topics like quantum mechanics, linear algebra, and probability theory because they have a long shelf life compared to technology that may disappear within a few years.

The Future of AI

In this section, the speaker discusses the future of AI and how it will impact our lives.

Another Revolution of AI

• There will be another revolution of AI in the next few years.

• Self-supervised learning and other advancements will drive this revolution.

Need for Common Sense Machines

• We need machines with common sense to help us deal with the flow of information we're bombarded with on a continuous basis.

• Household robots that are as agile in terms of their mind as a house cat are needed.

Virtual Assistants

• We need virtual assistants that can mediate our interaction with the digital world.

• These assistants should have some level of common sense and understand how the world works.

Personal Projects and Interests

In this section, the speaker talks about his personal projects and interests.

Philosophy on Personal Projects

• The speaker only works on things that he finds fun.

• Work and personal projects aren't far away from each other.

Robotics Project

• The speaker started working on robotics as a personal project.

• He wanted to build a robot to do household chores like loading up laundry.

Hobbies and Interests

Yann shares his hobbies and interests, including sailing, astrophotography, building electronic music instruments, and using computer technology for music.

Yann's Hobbies

• Yann enjoys sailing and astrophotography.

• He wrote a PyTorch script for processing photos in astral photography.

• Yann builds electronic wind instruments with microcontrollers that talk through synthesizers.

• He has always been fascinated by the use of technology, especially computer technology, in music.

The Future of AI

Yann discusses his thoughts on the future of AI and how it is likely to be very exciting with another revolution in AI perhaps brought about by things like self-supervised learning.

The Future of AI

• Yann believes the future of AI is likely to be very exciting.

• Another revolution in AI may be brought about by things like self-supervised learning.

• He is looking forward to this future but hopes it happens before his brain turns into mush.

Conclusion and Call to Action

The host thanks Yann for being part of the session and encourages viewers to check out other talks from PyTorch Live.

Conclusion

• The host thanks Yann for being part of the session.

• Viewers are encouraged to check out other talks from PyTorch Live, including updates on the new PyTorch release, ecosystem updates, and Lin's vision talk.