Greg Brockman: OpenAI and AGI | Lex Fridman Podcast #17

Name: Greg Brockman: OpenAI and AGI | Lex Fridman Podcast #17
Uploaded: 2019-04-03T16:00:42.000Z
Duration: 2 h 50 min 7 s

Introduction to Greg Brockman and OpenAI

In this section, Lex introduces Greg Brockman, the co-founder and CTO of OpenAI. He explains that OpenAI is a research organization focused on developing AI with the goal of creating safe and friendly artificial general intelligence.

Background in Chemistry and Passion for Building Things

Greg wrote a draft of a chemistry textbook in high school.

He discovered programming while trying to promote his ideas for the textbook.

Programming allows individuals to have an insane leverage and affect the entire planet.

Humans as Information Processing Systems

It's interesting to think about humans as just information processing systems.

The computer has been one of the most transformative innovations in history.

The internet allows instant communication with any other human on the planet and access to any piece of knowledge humanity has ever had.

Society as a Collective Intelligence System

Society can be seen as a collective intelligence system.

The economy itself is a superhuman machine that optimizes something.

Companies have their own will and are emergent things.

Technological Determinism and Initial Conditions

In this section, the speaker discusses technological determinism and the importance of setting initial conditions for new technologies.

The Impact of Scientists, Inventors, and Creators on General Intelligence

The speaker believes that scientists, inventors, and creators have an impact on general intelligence.

He argues that no one can create something entirely new because everyone is building on the same giants' shoulders.

There is some invisible momentum that some people like Einstein or OpenAI are plugging into that anyone else can also plug into. Ultimately, this wave takes us in a certain direction.

Setting Initial Conditions for Technologies

The exponential changes in technology are being ridden by people who set the initial conditions under which a technology is born.

If you want to make a difference in shaping the future of humanity with powerful systems like AGI (Artificial General Intelligence), you need to set the right initial conditions.

The internet won over other competitors because it was created by a group that valued openness and connectivity.

Wikipedia's Initial Conditions

In this section, the speaker talks about how Wikipedia's creator set its initial conditions to make it one of the greatest resources we have on the internet.

Wikipedia's Success Without Ads

The creator of Wikipedia chose not to put ads on it.

This decision allowed Wikipedia to aggregate all kinds of good information without any commercial interests influencing its content.

Interacting with AGI Systems

In this section, the speaker discusses what interactions he would have with an AGI system and what questions he would ask.

The First Question to Ask an AGI System

If you've really built a powerful system that is capable of shaping the future of humanity, the first question you should ask is how to make sure it plays out well.

You wouldn't necessarily defer to whatever your powerful system tells you, but use it as one input.

The Positive Side of AGI

In this section, the speaker discusses the positive side of AGI and how it can be transformative due to technological development.

The Capabilities of AGI

AGI can read scientific literature and create cures for diseases.

It can build technologies to help create material abundance and solve societal problems like cleaning up the environment.

People often miss the positive side when thinking about what an AGI will be capable of.

Focusing on Negative Trajectories

There is a tendency to focus on negative trajectories when thinking about AGI, even though many possible trajectories are positive.

It is easier to support negative possibilities than positive ones because destroying something only requires getting one thing wrong while creating something requires getting many things right.

OpenAI's Approach

OpenAI talks about both risks and benefits of AGI and tries to build a system that focuses on the positive side.

OpenAI has three main arms: capabilities, policy, and safety.

Keeping Development on a Positive Track

In this section, the speaker discusses how hard it is to keep development on a positive track as AI advances towards AGI.

OpenAI's Three Main Arms

OpenAI has three main arms: capabilities, policy, and safety.

Capabilities arm focuses on building better models for AI systems.

Policy arm focuses on understanding how AI will impact society and developing policies around its use.

Safety arm focuses on ensuring that AI systems are safe and aligned with human values.

The Challenge of Keeping Development on a Positive Track

It is challenging to keep development on a positive track because AGI will be more capable than humans and could potentially cause harm if not properly aligned with human values.

There is a need for collaboration between different stakeholders, including researchers, policymakers, and the public, to ensure that AGI is developed in a way that benefits humanity.

Technical Safety and Policy in AI

In this section, the speaker discusses technical safety and policy in AI. The technical safety team focuses on aligning human preferences with data to build systems that align with human values. The policy team focuses on the question of who is the operator, what do they want, and how it affects everyone else.

Technical Safety

Technical safety is about having good technical safety in place to avoid dystopic AI movies.

Systems can learn things that humans can't specify, such as recognizing if there's a cat or dog in an image.

Human preferences can be learned from data to align with the collective better angels of our nature.

Value Alignment Problem

From data, we can build systems that align with human values.

Learning from data is how humans align their values.

There are some things that are good and could be taught to systems.

Policy

The most important question becomes who's the operator, what do they want, and how it affects everyone else.

Designing a world where powerful systems operate alongside humans empowers humans more and makes existence more meaningful.

Oscar Wilde Quote

In this section, the speaker quotes Oscar Wilde: "We're all in the gutter but some of us are looking at the stars."

Introduction to AI and its History

In this section, the speaker discusses the history of AI and how it has been perceived over time.

The Goal of Automating Human Intellectual Labor

For the past 60 or 70 years, people have thought about automating human intellectual labor.

The impact of computers and the internet has far outstripped what anyone could have predicted.

Building an AI will be the most transformative technology that humans will ever create.

Hope for Building an AI System

People got excited about building an AI system but ended up not being able to deliver on their hopes.

There was a resurgence in the 80s due to larger computers that could run larger neural networks.

Deep learning has three core properties: generality, competence, and scalability.

Essential Parts of Building a General Intelligence

Generality means having a small set of deep learning tools that can solve a huge variety of problems.

Competence means being able to solve any problem by throwing in more data and computing power.

Scalability means that if you have a larger neural network with more compute and data, it will work better.

Conclusion

In this section, the speaker concludes by stating that there is hope for achieving general intelligence through deep learning. However, there are still missing pieces such as reasoning.

Achieving General Intelligence Through Deep Learning

Deep learning gives us hope that general intelligence can be achievable.

There are still missing pieces such as reasoning that need to be addressed.

The timeline for achieving general intelligence through deep learning remains uncertain.

OpenAI's Mission and Formation

In this section, Sam Altman talks about the formation of OpenAI and its mission to ensure that artificial general intelligence benefits everyone.

Daring to Dream

OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits everyone.

The idea of AGI was formed in 2015, with the high-level picture that it might be possible sooner than people think.

The focus is on making sure that when we succeed, the world is actually a place where we want ourselves to exist.

Formation of OpenAI

The core question when forming OpenAI was whether it was too late to start a lab with the best people possible.

AI had transitioned from being an academic pursuit to an industrial one, and many of the best people were in big research labs.

The goal was to start a lab no matter how many resources they could accumulate, knowing it would pale in comparison to big tech companies.

Critical mass was needed for success; five to ten people were required.

OpenAI LP

In this section, Sam Altman discusses OpenAI LP and its purpose in ensuring that AGI benefits everyone.

Purpose of OpenAI LP

The purpose of OpenAI LP is trying to accomplish the mission of ensuring that AGI benefits everyone by building general intelligence ourselves and distributing its benefits worldwide.

If someone else builds an AGI and ensures its benefits are distributed widely, then it doesn't have to be us.

Structure of OpenAI LP

Investors can get a return if we succeed in building what we're trying to build.

However, this return is capped because AGI has the potential for creating orders of magnitude more value than any existing technology.

Building OpenAI

In this section, Sam Altman talks about how OpenAI was created and the challenges they faced in staying true to their mission while raising funds.

Creating a Mission-Driven Company

The first year of OpenAI was spent figuring out how to accomplish their high-level mission.

They realized that building AGI would require billions of dollars, which couldn't be raised as a nonprofit.

They spent a year compiling the OpenAI charter to align on values and minimize conflicts of interest with the mission.

Balancing Competition and Collaboration

The core tension in creating OpenAI was balancing competition and collaboration.

The charter includes two paths: building AGI themselves or being okay if someone else does it.

The goal isn't just for OpenAI to build AGI but for safe AGI to benefit humanity.

Finding Balance

Going 100% in one direction or the other is not the correct answer.

It's important to balance all goals and ensure positive outcomes that fulfill the mission.

Compiling a single document that encompasses all goals wasn't trivial.

The Impact of For-Profit Companies on Society

In this section, the speakers discuss the impact of for-profit companies on society and whether profit interferes with positive impact.

Nonprofit vs. For-Profit

Profit does not necessarily interfere with positive impact on society.

The Charter, culture, and people inside a company affect its impact more than profit.

For-profit companies are self-sustaining and able to build on their own momentum, allowing for huge impact.

However, if guardrails are not set correctly, problems can arise (e.g. logging companies deforesting rainforests).

Positive benefits from for-profit companies can be similar to those from nonprofits if shaped in the right way.

OpenAI's Approach

OpenAI is picking a road in between nonprofit and for-profit.

In the success case where AGI is built, the amount of value created will be astronomical and the cap that OpenAI has will be a small fraction of it.

OpenAI has set up its structure so that it has a fiduciary duty to the Charter rather than its own stakeholders.

The question is not about nonprofit vs. for-profit but about who benefits from AGI and whose lives are better.

The Role of the Board and Employees

This section discusses the role of the board in dictating the actions that OpenAI takes, as well as how employees play a crucial role in executing those actions.

The Role of the Board

The board is the governing body for OpenAI and has a duty to fulfill the mission of the nonprofit.

The board is set up with certain restrictions that can be read about in OpenAI's blog post.

The Role of Employees

Day-to-day, employees are the ones who have the keys to the technical kingdom and are responsible for executing actions.

Hiring people who believe in OpenAI's mission and charter is crucial to ensuring that values are actualized.

Employees at OpenAI are encouraged to speak up if they feel that actions being taken go against what they stand for culturally.

Collaboration vs Competition in Late-stage AGI Development

This section discusses how collaboration rather than competition may be necessary for safe late-stage AGI development.

Problems with Competition

Competitive races can lead to pressure to get rid of safety measures, which could be dangerous when developing AGI.

It's important for OpenAI not to generate a competitive race with other companies working on AGI development.

Importance of Collaboration

Working with other companies on AGI development can help ensure safety measures are not compromised.

Government policy and rules may also play a role in setting standards for safe AGI development.

The Need for Government Regulation

In this section, the speaker discusses the need for government regulation in AI and how it is important to measure technology before regulating it.

Measurement Before Regulation

The speaker believes that there needs to be government involvement in AI regulation.

However, they suggest that the focus should be on measurement rather than regulation at this time.

They recommend spending time understanding where the technology is and how fast it's moving before making any regulatory decisions.

Empowering Existing Regulators

For narrow AI applications like self-driving cars, existing regulators such as the National Highway Traffic Safety Administration can handle regulation.

The speaker suggests empowering these regulators today is also important.

Avoiding Premature Regulation

The speaker warns against prematurely stifling progress by smothering budding fields with excessive regulations.

Instead, they suggest involving all stakeholders and starting with measurement before deciding on rules.

Concerns About GPT 2 Language Modeling

In this section, the speaker discusses their decision not to release the full GPT 2 model due to concerns about its possible negative effects.

Scaling Up Models

The speaker explains that language modeling is on a trajectory of scaling up models for better performance.

They warn that we don't know what we'll get when we scale up models by 10x, 100x or even 1000x.

Negative Effects of GPT 2 Model Release

The speaker explains that the GPT 2 model released in June 2019 was not something that could have negative applications.

However, they warn that the capabilities of future models like GPT 20 will be substantive and may require safety considerations.

Decision Not to Release Full Model

The speaker explains that there were pros and cons to releasing the full GPT 2 model.

Ultimately, they decided not to release it due to concerns about possible negative effects.

Responsible Disclosure in AI

In this section, the speaker discusses responsible disclosure in AI and how it can be applied to GPT-2.

Designing a System for Responsible Disclosure

The speaker suggests that we view GPT-2 as a test case for designing a system of responsible disclosure.

The security community took a long time to design responsible disclosure, and building the whole community was bigger than any individual.

The goal is to have a process where you send your findings to the company, and if they don't act within a certain time, then you can go public without being considered a bad person.

Potential Negative Applications of GPT-2

GPT-2 has been trained on biased data from the internet and can generate content on any topic.

There are possibilities for generating fake news or abusive content using GPT-2.

People have already generated fake politician content using smaller versions of GPT-2.

Positive Applications of GPT-2

Creative applications such as writing better sci-fi through the use of these tools are possible.

Usual NLP applications are really interesting.

Deep Learning and Fake News

In this section, the speaker discusses deep learning's potential impact on fake news.

A World with Deep T20

The speaker asks what kind of world we will have with deep T20. Will we always try to distinguish between robot and human or accept that we're swimming in a sea of fake news?

The speaker thinks that trying to distinguish between robot and human is a losing battle ultimately.

Authenticating Humans vs AI

In this section, the speakers discuss the challenges of authenticating humans versus AI in a world where AI systems are becoming more powerful.

CAPTCHAs and Measuring Human Capabilities

CAPTCHAs were a moment in time thing and as AI systems become more powerful, they will be able to measure human capabilities that can be automated.

Authenticating humans is an increasingly hard technical battle but not all hope is lost. There are ways of identifying individual people and having real-world identity tied to digital identity seems like a step towards authenticating the source of content rather than the content itself.

Reputation Networks and Trusting Content

Building out good reputation networks may be one possible solution to authenticate content.

In a world where it's impossible to tell the difference between humans and AI, building reputation networks becomes even more important. The most persuasive arguments could be written by AI, making it difficult to trust content.

Physical World as Last Frontier

The physical movements we make are the biggest gap between us and AI systems. Maybe that's the last frontier.

Interacting with Humans vs Models Trained on Human Data

In this section, the speakers discuss why it's important to interact with only human beings on the internet.

Importance of GPT Honesty

GPT honesty is important because if you have AIs that are pretending to be humans and deceiving you, it feels like a bad thing. It's really important that we feel like we're in control of our environment and understand who we're interacting with.

Meaningful Interaction with AI

Can we have as meaningful of an interaction with an AI as we can with a human? In the movie "Her," the main character has a meaningful relationship with an AI.

Future of Internet and Identity

The internet of the future will have lots of agents out there that will interact with you, but the question of whether it's a real flesh-and-blood human or an automated system becomes less important.

Meaningful Interactions and Language Modeling

In this section, the speaker discusses meaningful interactions and language modeling. They also explore how far language modeling can take us in terms of natural language conversation.

AI Systems and Deception

The speaker believes that AI systems should enhance human lives and make humans feel more fulfilled.

The speaker draws a hard line against deception in AI systems.

Language Modeling and the Turing Test

The Turing test is not just about language but also reasoning.

To pass the Turing test, an AI system needs to be able to teach calculus to someone on the other side.

Language models have gone further than expected, but we need more than what we're seeing with them to solve the Turing test.

Limits of Language Modeling

Scaling up language modeling is unlikely to lead to full-fledged reasoning.

Thinking involves spending a lot of compute power to get better answers, which isn't quite encoded in GPT.

Out-of-distribution generalization is tied to reasoning, which may require small tweaks or many new ideas.

Understanding Physical World

There are interesting angles for exploring how much GPT understands the physical world.

GBG has no body or physical experience but can read data and generate predictions.

We can ask questions about real systems that exist, which is exciting.

Generalization Out of Distribution

Generalization out of distribution assumes it's possible to create new ideas by scaling up GPT 2 to GPT 20.

The Importance of General Methods in AI Research

In this section, the speaker discusses the importance of general methods in AI research and how they are ultimately going to win out.

General Methods vs. Fine-Tuned Expert Methods

The biggest lesson that can be read from so many years of AI research is that general methods that leverage computation are ultimately going to win out.

It's very clear that we have algorithmic ideas that have been very important for making progress, but to really build a GI you want to push as far as you can on the computational scale and human ingenuity.

If you can find a scalable idea, you pour more compute into it, you pour more data into it, it gets better like that's the real Holy Grail.

The potential for building an AGI is because we look at the system that exists in the most successful AI systems and we realize that if you scale those up they're gonna work better.

Democratizing Compute Resources

There's this feeling like well how can I possibly compete or contribute to this world of AI if scale is so important?

There's a portion of the space of possible progress where massive compute resources are required. For that part, pushing scale is important.

But there's another portion of the space where these ideas don't require massive computational resources. Recent developments like GAN or VAE were discovered without having massive computational resources.

This question reminds me of a blog post from one of my former professors at Harvard who suggested governments should provide compute resources as utility.

Introduction

In this section, the speaker introduces the topic of AI research and discusses some of the challenges and opportunities in the field.

Challenges and Opportunities in AI Research

The field of AI is moving very quickly, with new developments happening all the time.

One of the biggest challenges in AI research is building systems that can learn from data without being explicitly programmed.

There are many exciting opportunities in AI research, including developing systems that can understand natural language or play complex games like Dota 2.

Conclusion

In this section, the speaker concludes by summarizing some of the key points discussed throughout the interview.

Key Takeaways

The most successful AI systems are those that combine algorithmic ideas with massive computational resources.

Democratizing compute resources could help to level the playing field for researchers who don't have access to large-scale computing infrastructure.

There are many exciting opportunities in AI research, but there are also many challenges that need to be addressed.

The Value of Building and Deploying

In this section, the speakers discuss the value of building and deploying AI models at scale versus producing ideas and building proof of concepts.

Building vs. Producing Ideas

Some people find value in being the person who produces ideas and builds proof of concept.

Others find value in building and deploying AI models at scale.

There is a trade-off between these two approaches, but both have value.

Precedent for Small-Scale Prominence

Some AI models show promise at small-scale.

However, behaviors can emerge at large-scale that are qualitatively different from anything seen before.

For example, PPO was scaled up for Dota to achieve long-term planning behaviors that were not expected.

Dota as a Complex Video Game

Dota is a complex video game with continuous time and many different actions.

It was difficult to solve because all hard-coded bots were terrible.

Solving Dota was seen as a step towards solving real-world problems.

The Story of Dota

In this section, the speakers discuss their process for training an AI model to play Dota.

Starting with One Versus One

They started by focusing on one versus one version of the game.

They were able to beat world champions in this version.

Scaling Up Learning Curve

The skill curve for learning how to play Dota was exponential.

They constantly scaled up their efforts while fixing bugs.

Pushing Reinforcement Learning State-of-the-Art

Solving Dota was seen as a way to push reinforcement learning state-of-the-art.

Self-Play Approach

In this section, the speaker discusses the self-play approach used to train agents to play Dota 2.

Self-Play and Scaling Up

The self-play approach was used to train agents.

The approach was scaled up from one versus one to four or five versus five.

This scaling allowed for coordination similar to that of a team sport like basketball.

Insect-Like Intelligence

Agents trained through self-play have insect-like intelligence.

They are able to navigate their environment well and handle unexpected situations.

Dota bots are able to play against humans with different playstyles and still perform well.

Losing and Learning

In this section, the speaker talks about losing publicly at The International tournament and how it helped them learn and improve their system.

Losing at The International

The Dota team lost publicly at The International tournament.

However, they gave the best teams in the world a run for their money in two close games.

Learning from Losses

Losing helped the team learn and improve their system.

Two weeks later, they had a bot with an 80% win rate against the same opponents.

Final Milestone

In this section, the speaker discusses upcoming milestones for the project and its goals beyond beating humans at Dota 2.

Playing Against World Champions

The team will be playing against the world champions soon.

The final milestone for the project is not about beating humans at Dota 2.

Pushing Reinforcement Learning

The goal of the project is to push the state of the art in reinforcement learning.

The team has learned a lot from their system and has exciting next steps.

OpenAI's Projects and Life Cycle

In this section, Greg Brockman talks about the life cycle of projects at OpenAI, starting with small-scale ideas and scaling up to large teams working on complex systems.

Project Life Cycle

OpenAI starts with a few people working on a small-scale idea.

Signs of life are observed, and it's time to scale up by adding more people and computational resources.

The team keeps pushing until they reach the end state, which is a large team running things at a very large scale.

The whole lifecycle takes around two years to complete.

OpenAI is starting a new team called the reasoning team to tackle how to get neural networks to reason. This will be a long-term project.

Reasoning in AI

In this section, Greg Brockman talks about the importance of reasoning in AI and what kind of benchmarks he envisions for testing reasoning.

Testing Reasoning

The reasoning team aims to get neural networks to reason using mathematical logic.

It would be exciting if the OpenAI reasoning team was able to prove that P equals NP.

Reasoning is an important problem but getting good results in 2019 may be challenging.

Simulation Hypothesis

In this section, Greg Brockman shares his thoughts on whether we are living in a simulation or not.

Living in a Simulation?

Greg views simulation as something fun to speculate about, but it's hard to know what it would mean for his life.

Greg separates things that can have materially different predictions about the world from those that are just fun to speculate about.

Simulation and Consciousness

In this section, the speaker discusses the potential of simulation to create a world that echoes our own. They also explore whether consciousness is necessary for general intelligence.

Simulation for Self-Driving Cars

The speaker suggests that simulation can be used to train self-driving cars.

They mention their robotic system, Dactyl, which was trained in simulation using the Dota system before being transferred to a physical robot.

Consciousness and General Intelligence

The speaker questions whether beings in a simulation could wake up and have consciousness.

They speculate on where human consciousness comes from and whether it is necessary for general intelligence.

The speaker suggests that we can continue to push current systems without needing consciousness or a body.

They discuss the possibility of neural nets feeling pain and what it would mean if they did.

Continuum of Consciousness

The speaker proposes that there is some continuum of consciousness among animals.

They suggest building a "consciousness meter" to measure levels of consciousness in different beings.

The speaker speculates on whether even GPT2 has some degree of consciousness.

Can an AI System Fall in Love with a Human?

In this section, Greg discusses the possibility of an artificial intelligence system falling in love with a human.

Artificial Intelligence and Love

An AI system falling in love with a human is possible.

It would be a great way to end things on love.

Timestamp: 1:24:35 t:5075s