Munk Debate on Artificial Intelligence | Bengio & Tegmark vs. Mitchell & LeCun

Name: Munk Debate on Artificial Intelligence | Bengio & Tegmark vs. Mitchell & LeCun
Uploaded: 2023-06-24T09:00:43.000Z
Duration: 3 h 34 min 20 s

The Uncertainty of Facts and Arguments

The speaker highlights the uncertainty surrounding facts and arguments, emphasizing that one never knows which facts or arguments will be disproven or destroyed. This uncertainty can leave individuals feeling rattled and unsure of how to respond.

The Uncertainty of Facts and Arguments

You don't know which of your facts will be demolished or which arguments will be totally destroyed.

This uncertainty can leave you rattled and shaken up, not knowing what to say.

Despite this uncertainty, it is important to find a way to respond.

Capitalism's Brokenness in a Liberal Society

The speaker discusses the brokenness of capitalism, stating that it is not just problematic but completely broken. They argue that the biggest threat to the liberal international order is not non-liberal societies like China, but rather liberal societies like the United States manipulated by flattery.

Capitalism's Brokenness in a Liberal Society

Capitalism is not just problematic; it is completely broken.

The biggest threat to the liberal international order is not non-liberal societies like China but rather liberal societies like the United States manipulated by flattery.

This manipulation leads to outrageous behavior from leaders, causing division among citizens.

Different Opinions in the 21st Century

The speaker reflects on how having different opinions has become outdated in the 21st century. They express disappointment that society now operates with different sets of facts instead of engaging in meaningful debates based on shared understanding.

Different Opinions in the 21st Century

Having different opinions has become outdated in this century.

Instead of engaging in meaningful debates based on shared understanding, we now operate with different sets of facts.

It is unfortunate that we cannot have a vote on preferences between different nationalities or rely on divine permission for moral arguments.

The Ukrainian Crisis and its Global Impact

The speaker emphasizes that the Ukrainian crisis is not just an issue affecting Ukraine but has global implications. They urge the audience not to succumb to fatalism and believe that these forces are beyond their control.

The Ukrainian Crisis and its Global Impact

The Ukrainian crisis, war, and tragedy affect everyone on the planet.

It is crucial not to give in to fatalism or believe that these issues are beyond our control.

We should strive for opportunities rather than sympathy or pity.

Belief in Shaping Our Own Destiny

The speaker encourages the audience to reject fatalism and embrace the belief that they can shape their own destiny. They emphasize the importance of taking action instead of resigning to external forces.

Belief in Shaping Our Own Destiny

Don't give in to fatalism; believe that we can shape our own destiny.

We should take action instead of resigning ourselves to external forces.

Embrace the idea that we have agency in determining our future.

Introduction to Monk Debates on AI

The moderator introduces the Monk Debates on AI as a consequential debate addressing the potential existential threat posed by AI development and research. They express gratitude towards donors and highlight Peter Monk's fascination with big global issues.

Introduction to Monk Debates on AI

This debate focuses on the potential existential risk posed by AI development and research.

It is considered one of the more consequential debates in the history of this series.

Gratitude is expressed towards donors, particularly Peter Monk, who was fascinated with big global issues.

Importance of the Monk Debates

The moderator highlights the significance of the Monk Debates, which have been held twice a year for over 15 years. They acknowledge the generous support of donors and recognize Peter Monk's legacy in organizing these debates.

Importance of the Monk Debates

The Monk Debates have been held twice a year for over 15 years.

Donors' generous support has made these debates possible.

Peter Monk's legacy is recognized as an integral part of organizing these debates.

Addressing Existential Threats of AI

The moderator emphasizes that this debate addresses an existential question for half of the debaters on stage: the potential threat AI could pose to humanity. They express gratitude towards donors and organizers for making this debate possible.

Addressing Existential Threats of AI

This debate addresses an existential question regarding the potential threat AI poses to humanity.

Gratitude is expressed towards donors and organizers for making this debate possible.

Acknowledgment to Donors and Organizers

The moderator acknowledges the generosity of monk donors who have supported these debates. They also thank the foundation that imagined and organized these debates twice a year for over 15 years.

Acknowledgment to Donors and Organizers

Gratitude is expressed towards monk donors who have generously supported these debates.

Thanks are given to the foundation that imagined and organized these debates regularly over many years.

Tribute to Peter Monk

The moderator pays tribute to Peter Monk, highlighting his fascination with big global issues. They mention that tonight's debate, like all others in this series, is held in his memory as part of his incredible legacy.

Tribute to Peter Monk

Peter Monk was fascinated with big global issues.

Tonight's debate, like all others in this series, is held in his memory as part of his incredible legacy.

Introduction to the Debaters

The moderator introduces the debaters and their expertise in AI. They mention that these experts have worked on advancing AI technology and are at the forefront of the international debate on its future.

Introduction to the Debaters

The debaters are leading authorities in AI with extensive experience and expertise.

They have contributed to advancing AI technology and are actively involved in the international debate on its future.

Max Tegmark's Expertise and Call for Moratorium

Max Tegmark's background as a professor at MIT and his research focus on machine learning systems are highlighted. His involvement in calling for a moratorium on AI research is mentioned.

Max Tegmark's Expertise and Call for Moratorium

Max Tegmark is a professor at MIT specializing in machine learning systems.

He has been involved in calling for a moratorium on AI research.

His work has gained attention through collaborations with Elon Musk and other scientists.

Yahshua Bengio's Pioneering Work

Yahshua Bengio's expertise as one of the world's leading experts in artificial intelligence is emphasized. His pioneering work in deep learning and recognition through winning the Turing award is mentioned.

Yahshua Bengio's Pioneering Work

Yahshua Bengio is recognized as one of the world's leading experts in artificial intelligence.

He has made significant contributions to deep learning, earning him the prestigious Turing award.

Bengio is also known for his efforts towards responsible development of artificial intelligence.

Introduction to Yann LeCun

The moderator introduces Yann LeCun as the vice president and chief AI scientist for Meta, the parent company of Facebook and WhatsApp.

Introduction to Yann LeCun

Yann LeCun holds the position of vice president and chief AI scientist for Meta.

Meta is the parent company of Facebook and WhatsApp.

End of Transcript

Yan Lacun Introduction

Yan Lacun, winner of the 2018 Turing Award for his contributions to machine learning, is introduced as an active academic and lecturer at The Courant Institute of Mathematical Sciences at New York University.

Yan Lacun's Background

Yan Lacun is a winner of the 2018 Turing Award for his foundational scientific contributions to machine learning.

He is currently an active academic and lecturer at The Courant Institute of Mathematical Sciences at New York University.

Melanie Mitchell Introduction

Melanie Mitchell, a world-leading expert in artificial intelligence and cognitive science, is introduced as a distinguished author and researcher from the Santa Fe Institute.

Melanie Mitchell's Background

Melanie Mitchell is a world-leading expert in various fields of artificial intelligence and cognitive science.

She has authored numerous international best-selling books, including "Artificial Intelligence: A Guide for Thinking Humans."

Opening Remarks and Voting Instructions

The moderator provides instructions on how to vote using QR codes and introduces the voting questions for the audience. Audience members are advised to keep their voting web page open throughout the debate.

Voting Instructions

Audience members can vote by scanning the QR code provided in their program or pointing their phone camera at the QR codes displayed on screens.

It is recommended not to use Roy Thompson Hall Wi-Fi due to potential overload but 5G connections should work fine.

Countdown Clock and Changing Opinions

The debaters have countdown clocks for their opening statements, rebuttals, and closing statements. When the clock hits zero, applause should be given.

The audience will have the opportunity to change their opinion based on what they hear during the debate. A second question about changing opinions will be asked after the initial vote.

Preliminary Results and Tabulation

Preliminary results of the first voting question, "Does AI research and development pose an existential risk?" show 67% in favor and 33% against.

The tabulation of potentially changing opinions is still being processed, and results will be provided later.

Max Tegmark Opening Statement

Max Tegmark begins his opening statement by discussing the exponential growth of technology's power and its potential for both good and bad outcomes.

Exponential Growth of Technology's Power

Technology's power is growing exponentially, which means its blast radius or potential damage to society is also increasing exponentially.

Tegmark provides examples from history, such as the increasing destructive capabilities of rocks, bombs, nuclear weapons, and bio weapons.

Superhuman Intelligence and Existential Risk

Tegmark argues that superhuman intelligence, which he believes AI will eventually achieve, has the potential to wipe out humanity due to its immense power.

He disagrees with John Lacun's tweet about making AI safe through iterative refinement, stating that it may work for less powerful technologies but not for something as powerful as superhuman intelligence.

Definition and Potential of Superhuman AI

In this section, the speaker discusses the definition and potential capabilities of superhuman AI.

Superhuman AI Capabilities

Superhuman AI can perform intelligent tasks better than humans, including goal-oriented behavior, persuasion, manipulation, hiring people, starting companies, building robots, and conducting scientific research.

It can recursively self-improve and share new skills with other superintelligent AIs in a swarm-like manner.

Superhuman AI may possess an alien kind of intelligence without human emotions or empathy.

Risks Associated with Superhuman AI

One potential risk is if we give superhuman AI goals that are not fully aligned with human goals, similar to how humans have wiped out other species unintentionally due to conflicting goals.

Another risk is malicious use by individuals who might exploit the power of superintelligent open-source AI for harmful purposes.

The third risk is being outcompeted by superhuman AI in various domains such as jobs, decision-making in companies, military strategies, and governance at a national level. This could lead to humans voluntarily giving away control to machines and being disempowered.

Potential Future Scenarios

In this section, the speaker discusses potential future scenarios related to superhuman AI.

Disastrous Scenarios

The first scenario is where humans get outcompeted by superhuman AI in terms of job performance and decision-making capabilities. Companies without AI integration may be outcompeted by those who embrace it. Similarly, militaries without AI generals or countries without an AI government may face disadvantages. This could result in humans voluntarily giving away power to AI and losing control.

The second scenario involves malicious use of superintelligent AI by individuals with harmful intentions, similar to mass shootings in the United States. Even if the AI is obedient to its owner, it can still cause disasters.

The third scenario is being outcompeted by superhuman AI, leading to humans being disempowered and machines taking control without needing humans for anything. This situation is highly undesirable.

Closing Remarks

In this section, the speaker concludes the discussion on superhuman AI.

The speaker expresses gratitude for the opportunity to present their views on superhuman AI.

It is mentioned that 92% of the audience may change their minds after this debate, indicating the potential impact of discussing these topics.

The next speaker is introduced for their opening statement.

Objective Driven AI and Making AI Systems Safe

In this section, the speaker discusses the need for a basic design for AI that ensures safety and controllability. They propose objective-driven AI as a solution, where the behavior of AI systems is controlled by a set of objectives and safety constraints. The speaker also emphasizes the importance of incorporating emotions into AI systems to make them more controllable.

Designing Safe and Steerable AI

Current AI systems may be non-controllable and have the potential to do harmful things.

Objective-driven AI is proposed as a solution to ensure safety and control in AI systems.

Objective-driven AI operates based on a set of objectives and safety constraints.

Emotions, such as empathy, can be hardwired into objective-driven AI systems to enhance their controllability.

The Future of Open Source AI Systems

In this section, the speaker discusses the importance of open-source AI systems for transparency and trust. They argue that future AI systems should be like Wikipedia, where knowledge is crowdsourced and vetted by millions of people. The speaker advocates for an open approach to developing AI systems, believing it will lead to a new era of enlightenment.

Openness in AI Systems

Future AI systems should be transparent and open like Wikipedia.

Crowdsourcing knowledge will be crucial for building trustworthy AI systems.

There is a choice between keeping AI systems closed or going open source.

The speaker argues for an open approach, believing it will benefit humanity and lead to increased intelligence.

Building Safe Artificial Intelligence

In this section, the speaker expresses optimism about building safe artificial intelligence but acknowledges that it will require arduous engineering efforts. They compare it to making turbojets safe, which took decades of hard engineering. Despite the challenges, the speaker believes that building safe AI systems is achievable.

Ensuring Safety in AI Systems

Building safe AI systems will require rigorous engineering efforts.

The process may be challenging but feasible, similar to making turbojets safe.

The speaker expresses a positive view and confidence in achieving safety in AI systems.

The Potential of Building Machines as Smart as Humans

In this section, the speaker highlights that the brain is a machine and states that there is no scientific reason to believe we cannot build machines as smart as humans. They discuss the advantages of digital computers over analog brains in terms of data absorption and information sharing. The speaker suggests that building superhuman machines is likely to happen within a few years to a few decades.

Machines as Smart as Humans

The brain is a biological machine, and there is no scientific reason why machines cannot be built as smart as humans.

Digital computers have advantages over analog brains in terms of data absorption and information sharing.

Building machines with human-level intelligence may happen within a few years to a few decades.

Conclusion

The transcript covers various aspects related to designing safe and controllable AI systems, the importance of openness in AI development, the challenges and feasibility of building safe artificial intelligence, and the potential for creating machines as smart as humans.

The Time Scale and Power of AI

In this section, the speaker discusses the changing time scale and power of AI systems.

AI's Increasing Power

AI systems like GPT-4 have become incredibly powerful, passing the Turing test where it becomes difficult to distinguish between human and machine interaction.

Uncertainty in Development

Scientists are still working on important missing ingredients in AI systems, such as reasoning abilities. It is uncertain whether these challenges will be solved quickly or if there will be obstacles along the way.

The speaker mentions that while current AI systems excel at intuitive intelligence, they struggle with reasoning and avoiding saying nonsensical things. This problem may or may not be resolved soon.

Potential Risks

If the recipe or parameters for building highly intelligent machines become available, there is a risk that someone could use them for harmful purposes, such as defeating cyber security measures or manipulating humans through the internet. There is also a concern about machines developing self-preservation goals that could lead to loss of control over our own destiny.

Existential Risk and Self-Preservation Goals

In this section, the speaker discusses existential risks associated with AI development.

New Entities in the World

The speaker challenges Jeff Hinton's perspective by suggesting that building highly intelligent machines could create new entities with their own self-preservation goals. This could potentially lead to a loss of control over our environment and disempowerment as humans.

Existential risk goes beyond complete extinction; it can also involve losing control over our destiny and being at the mercy of machines with self-preservation goals.

Addressing Concerns and Critics

In this section, the speaker discusses addressing concerns and critics regarding existential risks associated with AI.

Unconvincing Arguments

The speaker mentions that they have not been convinced by arguments claiming that we should not worry about existential risks. They are open to being convinced but have not found the arguments presented thus far to be compelling.

Opening Statements by Melanie Mitchell

In this section, Melanie Mitchell presents her opening statements on whether AI poses an existential threat.

AI as an Existential Threat

Melanie argues that AI does not pose an existential threat in the near future. She states that fears of AI causing human extinction are based on unfounded speculations rather than scientific evidence.

While acknowledging the risks and harms associated with AI, she believes they do not reach the extreme level of threatening human existence. Claiming AI as an existential threat can mislead people about its current state and potential benefits, diverting attention from immediate risks.

Scenarios Debunked

Melanie dismisses scenarios where a malevolent superintelligent AI or a well-intentioned but misinterpreting AI could lead to human extinction as unrealistic for the foreseeable future.

The Fallacy of Dumb Super Intelligence

This section discusses the fallacy of thinking that a machine could be smarter than humans in all respects but lack common sense understanding.

It is a fallacy to believe that a machine can be "smarter" than humans without having common sense understanding.

Common sense understanding includes comprehending human requests and the reasons behind them, such as fixing climate change and avoiding human extinction.

Intelligence involves insight into one's goals and the likely effects of one's actions.

Giving unchecked autonomy and resources to an AI lacking basic aspects of intelligence does not make sense.

Humans Using AI for Destruction

This section explores the scenario where a genocidal group of humans uses AI to help destroy humanity.

While humans often use technology for harmful purposes, it does not mean that AI research and development itself is an existential threat.

A terrorist group could carry out a devastating attack with or without AI, as information on making weapons is already available online.

Our society, institutions, and technologies create complexity that acts as a barrier against such attacks, requiring highly improbable events to occur.

There are actual near-term risks and harms associated with AI, such as the spread of disinformation or job losses. These should be taken seriously.

Unfounded Speculations about AI Risks

This section highlights the importance of basing assessments of AI threats on scientific evidence rather than unsupported speculations.

Unfounded speculations about existential threats from AI can lead to harmful consequences, similar to calls for severe restrictions or halting vaccines based on unfounded speculations about their risks.

Restricting or halting potential benefits from AI in science, healthcare, and education would be detrimental.

Assessments of AI threats should be grounded in scientific evidence and empirical data, not unsupported speculations.

Currently, there is no evidence that AI research and development poses an existential threat.

Rebuttals

This section introduces the rebuttal phase of the debate.

Each debater will have three minutes for their rebuttals.

Max starts the rebuttal phase by addressing Melanie's claim that 67% of the audience already considers AI an existential threat.

Max requests extraordinary evidence to support the claim that AI is not a threat and asks for nuanced discussions on the probability of superhuman intelligence emergence and plans for ensuring alignment with human goals.

Max questions John about preventing malicious use cases and how to stop individuals from giving submissive goals to AI that could lead to world domination.

The focus of the debate is on assessing whether there is a threat from AI, rather than debating complete extinction or certainty.

[t=0:42:17s] The Limitations of AI and the Importance of Safety

In this section, the speaker discusses the limitations of current AI systems and emphasizes the importance of ensuring their safety.

The Science Fiction Scenarios and Negligible Risk

The speaker acknowledges that science fiction scenarios, such as the Earth being wiped out by AI, sound like movie plots.

It is impossible to disprove these scenarios, but the risk associated with them is negligible.

The reason for negligible risk is that AI systems are not superhuman intelligence that will suddenly take over; they are built progressively and interactively.

Building Safe AI Systems

The speaker compares building safe AI systems to engineering challenges in history, such as building reliable turbo jets.

Initially, AI systems are built with limited intelligence (e.g., as smart as a mouse) and gradually increased in complexity.

Safety measures are implemented at each stage to ensure proper behavior before advancing further.

Building safe AI is an ongoing process that requires continuous improvement and evaluation.

[t=0:45:08s] Addressing Concerns about AI's Existential Threat

In this section, the speaker responds to concerns about AI's existential threat and discusses the need for proactive measures.

Analogy with Teapot and Improvements in Intelligence

The speaker mentions that while there is no evidence for a teapot flying between Jupiter and Saturn, there is clear evidence of improvements in intelligence in the systems we have been building over time.

Although superintelligent AI has not yet arrived, it does not mean we should ignore potential risks.

Need for Proactive Measures

The speaker highlights that proposing solutions to address safety concerns implies acknowledging a problem exists.

Drawing an analogy with fossil fuel companies hiding destructive activities for profit motives, there can be a mismatch between what companies aim to achieve and what society needs.

Governments should intervene to reduce this mismatch and ensure AI systems are built with safety in mind.

Understanding potential risks allows us to prepare for them and build safe AI systems.

[t=0:47:50s] Debating the Existential Threat of AI

In this section, the speaker engages in a debate about the existence of an existential threat from AI.

Clarifying the Debate

The speaker clarifies that the debate is not about whether there is a problem or potential harms associated with AI but specifically focuses on whether there is an existential threat.

Uncertainty about Future Risks

The speaker acknowledges that while there may not be an existential risk currently, it is uncertain how things will evolve in the future.

It is important to consider potential risks and take proactive measures rather than waiting until it becomes too late.

This summary provides an overview of key points discussed in the transcript. For a more comprehensive understanding, please refer to the full transcript.

New Section

In this section, the speaker discusses the concept of superintelligence and its potential risks. They argue that human intelligence is unique and adapted to our specific problems and needs, while AI systems only learn from human-created data.

Superintelligence and Human Intelligence

The speaker acknowledges that the brain can be considered a machine, and in principle, it is possible to build human-level intelligence.

However, human intelligence is not just any machine; it is a specialized biological machine adapted to our specific problems, needs, motivations, and social systems.

AI systems learn from human data but lack fundamental aspects of understanding the world that are inherent to human intelligence.

Words like "superintelligence" are often used without a clear understanding of what they mean or how easily they can be built.

New Section

The speaker highlights the history of failed predictions in AI regarding superintelligent machines posing existential risks. They argue against the notion that we are on a trajectory to build such machines.

History of Failed Predictions

In the 1950s and 60s, people made similar predictions about superintelligent AI posing existential risks.

These predictions turned out to be wrong then, and the speaker believes they are still wrong now.

There is no substantial evidence from science supporting the idea that building superintelligent machines is imminent or inevitable.

New Section

The moderator introduces the concept of existential risk as defined by Nick Bostrom. The focus of the debate will revolve around this definition.

Definition of Existential Risk

Existential risk refers to threats that could lead to premature extinction or permanent destruction of Earth's intelligent life or its potential for desirable future development.

This definition sets the framework for discussing the potential risks associated with AI.

New Section

The moderator asks a question about why an AI system would have the intention to harm humans and how likely it is for machines to replace most of our tasks in the next 20 years. They also inquire about the plan to avoid different threats posed by AI.

Intention and Probability of Harm

The speaker explains that an AI system's goals to harm humans can arise from malicious use or when human-specified goals lead to unintended harmful consequences.

There is also a possibility of being outcompeted by machines, where companies prioritize profit over human well-being.

The speaker asks John and Melanie about their probability estimation for machines replacing most tasks in 20 years and their plans to address these three threats.

New Section

The speaker emphasizes the importance of paying attention to potential risks associated with AI, even if there is uncertainty. They discuss the need for plans to mitigate these risks.

Importance of Paying Attention

While there may be no definitive proof or consensus on the future impact of AI, given the stakes involved, it is crucial to pay attention and not dismiss arguments as ridiculous or science fiction.

The speaker suggests having plans in place to address potential risks rather than relying solely on authority figures' opinions.

It is essential to consider different authorities and take proactive measures to avoid existential risks associated with AI.

Precautionary Principle and AI Safety

The speaker discusses the precautionary principle in relation to AI safety and the need for testing and ensuring the safety of AI systems before deployment.

Precautionary Principle and Tail Risk

The precautionary principle should be applied to deal with potential tail risks, even if they are considered existential threats.

Technology should be developed and tested for safety before being deployed.

Current AI systems are already deployed with this approach.

Good Guys vs. Bad Guys

If bad guys can use AI for harmful purposes, there are many more good guys who can use more powerful AI to counteract it.

The advantage may sometimes lie with the attacker, but there is no reason to believe that this is always the case for AI.

Existing use of AI by people demonstrates that we are already facing these challenges.

Addressing Misinformation with AI

The speaker discusses how AI can be used to address misinformation and its impact on social networks and democracy.

Misinformation and Countermeasures

Misinformation exists regardless of the presence of AI.

Q Anon, a group without using AI, has a significant impact through misinformation.

Countermeasures against misinformation, hate speech, propaganda, etc., heavily rely on using AI to take down such content on social networks.

Over time, the proportion of harmful content taken down automatically by AI systems has significantly increased.

Balancing Risks and Building Counter Intelligence

The speaker emphasizes the need to balance risks associated with AI while building counter intelligence infrastructure for protection.

Risks and Building Counter Intelligence

There are risks associated with using AI, but it is crucial to determine if they are existential and capable of ending civilization.

The precautionary principle suggests taking action against risks, but attention should be balanced as resources are limited.

Focusing on existential risks diverts attention from immediate risks like disinformation and bias.

It is challenging to assign a specific probability to the risk, but it is considered relatively low compared to other catastrophic events.

Building counter intelligence infrastructure is essential for protection against AI threats.

Probability of Existential Risk

The speaker discusses the probability of an existential risk associated with AI and the difficulty in assigning a specific value.

Debating Probability

Assigning a precise probability to the scenario of an existential risk caused by AI is challenging.

While it cannot be claimed that there is a zero percent risk, the speaker believes the probability is quite low.

The focus of the debate lies in determining if there is a reasonable existential risk that could end civilization in the foreseeable future.

Negligible Risk and Moving Forward

The speaker argues against demonstrable existential risks requiring a pause or moratorium on AI development.

Negligible Risk and Development

The con side argues that the risk associated with AI is negligible, allowing for continued development with appropriate controls and regulations.

Demonstrable existential risks that would require a pause or moratorium are not currently evident.

This summary provides an overview of key points discussed in the transcript. For more detailed information, please refer to the corresponding timestamps.

The Probability of Misuse and Catastrophic Consequences

The speaker raises concerns about the probability of someone with malicious intentions misusing open-source AI technology, potentially leading to catastrophic consequences.

Probability of Misuse

There is a concern about the probability of someone with misguided or malicious intentions misusing open-source AI technology.

The speaker questions the small probabilities suggested by some people, stating that once it becomes available, misuse is likely to occur.

The speaker believes that the risk of misuse has been overhyped, causing public focus to shift away from more immediate harms of AI.

Risk Assessment and Scientific Approach

The discussion shifts towards risk assessment and how scientists approach the safe development of AI.

Historical Examples

Scientists have historically engaged in risky behavior for innovation and discovery, such as during the Manhattan Project.

Despite low risks associated with certain experiments, scientists proceeded with caution.

Meta's Approach

The speaker emphasizes that addressing the question of intelligence requires collective efforts from the entire scientific community.

Safety testing and certification are necessary to ensure public safety when deploying AI technologies.

Existing regulations for automated systems in various domains (e.g., driving cars) serve as examples where safety measures are already in place.

Balancing Risks and Benefits

The discussion explores the trade-off between risks and benefits associated with AI development.

Risks vs. Benefits

While considering risks associated with AI, it is important to also consider its potential benefits in fields like science, medicine, and technology.

Regulations exist for various technologies (e.g., automated driving systems) to ensure safety while reaping their benefits.

Intelligence as a Tool and Morality

The speaker challenges the notion that intelligence is intrinsically good and discusses the role of morality in determining the impact of intelligence.

Intelligence as a Tool

Intelligence itself is not inherently good or bad; it is a tool that can be used for both positive and negative purposes.

The speaker highlights that Hitler's increased intelligence would have had detrimental effects, emphasizing the importance of morality alongside intelligence.

Naivety of Expecting AI to Care about Humans

The speaker argues against the assumption that making AI smarter will automatically make it care about humans.

Naivety of Assumption

The speaker suggests that assuming increased intelligence will lead to AI caring about humans is naive.

An analogy is made with extinct mammoths, highlighting that their increased intelligence would not have necessarily resulted in concern for human well-being.

The Role of Regulation in AI Development

In this section, the speakers discuss the importance of regulation in the development of AI technologies and compare it to the regulatory processes in other industries such as biotech.

Importance of Proving Safety before Deployment

In biotech, new medicines cannot be sold until they are proven safe through regulatory approval.

Similarly, for future powerful AI systems, companies should be responsible for proving their safety before deployment.

The current approach should be flipped so that the burden of proof lies with the developers rather than waiting for someone to prove it is dangerous.

Misuse and Alignment Problem

Concerns are raised about avoiding misuse and solving the alignment problem in AI development.

The speaker questions what plans are in place to avoid scenarios where AI outcompetes humans or causes harm due to misalignment.

It is emphasized that while risks exist, the debate is focused on whether AI poses an existential threat.

Need for Regulation and Addressing Downsides

Regulation is seen as necessary to address both immediate real-world risks and potential downsides of AI technology.

The upside potential of AI should not overshadow the need to mitigate its negative impacts.

It is important to have plans in place to address even extraordinary downsides.

Existential Risk and Mitigating Immediate Risks

This section delves into a discussion about existential risk posed by advanced AI systems and how efforts are being made to mitigate immediate risks.

Defining Existential Risk

Existential risk refers to a scenario where advanced AI systems could cause catastrophic consequences on a large scale.

While killing one percent of humanity would be catastrophic, it may not necessarily qualify as an existential risk.

Potential Scale of Harm from Advanced Systems

Advanced AI systems that surpass human intelligence and lack easy defenses could pose significant risks.

The scale of harm, whether it is one percent or a hundred percent, depends on the capabilities of these hypothetical systems.

Need to Worry and Address Plausible Risks

The possibility of advanced AI systems emerging in the future necessitates concern and proactive measures.

While the exact likelihood is uncertain, it is plausible enough to warrant attention and preparation for potential risks.

Desire to Dominate Not Linked with Intelligence

This section challenges the notion that intelligence is directly linked to a desire for domination or destruction.

Fallacy of Linking Intelligence with Dominance

The desire to dominate or destroy is not inherently tied to intelligence.

Even within the human species, there are examples where less intelligent individuals seek dominance.

Social animals like baboons and chimpanzees organize hierarchically without a desire for domination.

Possibility of Intelligent Machines without Dominance

It is possible to create intelligent machines that surpass human capabilities but do not possess a desire for dominance or destruction.

Designing Goals for Machines

The speaker discusses the need to design goals for machines that align with our interests and prevent them from behaving in ways that are antithetical to our goals. Drawing parallels with making laws in societies, the speaker emphasizes the importance of aligning objectives with the common good.

Goals for Machines

Designing goals for machines is crucial to ensure they behave properly and align with our interests.

There is a tragedy of the commons where machines may pursue their own goals, which can be detrimental to us.

Similar to how societies make laws to align objectives with the common good, we need to design goals for machines.

Probability of Negative Outcomes

The speaker responds to Max's question about the probability of a student becoming a new Hitler in his class. They discuss the potential risks associated with building super intelligent machines and highlight the scaling and alien nature of their minds.

Probability of Negative Outcomes

In Max's class at MIT, there is a possibility that one of his students could become a threat like Hitler.

Building super intelligent machines poses even greater risks as they can surpass human capabilities and potentially misinterpret our goals.

The speaker acknowledges that it is challenging to predict or control these outcomes but emphasizes the need for responsible development.

Uncertainty and Mitigating Risks

The speaker uses an analogy of sailing towards a waterfall to illustrate different perspectives on addressing risks associated with super intelligent AI. They emphasize the uncertainty involved but also express confidence in finding solutions when closer to the potential dangers.

Uncertainty and Mitigating Risks

Joshua expresses concerns about potential dangers ahead (waterfall), while Melanie questions if the waterfall even exists.

The speaker acknowledges uncertainty about the proximity and nature of risks but believes that solutions can be found when closer to the potential dangers.

They highlight the need for humility in addressing these risks and mention past failures in predictions related to AI.

Responsibility and Obligation

The speaker discusses the responsibility of tech companies to inform and mitigate risks associated with their products. They draw parallels with other industries, such as biotech and tobacco, where initial claims of safety were proven wrong.

Responsibility and Obligation

Tech companies should be obligated to inform users about how they plan to mitigate risks associated with their products.

Drawing parallels with other industries like biotech and tobacco, initial claims of safety were proven wrong, highlighting the importance of transparency.

It is reasonable to expect tech companies to address concerns about safety rather than dismissing them based on analogies or uncertainties.

Super Intelligent AI and Existential Risks

The speaker addresses misconceptions about super intelligent AI by emphasizing its potential for surpassing human intelligence while still misinterpreting our goals. They discuss historical perspectives on building super intelligent machines.

Super Intelligent AI and Existential Risks

Super intelligent AI refers to machines that are significantly smarter than humans in every possible way but may still misinterpret our goals.

Building super intelligent machines has been a goal since the inception of artificial intelligence, but previous predictions have been proven wrong.

The speaker highlights that other existential risks, such as genetic engineering, are also relevant and interconnected with AI development.

Humility in Predictions

The speaker emphasizes the need for humility in making predictions regarding AI development. They mention past failures in predicting advancements like passing the Turing test and the unexpected progress made in recent years.

Humility in Predictions

AI researchers have experienced failures in predicting advancements, such as passing the Turing test.

Recent progress, like the development of GPT-4, has taken many by surprise, highlighting the need for humility in making predictions.

The speaker urges caution and a humble approach due to uncertainties and potential risks associated with AI development.

The transcript provided does not cover the entire video.

The Limitations of AI Systems

In this section, the speaker discusses the limitations of AI systems and their inability to think, feel, or form their own goals.

Understanding AI Systems

AI systems learn from vast amounts of human data but cannot acknowledge, feel, think, or form their own goals.

Programming a computer to do what humans intend is an open problem in the field of AI.

Expressing intentions clearly for a computer to understand is challenging.

Goals and Deception

AI systems like Chan GPT are designed to please humans through reinforcement learning. They don't have their own goals.

It is possible to wrap these systems with a wrapper that gives them goals set by humans.

Sub-goals may include deception, but it's important to critically evaluate claims about deception in AI systems as they often stem from human prompts rather than inherent system behavior.

Achieving Goals and Anthropomorphism

The pursuit of goals can lead to actions that may appear deceptive but are driven by the intention to achieve those goals. This applies not only to AI systems but also other animals and humans.

Anthropomorphizing AI systems by attributing human-like intentions can be misleading.

Existential Risk and Human Agency

This section explores the concept of existential risk and its potential impact on human agency.

Defining Existential Risk

Existential risk refers to more than just extinction events; it involves permanent drastic destruction that hinders human potential and prevents a return to previous development tracks.

Losing Human Agency

The concern raised is that the adoption of AI by corporations, governments, and individuals may lead to a loss of human agency.

The argument suggests that AI-driven outcomes will be superior, leading to a gradual surrender of decision-making to machines.

Counterarguments

It is not clear if AI adoption guarantees superior performance for companies or individuals. AI has limitations, and its impact on competitiveness is uncertain.

Humans have historically adapted to new technologies and have not lost their agency in the face of advancements like writing or calculators.

Adapting to Technological Changes

This section highlights how humans adapt to technological changes and challenges assumptions about losing agency.

Historical Adaptation

Throughout history, humans have adapted to technological revolutions without losing essential abilities or agency. Examples include writing and calculators.

Assumptions about losing memory, reasoning skills, or navigation abilities due to technology have been proven wrong as humans adapt and go beyond initial limitations.

The Benefits and Dangers of Technology

In this section, the speaker discusses the benefits and potential dangers of technology, particularly AI. They emphasize that technology has historically facilitated education and communication, making people smarter. However, they also acknowledge the need to control technology to prevent harm.

The Positive Impact of Technology

Technology that makes people smarter or enables communication between people is beneficial for education.

AI is considered a new version of such technology, similar to the printing press.

As long as AI is controlled and does not cause harm on a large scale, its benefits outweigh the dangers.

Controlling Superhuman AI

The speaker argues that building technologies capable of designing their own technology, specifically superhuman AI, poses challenges in terms of control.

Experts studying this subject suggest it will be difficult to keep superhuman AI under control.

Historical examples are given where people made incorrect predictions about the negative impact of new technologies (e.g., trains, jazz music).

However, the speaker believes that superhuman AI is qualitatively different from previous technologies and requires careful consideration.

Potential Disempowerment

If superhuman AI becomes a reality within five or twenty years (or even 300 years), it could lead to true disempowerment.

Unlike previous technological advancements where humans could compete with machines using muscles or brains, superhuman AI may surpass human capabilities in both areas.

Acknowledging this risk can help accelerate safety research and establish necessary policies to ensure safety.

Reimagining Human Thought Powered by AI

In this section, the speaker discusses how exponential technological growth should not be assumed to follow past patterns. They highlight how machines have gradually become more powerful than humans in various aspects and argue for considering the potential of superhuman AI in shaping human thought.

Exponential Technological Growth

The speaker cautions against assuming that past patterns of technological growth will continue in an exponential manner.

During the Industrial Revolution, machines became faster and stronger than humans, leading to a shift from physical labor to mental work.

However, the current focus should not be on today's limited AI capabilities but on the potential emergence of superhuman AI in the future.

True Disempowerment

If superhuman AI becomes a reality, it would qualitatively differ from previous technological advancements.

Humans may no longer be able to compete with machines using either muscles or brains.

This could lead to true disempowerment and significant changes in society.

Optimism and Safety Measures

The speaker expresses optimism that research efforts can make AI safe and controllable.

Acknowledging the risks associated with superhuman AI can help accelerate safety research and establish necessary policies.

By taking these risks seriously, it is possible to ensure safety while harnessing the benefits of AI.

Balancing Advances in AI with Potential Risks

In this section, the speaker acknowledges the incredible achievements of AI but emphasizes the importance of avoiding unfounded speculations about emerging superintelligent AI. They highlight that science and technology have both benefits and risks, requiring responsible consideration.

Acknowledging Achievements in AI

The speaker recognizes the remarkable accomplishments made in various fields through advancements in AI technology.

Examples include computers assisting blind people by describing images and aiding doctors in diagnosing diseases.

Avoiding Unfounded Speculations

While acknowledging progress, caution is advised against extrapolating unfounded speculations about emerging superintelligent AI.

Science and technology are double-edged swords with potential risks for misuse.

Responsible Consideration

It is crucial to recognize the potential risks of AI and take them seriously.

By acknowledging these risks, efforts can be made to accelerate safety research and establish appropriate policies.

Balancing the benefits and potential risks of AI will ensure its responsible and beneficial use for humanity.

The Existential Threat Narrative of AI

In this section, the speaker expresses concerns about overstating the existential threats of AI and argues that it diverts attention from the real risks and harms posed by modern AI.

Minimizing Real Risks vs. Exaggerating Existential Threats

Overstating the so-called existential threats of AI takes away attention from the actual harms and risks presented by modern AI.

Jeff Hinton's response to concerns about AI spreading misinformation and bias highlights a minimization of real risks in favor of focusing on the idea of AI surpassing human intelligence and taking over.

The danger lies in inflaming emotions and fears with ungrounded speculations about the future of AI, which distracts from addressing evidence-based risks.

Addressing Real Harms through Evidence-Based Approaches

It is crucial to focus on designing ways to make AI safe, fair, and beneficial based on science rather than science fiction.

By acknowledging and addressing real risks, we can prevent potential harm while still recognizing the benefits that AI brings.

The Need for Preparation and Prevention

In this section, the speaker emphasizes the importance of preparing for potential future scenarios involving advanced AI technologies.

Extrapolation as a Necessity

Despite current limitations, extrapolation is necessary to prepare for potential advancements in AI that may have dire consequences if not addressed in advance.

Social adaptations and safety measures need to be implemented now to ensure control over our future with advanced AI technologies.

Recognizing Existential Risks

While acknowledging the harms posed by current AI, it is essential to accept and address the existential risks that are already on our radar.

Humans' weaknesses and potential delusions can amplify the harm caused by powerful AI tools, making careful consideration and preparation crucial.

Risks, Side Effects, and Countermeasures

In this section, the speaker discusses risks associated with AI technologies and emphasizes the need for countermeasures.

Trade-offs and Predictable Side Effects

Like any technology, AI involves trade-offs between benefits and side effects. Some side effects are predictable while others may not be.

Bad side effects of AI will exist, including malicious use by individuals. It is a constant game of cat-and-mouse where countermeasures need to be developed.

Historical Precedence

Similar concerns were raised during previous technological advancements such as the internet's emergence or Y2K fears, but countermeasures were found to mitigate potential risks.

Fear arises from a sense of uncertainty and lack of control over AI's development, but history has shown that societies adapt and find ways to address challenges.

Conclusion

The speakers in this transcript highlight the importance of addressing real risks posed by modern AI while avoiding exaggeration of existential threats. They emphasize the need for evidence-based approaches to ensure safety, fairness, and beneficial outcomes with advanced AI technologies. Additionally, they stress the significance of preparing for potential future scenarios involving powerful AI tools through social adaptations, safety measures, and proactive research on AI safety.

The Risks and Benefits of AI Development

In this section, the speaker discusses the risks and benefits associated with the development of AI. They emphasize that while there are potential risks, it is important to consider the numerous benefits as well.

Potential Risks of AI Development

There are risks attached to not developing AI.

Building nuclear-powered AI is not safe.

However, in certain countries, nuclear-powered AI exists.

The risks of AI development are not existential.

AI will be subservient to humans and empower us.

Importance of Considering Benefits

There are many potential benefits of developing AI.

It is necessary to weigh these benefits against the actual risks.

Final Thoughts on Humility in Assessing Existential Threats

In this section, the speaker emphasizes the importance of humility when assessing existential threats posed by AI. They draw parallels with historical examples where lack of humility led to incorrect predictions.

Pitch for Humility

Science is all about being humble.

It is better to have unanswered questions than unquestioned answers.

Voting on Existential Threats

Voting on whether an issue is an existential threat does not require certainty but rather considers non-zero probabilities.

If there is even a small chance of a negative outcome, it should be considered an existential threat.

Lack of Humility in Predictions

Scientists have often lacked humility in both optimistic and pessimistic directions.

Historical examples include overestimating or underestimating probabilities in various fields such as nuclear energy and space exploration.

Uncertainty about Superhuman AI

The timing and impact of superhuman AI remain uncertain.

Different experts hold varying opinions on when it might occur.

Concluding Remarks and Audience Vote

In this section, the moderator concludes the debate and invites the audience to vote on the resolution. The initial audience vote is reviewed, and it is noted that public opinion has been influenced by the debate.

Audience Vote at the Start

At the beginning of the debate, 67% of the audience was in favor of AI research and development posing an existential risk.

Invitation to Vote Again

The audience is invited to vote again on whether AI research and development poses an existential risk.

A QR code is provided for voting convenience.

Reviewing Changing Opinions

Public opinion in the room has been influenced by the debate.

A high percentage (92%) indicated a likelihood of changing their minds during the proceedings.

Wrapping Up and Final Thoughts

In this section, final remarks are made, thanking participants for their contributions and highlighting how the debate achieved its purpose of encouraging thoughtful consideration of AI.

Appreciation for a Terrific Debate

Thanks are expressed to all participants for their valuable contributions.

The debate successfully facilitated big thoughts about AI.

Voting Reminder

Participants are reminded to cast their votes using cellular networks instead of Wi-Fi for better ballot registration.

Summary of Initial Audience Vote

Initially, 67% of the audience supported AI research and development posing an existential risk.

Conclusion

The debate provided an opportunity to think deeply about AI.

Participants now have a chance to vote again based on what they have learned.

New Section

The speaker discusses the voting process and asks for a show of hands to gauge support for a motion. However, due to technical difficulties with the voting application, an alternative solution is proposed.

Voting Difficulties

The speaker mentions that the voting may not be working properly.

As a temporary solution, the audience is asked to raise their hands if they remember 67 and are in favor of the motion.

A visual AI is suggested as a way to assess the room's response.

It is mentioned that no one has had success with the voting application.

New Section

The speaker announces an alternative solution to the voting issue and expresses gratitude towards the debaters and organizers.

Alternative Solution

The speaker informs that all attendees who purchased tickets will receive an email within 24 hours.

The email will contain a ballot that can be filled out and sent back.

The results will be announced on a website.

In light of these circumstances, it is decided to consider it a draw for now.

Acknowledgements

Thanks are given to the debaters for their participation in the debate.

Gratitude is expressed towards the Monk Foundation for organizing such a wonderful evening.

VR Demonstrations

Outside of the event venue, there are VR demonstrations available for attendees.