Los HÉROES Olvidados de la INTELIGENCIA ARTIFICIAL

Transformers: The Unsung Heroes of AI

Introduction to the Scientists

• The video introduces eight scientists, referred to as "transformers," who are revolutionizing artificial intelligence (AI) and changing the world.

• These scientists emerged in 2017 when voice recognition technology began to gain traction, surprising the public with its capabilities.

The State of AI in 2017

• Despite advancements in voice recognition, generative AI was stagnant, and there were concerns that China was ahead in this field without sharing their progress.

• Early automatic translators produced poor results, and existing voice assistants like Alexa and Siri could only handle specific commands.

Breakthrough Innovation

• The eight researchers from various countries (Ukrainia, India, Poland, Wales, Germany) worked on a groundbreaking idea over six months.

• Their work culminated in a 15-page document containing the formula for what would become known as "the transformer," a model enabling machines to mimic human intelligence.

Impact of Their Work

• This innovation is described as more valuable than Coca-Cola's secret formula; it laid the groundwork for technologies like ChatGPT and Nvidia's market value.

• Without this foundational work, future advancements such as autonomous vehicles would not be possible.

Personal Reflection and Genealogy Exploration

• The narrator shares a personal anecdote about exploring family history through MyHeritage.com, emphasizing the importance of understanding one's roots.

Contextualizing AI Development

• In 2017, consumers were excited about new devices like Amazon Echo but faced limitations with current AI capabilities.

• A comparison is made between initial excitement over simple queries to Alexa versus more complex questions that reveal limitations in current technology.

Silent Revolution in AI

• While consumer-facing products struggled with complexity, significant advancements were happening behind the scenes in machine learning and image processing.

• Researchers were making strides toward autonomous driving by teaching machines to interpret visual data effectively.

Elon Musk's Role

The Evolution of AI: From Limitations to Breakthroughs

The State of AI in 2017

• Google was applying machine learning across various applications, but progress was slow due to limitations in data computation formulas that struggled with human language comprehension.

• Rumors emerged about China's ambitions in AI, fueled by vast amounts of data and lax privacy regulations, suggesting they were laying the groundwork for AI supremacy.

• By 2017, algorithms were impressive yet niche and scientific; there was a sense of stagnation as China appeared to be advancing rapidly in the field.

Jakob Uzkoreit’s Journey at Google

• Jakob Uzkoreit, a young German researcher at Google, aimed to enhance voice recognition for Google Assistant and improve Google Translate but faced insurmountable challenges with long text comprehension.

• To understand Jakob's story better, we look back at his father Hans Uzkoreit’s past—an expert in computational linguistics who fled East Germany after protesting against the Soviet invasion.

A Legacy of Linguistic Expertise

• Hans Uzkoreit became a leading figure in linguistic computing while his son Jakob initially sought to avoid following in his father's footsteps by focusing on artificial intelligence instead.

• Upon joining Google, Jakob realized that the most intriguing problems within AI were indeed those related to language processing.

Challenges with Existing Technology

• At this point, both Google Translate and Assistant had significant limitations; translations were subpar and interactions with assistants like Alexa or Siri felt constrained due to their deterministic software nature.

• Deterministic software could only perform tasks it was explicitly programmed for, which hindered more natural interactions.

A New Perspective on Language Understanding

• Hope emerged from two unexpected sources: another immigrant named Ilia Poloshukhin and inspiration from science fiction.

• Ilia's journey from Ukraine led him to study how users began asking direct questions on search engines as if conversing with a person. He discovered that traditional search methods relied heavily on keyword matching rather than understanding context.

The Meeting That Changed Everything

• In a pivotal moment at Google's headquarters, Ilia met Jakob. They shared frustrations over existing limitations in language processing technology.

• Their conversation sparked an idea inspired by the film "Arrival," where complex ideas are represented through single symbols. This led them to consider whether AI could comprehend raw text similarly to how humans read books.

The Formation of a Groundbreaking AI Team

The Initial Vision and Team Assembly

• Ilya and Jacob aim to develop machines that understand complex texts rather than just isolated words, leading them to collaborate on an ambitious project.

• Ashish Waswani joins the team; he is a prodigy from Oman with dual Indian heritage, having completed degrees in computer science and mathematics by age 16.

• Nikki Parmar, the only woman on the team, faces significant obstacles in her journey to AI. Despite setbacks, she self-studies AI and eventually secures a place at USC.

Overcoming Financial Challenges

• Upon arriving in Los Angeles for her studies, Nikki learns that her financial aid has been revoked, prompting panic for both her and her father.

• With support from family and friends, Nikki manages to stay afloat financially while pursuing her education at USC.

Completing the Team

• Lukazs Keiser joins as the elder statesman of the group; he has been with Google since 2013 after growing up in Communist Poland.

• Aidan Gómez enters as a young intern from Toronto who impresses Lukazs with his potential despite not holding a doctorate—an unusual requirement for joining Google Research.

Personal Struggles and Resilience

• Illion Jones rounds out the team; despite early academic success, he faces unemployment before finally accepting an offer from Google after initial hesitation.

• Each member of this innovative team has faced significant personal challenges that shaped their resilience and determination.

Skepticism as an Obstacle

• The path to success is fraught with skepticism; many doubted their ambitious project would succeed due to past failures in similar endeavors.

• Matt Kelly's harsh dismissal of Illion’s interest highlights internal resistance within Google against new ideas challenging established methods.

Resistance Within Google

• Hans Uzkoreit expresses doubt about the feasibility of global attention mechanisms proposed by Jakob's team.

Understanding the Evolution of AI: From RNNs to Transformers

Traditional Methods and Challenges in AI (2017)

• The discussion begins with an exploration of traditional methods in AI and the stagnation faced by the field around 2017, highlighting a need for innovation.

Rise of Recurrent Neural Networks (RNNs)

• In 2017, recurrent neural networks (RNNs) were considered cutting-edge, capable of relating words within sentences but struggled with longer texts due to context loss.

• The analogy is made comparing RNN comprehension to watching a movie through sporadic still frames, illustrating their limitations in grasping full narratives.

Advancements with Long Short Term Memory (LSTM)

• LSTMs were introduced as an improvement over standard RNNs, offering slightly better memory capabilities but still insufficient for complex language processing tasks.

Transitioning from Childlike Learning to Adult Comprehension

• Researchers aimed to enhance AI's reading ability from a child-like word-by-word approach to an adult-level understanding that captures context and assigns relevance based on word position.

Introduction of Transformers

• The concept of the transformer model emerged, allowing parallel processing of text blocks akin to rapid reading techniques. This model would revolutionize how machines understand language.

Mechanism and Impact of Transformers

• Transformers transform input text into two steps: capturing complete text and generating output with interconnected vector maps representing terms' relationships.

• A larger training dataset results in more nuanced vector maps, enabling deeper understanding of textual meanings across various contexts.

Practical Application: Machine Translation

• Initial testing involved automatic translation between English and German using BLEU standards to compare machine translations against human quality benchmarks.

• Results showed that even a basic model outperformed existing translation applications, while a more advanced model surpassed human translation quality in many instances.

Importance of Deadlines in Innovation

• The team decided to present their findings at the Neural Information Processing Systems conference by May 19, 2017. This deadline catalyzed their project’s development significantly.

• Emphasizing Parkinson's Law, they noted that having specific deadlines can drive productivity and focus efforts towards achieving substantial goals rather than settling for less impactful outcomes.

Seeking Expertise for Project Advancement

• To elevate their project further, they recognized the need for expert assistance capable of expanding their technology beyond just text processing into other domains like images and music.

Serendipitous Encounter at Googleplex

• A chance meeting occurred at Googleplex where Noam Shazeer was identified as the key figure needed—referred to as "Gandalf"—to help propel their vision forward effectively.

Who is Noam Shazeer?

Background and Contributions

• Noam Shazeer, at 42 years old, is a notable figure at Google, having worked there since 2000. He is recognized for creating the "Did you mean" feature in search.

• In 2015, he attempted to develop a system similar to ChatGPT over a weekend, seeking permission from CEO Eric Schmidt to use a supercomputer for AI knowledge solutions.

Team Dynamics and Innovations

• During discussions with younger team members, Shazeer expressed interest in their work that could simplify tasks for AI scientists.

• The team moved to building 1965 at Google, where they rapidly developed multiple versions of the Transformer model with increasing efficiency and capabilities.

The Importance of Small Teams

Effective Collaboration

• A key lesson highlighted is the effectiveness of small teams in driving significant change without bureaucratic obstacles.

• Each member brought unique skills: Jakob's vision, Illia's determination, Ashish's mathematical prowess—demonstrating diverse contributions lead to innovation.

Breakthrough Discoveries

Model Optimization

• As experiments progressed, Aidan discovered that removing components from the model improved its performance unexpectedly.

• Just before submitting their paper on May 9th, Nikki Parmar received astonishing results showing their model outperformed human translations significantly.

Submission and Reception of "Attention Is All You Need"

Title Inspiration and Impact

• The paper titled "Attention Is All You Need" was suggested by Illion Jones as a playful nod to a Beatles song reflecting love as essential—paralleling the need for improved machine attention in AI.

• Despite being present in Silicon Valley during this pivotal moment, the speaker reflects on how they might have unknowingly crossed paths with this groundbreaking project.

Initial Reactions and Future Implications

Conference Presentation

• After submitting their paper on December 6th, they presented it modestly at a conference but quickly attracted significant attention from attendees.

• Despite initial excitement around their project launch, internally at Google there was little recognition or support following the presentation.

Overlooked Innovation

• The transformative potential of "Attention Is All You Need" went largely unnoticed within Google; leadership showed disinterest due to existing profitable systems.

The Evolution of Transformers and AI Innovations

The Initial Stages of Transformer Technology

• Sam Altman noted that "nobody at Google understood the potential of the transformer," highlighting a disconnect in recognizing transformative technology.

• In 2018, Google began applying transformers to Google Translate, marking a significant step forward in their AI capabilities.

• Noam Shazeer developed Meena, a conversational chatbot, but faced rejection from Google regarding its business model.

The Exodus from Google

• Many creators of transformers left Google due to the company's inability to embrace disruptive innovations as explained by Clayton Christensen's "Innovator's Dilemma."

• This exodus led to the rise of startups like OpenAI that capitalized on the potential of transformer technology without being hindered by past successes.

Impactful Startups and Innovations

• Noam Shazeer founded Character AI, which allows users to interact with historical figures and has gained millions of users; Google later acquired this team for $2.7 billion.

• Ashis Waswani and Nikki Parmar established ADEPT AI, later acquired by Amazon for $1 billion, focusing on automating workplace tasks through observation.

Diverse Applications of Transformer Technology

• Jakob Uzkoreit created Inceptive, utilizing transformers in biotech for designing molecules for vaccines and medications.

• Ilion Jones founded Sakana AI, inspired by nature to create models that draft scientific papers.

Legacy and Future Directions

• Aida Gómez launched Cohere, valued at $5.5 billion, developing AI models tailored for businesses.

• Lukazs Kayser joined OpenAI after leaving Google due to its superior transformer technology; he contributed significantly to ChatGPT versions 3.5 and 4.

Transformative Outcomes from Transformer Research

• The foundational work on transformers has led to revolutionary applications such as AlphaFold in biology and Waymo’s autonomous vehicles.

Discovering Your Family Legacy

The Importance of Family History

• The speaker emphasizes the significance of understanding one's family history, suggesting that it can reveal innovative and revolutionary figures within one's genealogy.

• They pose a reflective question about the potential visionaries in the listener's family tree, encouraging personal exploration of heritage.

• The mention of "transformers" refers to individuals who bring new ideas and change, highlighting their role in shaping legacies.

• MyHeritage.com is introduced as a tool for discovering personal legacies, indicating its collaborative efforts in this exploration.