VIDEO
HIGHLIGHT
Log InSign up
GPT 5 is All About Data
SummaryTranscriptChat

GPT 5 is All About Data

Understanding GPT-5

In this transcript, the speaker discusses GPT-5 and its potential release later this year. They summarize that data is a key factor in determining whether it will approach genius-level IQ. The speaker also discusses the accuracy of leaked information about GPT-5 and the number of GPUs it may be trained on.

Factors Affecting GPT-5's Release

  • Data is a key factor in determining whether GPT-5 will approach genius-level IQ.
  • The completion of GPT-4 or equivalent around late spring/early summer 2022 suggests that GPT-5 could be released later this year.

Importance of Data for Language Modeling Performance

  • Recent landmark models are wastefully big, and leveraging enough data can eliminate the need for large parameter count models.
  • High-quality data is important for training language models, and we are within one order of magnitude of exhausting high-quality data between 2023 and 2027.

Sources of High-Quality Data

  • Scientific papers, books, scraped content from the web, news, code, etc., plus Wikipedia are sources of high-quality data.

Estimate of Available High-Quality Data

  • An estimate suggests there are between 4.6 trillion to 17 trillion words worth of high-quality language data available.

Conclusion

The availability and quality of data play a crucial role in developing advanced AI language models like GPT. While there is still much to learn about how these models work and what they can do, understanding their limitations and potential applications is essential for anyone interested in AI research.

Introduction

The speaker discusses the potential reasons why GPT-4 or Bing's responses may sometimes turn out like emoting teenagers. He also suggests that there might be a reason that neither Google nor OpenAI have been forthcoming about where they get their data from.

Possible Reasons for Poor Responses

  • GPT-4 or Bing may have scraped the bottom of the web text barrel, which could explain why its responses sometimes turn out like emoting teenagers.
  • Neither Google nor OpenAI have been forthcoming about where they get their data from, which could be due to avoiding controversy over attribution and compensation.

GPT-5 Improvements

The speaker discusses how GPT-5 will likely improve upon previous models by scraping as much high-quality data as possible and making other upgrades.

Upgrades to Data Discussion

  • GPT-5 will learn lessons from previous models and scrape as much high-quality data as possible.
  • The stock of high-quality data grows by around 10% annually, even without further efficiencies in data use or extraction.

Other Upgrades

  • More ways might be found to extract high-quality data from low-quality sources.
  • Automating Chain of Thought prompting into the model can lead to small gains but would still be significant when Bing is already strong.
  • Language models can teach themselves to use tools such as calculators, calendars, and APIs.
  • Models can check if their code compiles and thereby teach themselves better coding.

Additional Data Sets

  • It may be possible to train a model several times using the same data, which could lead to significant improvements.
  • Models can generate additional data sets on problems with which they struggle, such as those with complex pans. Humans could filter their answers for correctness.

The Future of AI and Jobs

In this section, the speaker discusses how AI will revolutionize the job market and predicts that AI tutors could replace human tutors by the end of next year. He also talks about the likely divergence between changes to cognitive work and changes to physical work.

Implications for Jobs Market

  • AI will revolutionize the jobs market, although it may not lead to AGI.
  • Sam Altman tweeted that in 2023, creating a simple iPhone app would cost $30,000 while a plumbing job would cost $300. The speaker wonders what these relative prices will look like in 2028.

Changes to Cognitive Work

  • The best human raters will be beaten on some benchmarks such as reading comprehension when GPT5 is released.
  • This release would have huge implications for summarization and creative writing.
  • Logic and critical reasoning tasks such as debating topics or discerning causality in complex scenarios would also be impacted.
  • Physics and high school math problems could be solved with an order of magnitude improvement.

AI Tutors

  • AI tutors replacing human tutors could be with us by the end of next year.
  • The release of GPT5 will coincide with final refinements in text-to-speech, image-to-text, text-to-image, and text-to-video avatars.

Safety Research

  • Timelines for GPT5 depend partly on internal safety research at Google and OpenAI.
  • Safety and alignment are crucial goals before releasing AGI models.

Conclusion

In this section, the speaker ends with a quote from Sam Altman emphasizing that safety progress must keep up with capability progress as AI models become more powerful.

Safety Progress

  • The ratio of safety progress to capability progress must increase as AI models become more powerful.
  • The speaker thanks the audience for watching and encourages them to check out his other videos on Bing chat and its use cases.
Video description

Drawing upon 6 academic papers, interview snippets, possible leaks, and my own extensive research, I put together everything we might know about GPT 5: what will determine its IQ, its timeline and its impact on the job market and beyond. Starting with an insider interview on the names of GPT models, such as GPT 4 and GPT 5, then looking into the clearest hint that GPT 4 is inside Bing. Next, I briefly cover reports of a leak about GPT 5 and discuss the scale of GPUs require to train it, touching on the upgrade form A100 to H100 GPUs. Then the DeepMind paper that changed everything, focusing LLM research on data rather than parameter count. I go over a lesswrong post about that paper's 'wild implications'. And then the key paper: 'Will We Run Out of Data'. This encapsulates the key dynamic that will either propel or bottleneck GPT and other LLM improvements. Next, I examine a different take, that perhaps data is already limited and caused the Sydney model of Bing. This opens up to a discussion on the data behind these models and why Big Tech is so unforthcoming about where it originates. Could a new legal war be brewing? I then cover 4 of the ways these models may improve even without data augmentation, such as Automatic Chain of Thought, high quality data extraction, tool training, including Wolfram Alpha, retraining on existing data sets, artificial data generation and more. We take a quick look at Sam Altman's timelines and host of Big Bench benchmarks that they may impact, such as reading comprehension, critical reasoning, logic, physics and Math. I address Altman's quote about timelines being delayed by alignment and safety and finally, Altman's comments on AGI and how they pertain to GPT 5. https://www.patreon.com/AIExplained https://stratechery.com/2023/new-bing-and-an-interview-with-kevin-scott-and-sam-altman-about-the-microsoft-openai-partnership/ https://twitter.com/XiXiDu/status/1285225627390443521 https://www.linkedin.com/pulse/building-new-bing-jordi-ribas/ https://twitter.com/davidtayar5/status/1625140481016340483/photo/1 https://www.techradar.com/news/chatgpt-might-bring-about-another-gpu-shortage-sooner-than-you-might-expect https://www.nvidia.com/en-gb/data-center/h100/ https://arxiv.org/pdf/2203.15556.pdf https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications https://twitter.com/Meaningness/status/1628052277050376192 https://arxiv.org/pdf/2211.04325.pdf https://twitter.com/Meaningness/status/1628052277050376192 https://www.searchenginejournal.com/google-bard-training-data/478941/#close https://arxiv.org/pdf/2302.12822.pdf https://arxiv.org/pdf/2302.04761.pdf https://www.wolframalpha.com/ https://arxiv.org/pdf/2207.14502.pdf https://aclanthology.org/2022.findings-emnlp.508.pdf https://www.technologyreview.com/2022/11/24/1063684/we-could-run-out-of-data-to-train-ai-language-programs/ https://twitter.com/sama/status/1625980933861175306 https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/results/plot_BIG-bench_lite_aggregate.png https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/gre_reading_comprehension https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/logical_args https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/physics https://lifearchitect.ai/gpt-4/ https://www.lesswrong.com/posts/uxnjXBwr79uxLkifG/comments-on-openai-s-planning-for-agi-and-beyond https://www.patreon.com/AIExplained Non-Hype, Free Newsletter: https://signaltonoise.beehiiv.com/

GPT 5 is All About Data | YouTube Video Summary | Video Highlight