Introduction

In this section, David Shapiro introduces the video and provides a quick recap of GPT-4 before jumping into GPT-5.

Recap of GPT-4

• Microsoft Research claims that GPT-4 represents the first sparks of AGI and performs strikingly close to human-level performance on many tasks.

• The base model of chat GPT-4 has an 8,000 token window, which is a game-changer. It already has an officially announced 32,000 token window, which is eight times larger than GPT-3.

• Chat GPT-4 may have around 1 trillion parameters and is slower than its predecessor, indicating more processing and deeper layers.

• Chat GPT-4 is multimodal and supports images. It passed the bar exam in the 90th percentile and outperforms most humans on some benchmarks.

• MIT released a study showing that even just chat GPT 3.5 increased white-collar productivity by 40 percent.

The Great Pause

In this section, David Shapiro discusses the open letter signed by many people calling for a six-month moratorium on building anything more powerful than GPT-4.

The Great Pause

• An open letter was signed by many people calling for a six-month moratorium on building anything more powerful than GPT-4 due to safety, ethics, regulations concerns.

• There has been a call for a public version of AI research similar to CERN with funding at one percent of what goes into CERN could produce public versions or fully open-source versions.

• No major regulatory movements have been made yet, even in Europe, which is usually more proactive in legislation.

GPT-5

In this section, David Shapiro talks about rumors surrounding GPT-5.

Rumors Surrounding GPT-5

• According to rumors that circulated on Reddit a while ago, GPT-5 is already being trained on 25.

Introduction to GPT4 and GPT5

This section discusses how GPT4 came about from incremental improvements, and how we can expect the same for GPT5. It also covers rumors and predictions about when GPT5 will be released.

Incremental Improvements for GPT4 and Expectations for GPT5

• GPD4 did not come about from any paradigm shifts or new architecture but rather hundreds of small incremental improvements that had a multiplicative effect across the whole thing.

• Expectations are that GPT5 will also come about from ongoing improvements in data pre-processing, training patterns, and better algorithms.

• Rumors suggest that we can expect a shorter testing and release cycle for GPT5 compared to previous models.

• One Twitter user claims that training for GPT5 is scheduled to finish in December 2023, which aligns with a mid-2024 release date after the six to nine-month testing cycle.

Release Date Predictions

• There are varying predictions on when GPT5 will be released. One website expects a September 2022 release for GPT4. Another blog predicts an end-of-2024 or early 2025 release based on historical patterns.

• The consensus seems to be that we can expect more traction by the end of this year, even if it's just an incremental update. However, most predictions point towards a mid-to-late 2024 release date for GPT5.

Token Window Size Increase in GPT Models

This section discusses the increase in token window size from previous models to current ones and speculates on what we can expect for GPT5.

Increase in Token Window Size

• The jump from GPT3 to GPT4 saw an increase in token window size from 4000 to 8000 tokens, with a teased 32,000 token window for future models.

• If the pattern of doubling or increasing by multiples continues, we can expect anywhere from 64,000 to 256,000 tokens for GPT5.

• To put this into perspective, most novels are between 50,000 to 70,000 words. With a token window size of up to 170,000 words (or even up to 42,000 words), GPT5 could read and write an entire novel in one go.

GPT-4 and the Future of AI

This section discusses the potential compute requirements for GPT-4 and beyond, as well as the need for new attention mechanisms and architecture. It also explores the trade-off between utility value and functional value.

Compute Requirements

• Predictions suggest that GPT-4 may require 10 to 40 times more compute than GPT-3.

• There may be diminishing returns on an algorithmic level due to dilution, which could require new attention mechanisms or architecture.

• The number of tokens needed is also a consideration, with evidence suggesting that 8,000 tokens can satisfy 90% of current needs.

Utility Value vs Functional Value

• There may be a trade-off between speed, cost, and intelligence when it comes to increasing token count.

• A model with 256,000 tokens may not be necessary for most users.

Multimodal Capabilities in GPT-4

This section discusses the multimodal capabilities demonstrated by GPT-4 and its potential implications for vectors.

Multimodal Capabilities

• GPT-4 has demonstrated multimodal capabilities with images.

Implications for Vectors

• Multimodal vectors could have more abstract and human-like thoughts inside the model, potentially unlocking new capabilities within GPT-5.

Data Sources for GPT-5

This section explores the question of where OpenAI is getting the data for GPT-5.

Data Sources

• The source of data for GPT-5 is unknown.

GPT-4 and Beyond

In this section, the speaker discusses the potential capabilities of GPT-4 and beyond.

Multimodal Models

• GPT-4 is expected to be multimodal, which means it will be able to process multiple types of data simultaneously.

• Nvidia publishes hundreds of different models that have different specializations. OpenAI may follow a similar approach with GPT models.

Intelligence and Capabilities

• The relative performance of GPT-3 versus GPT-4 was a huge jump in intelligence.

• With the correct integrations, GPT-5 could outperform humans at 99% of all other tasks, including STEM jobs.

• Given enough time and the right integrations, it's possible to ask GPT-5 to design a spaceship and build it too.

Artistic Endeavors

• It's predicted that GPT-5 will surpass humans in most artistic endeavors such as writing symphonies, composing stories, and even acting on stage.

Regulation and Competition

• There are concerns about regulation for AGI or Singularity. However, competition is driving development forward.

Conclusion

In this section, the speaker concludes by saying that we should buckle up because things are about to get silly.

Final Thoughts

• The speaker predicts AGI within 18 months if GPT 5 qualifies.