Fable 5 vs GPT-5.6 - It Wasn't Close

Testing GPT 5.6: A Comprehensive Benchmark Analysis

Overview of the Testing Process

• The speaker discusses the recent hype surrounding the release of GPT 5.6 and shares their experience testing it against benchmarks used in their business for coding and bug fixing.

• They highlight that Soul Ultra performs well at a third of the price compared to Fable 5, indicating a potential cost advantage.

Competitive Landscape

• With Fable 5's subscription plan ending soon, there is uncertainty about its future pricing model—whether it will remain subscription-based or shift to usage-based billing.

• Benchmarks show that Codeex Terra 5.6 outperforms Fable 5, suggesting that users should consider practical applications rather than just benchmark results when choosing models.

Importance of Real-world Application

• The speaker emphasizes the necessity of using AI models in real production environments to form genuine opinions about their effectiveness, as benchmarks can be misleading due to marketing tactics by companies.

• They outline specific tests planned for comparison between Fable 5 Max and GPT 5.6 Soul Ultra, including creating a landing page and games, aiming for direct performance comparisons.

Initial Test Setup

• Tests will involve pasting prompts verbatim into both models to ensure consistency in evaluation; they aim to assess reasoning effort levels between the two models during these tests.

• Various tasks are set up for testing, including generating SVG files and debugging code through different scenarios like video game creation and remotion video production.

Running Tests in Real-time

• The speaker describes running tests within a voice-controlled environment called Becken, where both AI agents (Soul Ultra and Fable 5 Max) are monitored simultaneously for output efficiency and quality.

• As tests progress, they note significant differences in completion times between Soul Ultra (3 minutes) and Fable 5 Max (17 minutes), raising questions about efficiency versus output quality.

Analyzing Output Quality

• Upon reviewing outputs from both models after initial tests, Soul Ultra produced an SVG file with animation while Fable 5 Max did not include any animation despite taking significantly longer to complete its task—highlighting potential inefficiencies in Fable's processing time relative to output quality.

Subsequent Tests: Lava Lamp Simulation

• Moving on to more complex tasks like creating a realistic lava lamp simulation without external libraries or assets reveals further discrepancies; Soul Ultra finishes significantly faster than Fable 5 Max again (56 minutes vs over an hour).

UI Design Task Comparison

• In another test aimed at recreating a landing page design based on provided references, both models are evaluated on speed and token usage; results indicate that while both perform adequately, Soul Ultra remains more efficient regarding token expenditure compared to Fable’s higher costs per task completed.

Final Thoughts on Model Performance

• The speaker concludes that while there may be some advantages in output quality with certain tasks from Fable 5 Max, overall efficiency favors Soul Ultra due to lower token consumption rates—making it potentially more appealing for budget-conscious users looking for effective AI solutions moving forward as subscription plans change.

Call-to-action

The speaker invites viewers' thoughts on which model they prefer based on their experiences with either system as new iterations continue rolling out amidst evolving market dynamics around AI tools like these.