OpenAI o3 and o3-mini—12 Days of OpenAI: Day 12

Launch of OpenAI's Next Frontier Model

Introduction to the Event

• The event marks the conclusion of a 12-day series, celebrating the launch of their first reasoning model, O1.

• The focus shifts to announcing the next frontier model, humorously named O3 instead of O2 due to naming challenges at OpenAI.

Announcement of New Models

• Two models are announced: O3 and O3 Mini. Both are described as highly capable but will not be publicly launched today.

• Public safety testing for these models begins today, allowing researchers to apply for access to assist in testing.

Performance Insights on Model Capabilities

• Mark discusses O3's capabilities, highlighting its strong performance on technical benchmarks.

• On software style benchmarks (Sweet Bench Verified), O3 achieves 71.7% accuracy—over 20% better than O1.

Competitive Programming and Mathematics Performance

• In competitive programming contexts like Codeforces, O3 reaches an ELO score significantly higher than previous models.

• For mathematics benchmarks such as AMY, O3 scores 96.7%, compared to 83.3% from O1.

Advanced Benchmarking and Future Directions

• The model also excels in PhD-level science questions with an accuracy of 87.7%, outperforming prior versions by about 10%.

• There is a noted need for more challenging benchmarks as many existing ones approach saturation; Epic AI’s Frontier Math benchmark is highlighted as particularly tough.

Introduction of ARC Benchmark

Collaboration with ARC Foundation

• Greg Camad from the ARC Foundation discusses their mission towards AGI through benchmarking.

Understanding Arc AGI and Its Recent Achievements

Introduction to Arc AGI

• Arc AGI focuses on transforming input examples into output examples, aiming to understand the rules of transformation.

• The challenge lies in AI's difficulty in intuitively grasping these transformations, which are often easy for humans.

Unique Challenges in AI Learning

• Each task within Arc AGI requires distinct skills; no two tasks will be identical, promoting the model's ability to learn new skills dynamically.

• This approach prevents the model from merely repeating memorized responses, emphasizing genuine learning.

Performance Milestones

• Version 1 of Arc AGI took five years to achieve a score of 5%, but version 03 has now reached a state-of-the-art score of 75.7% on a semi-private holdout set.

• When given higher compute resources, version 03 scored an impressive 87.5%, surpassing human performance benchmarks (85%).

Implications for Future AI Development

• These results indicate a need to reassess perceptions about AI capabilities and highlight that we are still in the early stages of AI development.

• More enduring benchmarks like Arc AGI are essential for measuring progress and guiding future advancements.

Introduction of O3 Mini Model

Overview of O3 Mini

• O3 Mini is introduced as an efficient reasoning model capable of math and coding tasks at low costs.

• Although not yet available to all users, access is being granted to safety and security researchers for testing purposes.

Adaptive Thinking Time Feature

• The new API includes adaptive thinking time options (low, median, high), allowing users to adjust reasoning effort based on problem complexity.

Performance Evaluation Insights

• Initial evaluations show that with increased thinking time, O3 Mini outperforms its predecessor (O1 Mini), demonstrating significant improvements in coding performance.

Implementation of a Python Code Generator and Executor

Overview of the Project

• The speaker discusses testing three different models (low, medium, high) for generating and executing code using Python.

• The task involves launching a local server with a UI that allows users to input coding requests, which are sent to the model's API for code generation.

• Once the code is generated, it is saved locally and executed automatically in a terminal.

Initial Testing

• A simple coding request is tested by generating a random number; the output is stored as a local script.

• The speaker suggests testing if the model can generate its own GP QA numbers, indicating prior practice with this functionality.

Model Evaluation Process

• The model evaluates itself on hard GPQ datasets by downloading raw files and identifying questions and answers from them.

• Results show that the low reasoning effort model performs evaluations quickly, even on challenging datasets.

Performance Metrics

• The evaluation results indicate a performance accuracy of 61.6%, showcasing efficiency in self-evaluation tasks.

• The process involved asking the model to write scripts for its own evaluation based on user inputs through its UI.

Future Improvements and Features

Enhancements in Model Capabilities

• Discussion about improving math capabilities within the models shows comparable performance across different versions (all3 mini vs. all1 mini).

• Latency measurements reveal that all3 mini low significantly reduces response times compared to previous models.

Developer Support Features

• New features such as function calling and structured outputs are being integrated into all mini series models based on developer feedback.

• Performance metrics suggest that these enhancements provide cost-effective solutions while maintaining or improving performance levels.

Safety Testing Initiatives

External Safety Research Opportunities

• Announcement of external safety testing opportunities for researchers interested in exploring potential vulnerabilities or jailbreak scenarios with O3 mini.

Advancements in Safety Techniques

Deliberative Alignment: Enhancing Safety in AI Models

Understanding Deliberative Alignment

• Deliberative alignment is a technique that allows AI models to assess the safety of prompts by reasoning over them, determining whether they are safe or unsafe.

• The reasoning capabilities of the model can reveal hidden user intents, such as attempts to trick the system, even when prompts are obfuscated.

Performance Metrics and Results

• A performance benchmark illustrates the model's ability to reject unsafe prompts (x-axis) versus its capacity to approve safe ones (y-axis), with better performance indicated towards the right.

• The results show that deliberative alignment enables improved safety outcomes, achieving higher accuracy in both rejecting and approving prompts compared to previous models.

Future Developments and Community Involvement

• Plans for launching 03 mini at the end of January and full 03 shortly thereafter aim to enhance safety testing through community participation.