Microsofts New 'ORCA' Takes Everyone By SURPRISE! (Now ANNOUNCED!)

Orca: Progressive Learning from Complex Explanations with Traces of GPT-4

In this video, the speaker discusses Microsoft Research's recent paper on Orca, a new language model that uses progressive learning to enhance the capability of smaller models through imitation learning. The paper shows that Orca surpasses other large language models like Vicuna and reaches parity with Chat GPT on the BBH Benchmark.

Introduction

• Microsoft Research released a new research paper called Orca.

• The abstract explains that Orca uses progressive learning to enhance the capability of smaller models through imitation learning.

• The problem is that smaller language models tend to overestimate their capabilities when they learn to imitate larger language models' style but not their reasoning process.

Comparison with Other Language Models

• Orca surpasses Vicuna's 13 billion parameter large language model in effectiveness.

• Chat GPT is another well-known large language model that Orca reaches parity with on the BBH Benchmark.

• Orca excels slightly more than Chat GPT and Google's BART in professional and academic exams like SAT and LSAT.

• Zero-shot settings show that Orca can handle new things without seeing them before, indicating its effectiveness.

Parameter Count and Integration into Devices

• Despite having only 13 billion parameters compared to Chat GPT's 175 billion parameters, Orca is just as effective.

• A smaller yet equally effective model like Orca makes it more lightweight and integratable into certain devices.

• Google announced making Palm 2 available in four sizes from the smallest to the largest, with Gecko being so lightweight that it can work on mobile devices even when offline.

Conclusion

Orca is a game-changing language model that uses progressive learning to enhance smaller models' capabilities through imitation learning. It surpasses other large language models like Vicuna and reaches parity with Chat GPT on the BBH Benchmark. Its effectiveness in professional and academic exams like SAT and LSAT shows its potential for real-world applications. A smaller yet equally effective model like Orca makes it more lightweight and integratable into certain devices, making it an exciting development in the field of natural language processing.

Orca: A New Large Language Model

In this section, the speaker discusses the potential of Orca, a new large language model with 2.7 trillion parameters. The speaker also talks about the limitations of current large language models and how they overcomplicate simple concepts.

Potential of Orca

• Orca is a new large language model with 2.7 trillion parameters.

• It has the potential to be integrated into robots, making them more efficient by not needing cloud-based servers to process and generate responses.

• If people manage to get Orca offline, it could present interesting opportunities for offline use.

System Messages

• The research paper on Orca shows that there are 16 different system messages used to train this large language model.

• These system messages can be personally used to understand complex concepts better.

• Using these system messages as a framework can help in getting difficult concepts down.

Limitations of Current Large Language Models

• Current large language models such as GPT4, GPT3, Chat GPT, Vikuna and Chachi Beauty cannot understand common sense questions.

• They tend to overcomplicate simple things and lack common sense reasoning abilities.

Let's Think Step by Step Before Each Answer

In this section, the speaker talks about how stating "let's think step by step before each answer" increases the level of results and outperforms zero short large language models on diverse reasoning tasks including arithmetic and other logical reasoning tasks.

Importance of Thinking Step by Step

• Stating "let's think step by step before each answer" actually increases the level of results in diverse reasoning tasks including arithmetic and other logical reasoning tasks.

• This technique was originally used on Chat GPT and OpenAI's DaVinci 003 to make the system better.

Limitations of Large Language Models

In this section, the speaker talks about how large language models such as Orca, GPT4, GPT3 and Chat GPT lack common sense reasoning abilities and overcomplicate simple concepts.

Common Sense Questions

• Large language models such as Orca, GPT4, GPT3 and Chat GPT cannot understand common sense questions.

• They tend to overcomplicate simple things and lack common sense reasoning abilities.

Example Questions

• The speaker gives examples of basic questions that these large language models cannot answer correctly.

• These limitations show that while large language models are incredibly intelligent in some areas, they can also be shockingly stupid in others.