Unveiling of Moshi: the first voice-enabled AI openly accessible to all.

Music and Introduction

The video starts with music playing in the background, followed by applause and a brief introduction.

Music and Introduction

• The video begins with background music and applause.

• A welcoming message is conveyed to the audience gathered in Paris for an event that combines music and science.

Introduction to QAI

Patrick, the CEO of QAI, introduces the organization's mission and its focus on artificial intelligence research.

Introduction to QAI

• QAI is a nonprofit lab dedicated to open research on artificial intelligence.

• The team at QAI has been working on developing novel Foundation models for AI.

Unveiling Real-Time Voice AI

Patrick discusses the development of real-time voice AI by the team at QAI within six months.

Unveiling Real-Time Voice AI

• The team at QAI presents the first-ever released real-time voice AI during the event.

• The project involved a small team working with significant support from partners like Scaleway.

Introduction of Mushi - Voice AI Demo

Mushi, the voice AI developed by QAI, is introduced for a live demo during the event.

Introduction of Mushi - Voice AI Demo

• Mushi is introduced as the voice AI developed by QAI for real-time interactions.

• A live demo showcasing Mushi's capabilities, including interaction and information sharing, is presented.

Interactive Features of Mushi

The demo explores Mushi's ability to express emotions and engage in interactive conversations.

Interactive Features of Mushi

• Mushi demonstrates understanding emotions and engaging in conversations with different accents and topics.

Paris, Pirates, and Role-Playing Adventures

In this section, the conversation transitions from discussing Paris to engaging in role-playing scenarios involving pirates and mystery storytelling.

Paris - The City of Love

• Paris is described as the city of love situated in the heart of Europe along the Seine River.

• The imagery of twinkling lights and glowing stars adds to the beauty of Paris.

Pirate Adventure

• A tale unfolds in cobblestone streets about pirates, their bravery, and a life filled with salty winds and freedom.

• Captain Bob shares insights on pirate life focusing on training, fighting, sailing, and adhering to a code of loyalty and respect.

Mystery Storytelling

• Engaging in whispering voice narration unveils a tale blending darkness with light, secrets, hope, and intrigue.

• Transitioning to discussing "The Matrix" movie plot involving Neo discovering he lives in a simulation world.

Role Play: Starship Enterprise Mission

This segment involves role-playing aboard the Starship Enterprise on a mission to explore a new planet.

Setting Up the Mission

• Role-play scenario where one participant assumes the role of captain while another takes on navigation officer duties for an exploration mission to Serius 22.

• Planning trajectory course discussions highlight preparation aspects before embarking on the mission.

Exploration Phase

• Initiating hyperspace jump marks the beginning of a five-month journey towards Serius 22 for potential discovery of life forms.

• Discoveries during exploration include scanning atmospheric composition revealing nitrogen, oxygen, and traces of carbon dioxide hinting at potential habitability.

The Importance of Paralinguistic Communication in Conversations

In this section, the speaker discusses the significance of paralinguistic communication in conversations and how it adds depth to interactions compared to text-based communication.

The Role of Paralinguistic Communication

• Paralinguistic cues such as tone, pauses, and interruptions enhance the liveliness and spontaneity of conversations.

• Current voice AI systems rely on a complex pipeline involving voice activity detection, transcription, language modeling, and text-to-speech synthesis.

• Limitations of current voice AI systems include latency issues (3-5 seconds delay) and the loss of non-textual information during interactions.

• The speaker aims to address these limitations by merging separate components into a single deep neural network for more efficient communication with machines.

Innovative Approach: Audio Language Models

This segment delves into the innovative approach of using audio language models to bridge the gap between speech and text-based AI interactions.

Evolution from Text Models to Audio Language Models

• Traditional text models train on large datasets to predict words based on context, while audio language models focus on predicting speech segments without relying on textual input.

• By compressing annotated speech data into pseudo words for training audio language models, a deeper understanding of speech patterns is achieved.

• An example featuring a short voice snippet demonstrates how audio language models can capture nuances like emotion and hesitations in speech.

Advancements in Interaction Realism with AI

This part highlights recent breakthroughs aimed at enhancing interaction realism with AI through multimodality and multistream capabilities.

Enhancing Interaction Realism

• Moshi integrates both audio generation and textual thought processes for more comprehensive responses.

• Combining written text with audio inputs accelerates Moshi's training process and improves response quality significantly.

Moshi: Conversational AI Framework Overview

In this section, the speaker introduces Moshi as a conversational AI framework that goes beyond being just a speech model. It is highlighted that Moshi can be adapted to various tasks and use cases.

Introduction to Moshi Framework

• Moshi was trained on the Fisher dataset, where participants engage in discussions on various topics after being randomly matched.

• A sample discussion from the dataset showcases a conversation resembling a phone call between individuals from different time periods.

• The conversation involves discussing topics like the President of the US and personal technology devices.

Training Process for Moshi

This part delves into the training process of Moshi, emphasizing the importance of data quality and diversity in shaping the model's capabilities.

Data Training for Moshi

• The training data significantly influences the model's knowledge and abilities, with text serving as a primary source due to data availability challenges.

• Training begins with a text-only language model (elom), forming the foundation for Moshi before joint pre-training on textual and audio data occurs.

• The goal is to transfer knowledge from text to audio, resulting in an audio foundation model capable of processing audio inputs but not engaging in conversations.

Fine-Tuning for Conversational Abilities

Fine-tuning becomes essential to teach Moshi how to converse effectively by leveraging synthetic dialogues due to limited real conversation datasets.

Teaching Conversation Skills

• Fine-tuning involves training Alum to generate realistic transcripts mimicking actual discussions.

• Synthetic dialogues are created using oral-style transcripts synthesized through text-to-speech engines before training Moshi on this data.

Voice Integration for Enhanced Interaction

Integrating voice features into Moshi enhances user interaction by providing consistent voice output across various emotions and styles.

Voice Development

• Collaboration with a voice artist named Alice results in diverse recordings used to train text-to-speech engines supporting multiple emotions and speaking styles.

Infrastructure and Latency Measurement

In this section, the speaker discusses the infrastructure and latency measurement of the system.

Infrastructure and Latency Measurement

• The latency from the voice of the speaker back to the headphones is measured between 200 and 240 milliseconds, ideal for lively conversations. This low latency is considered one of the best globally for online users.

• Users can conveniently use their mobile phones to connect to Moshi for seamless communication. The next step involves running Moshi on devices like a standard MacBook Pro without internet connectivity.

• By disconnecting from the internet and launching Moshi on a device, users can interact with Moshi in real-time scenarios similar to cloud-based interactions.

Moshi's Assistance Capabilities

This section delves into Moshi's capabilities in assisting users with various tasks.

Moshi's Assistance Capabilities

• Moshi introduces itself as an assistant capable of helping users with tasks such as setting reminders, scheduling appointments, and providing information on diverse topics.

• Users can engage in conversations with Moshi, seeking assistance tailored to their needs even in front of large crowds. However, it emphasizes that it is not a substitute for professional help when necessary.

Efficiency through Model Compression

This part focuses on enhancing efficiency through model compression techniques.

Efficiency through Model Compression

• To make Moshi more compact and efficient, efforts have been made to compress both model weights and conversation history using state-of-the-art techniques like quantization.

• Challenges arise when dealing with multimodal models due to discrepancies between text and audio modalities post quantization; however, balancing these differences results in models two to four times smaller than the original ones.

Audio Compression Techniques

The discussion shifts towards audio compression techniques essential for real-time operations.

Audio Compression Techniques

• For real-time operation of Moshi while fitting memory constraints, compressing raw audio signals becomes crucial. A codec named Mimi excels at compressing audio significantly compared to traditional methods like MP3.

Detailed Overview of the Transcript

In this section, the speaker discusses two strategies related to audio generation and detection: signature extraction and watermarking. These strategies are crucial in distinguishing between generated and authentic audio.

Signature Extraction vs. Watermarking

• Signature extraction involves identifying a signature corresponding to an actual audio, aiding in recognizing generated audio.

• Watermarking is another strategy where inaudible marks are added to generated audio for detection using specific detectors.

Insights on Future Plans

The speaker expresses gratitude for the audience's participation and hints at future developments regarding their experimental prototype.

Future Developments

• The experimental prototype showcased will soon be available online for users.

• Commitment to open science includes sharing technical knowledge through detailed papers, models, and code for training and modification.

Potential Impact of MHI Technology

The speaker emphasizes the potential of MHI technology in revolutionizing communication with machines, highlighting its adaptability and accessibility features.

Impact of MHI Technology

• MHI technology offers promising opportunities for communication enhancement with minimal data requirements for customization.

• Applications extend to enhancing accessibility, particularly benefiting individuals with disabilities.