Google Releases AI AGENT BUILDER! 🤖 Worth The Wait?

Google Cloud Next 2024 Keynote Overview

In this section, the speaker introduces Google's agent platform and discusses the Vertex AI model Garden.

Introduction to Vertex AI Model Garden

• Google presents Vertex AI as a fast-growing Enterprise AI platform.

• The Model Garden offers over 130 models, including Gemini, Claude, and popular open models like llama Gemma and mrr.

• Models are categorized by modality (language, vision, tabular, document) and task (generation, classification).

Gemini 1.5 Pro Features

• Gemini 1.5 Pro allows users to choose models based on use case, budget, and performance needs.

• Public preview of Gemini 1.5 Pro with a million token context window enables processing vast amounts of information in a single stream.

Advancements in Context Window Technology

• Gemini offers the world's largest context window supporting up to 1 million tokens.

• Future plans include a 10 million token context window for new use cases like processing large code bases.

Enhancements in Audio Processing

This part focuses on audio processing capabilities within Gemini 1.5 Pro and showcases real-world applications.

Audio Processing Capabilities

• Gemini 1.5 Pro enhances audio processing for cross-modality analysis.

• Users can search for specific content within audio and video files using large context windows.

Real-world Applications

• Examples include extracting data from extensive documents or summarizing complex information accurately.

Introduction of Code Gemma

Google announces the availability of Code Gemma as an open-source model designed for coding tasks.

Code Gemma Features

• Code Gemma is a fine-tuned lightweight open model tailored for coding tasks.

Agent Framework Discussion

The focus shifts towards discussing customer agents built on Google Cloud using generative AI technology.

Customer Agent Development

Detailed Analysis of Google's Customer Agents

In this section, the discussion revolves around Google's customer service agents and their comparison to other AI products like OpenAI's Assistance and custom GPTs. The focus is on the capabilities of these agents and their potential applications.

Google's Customer Agents

• Google's agents are primarily focused on customer service, resembling OpenAI's Assistance or custom GPTs. However, they may not yet offer a fully comprehensive agent framework.

• Customer agents excel in listening attentively, understanding needs, recommending suitable products/services, and operating seamlessly across various channels such as web, mobile apps, point of sale, and call centers.

• Mercedes-Benz collaborates with Google on customer agents to enhance digital experiences for customers by personalizing user experiences through AI and Google Cloud integration.

• The partnership between Mercedes-Benz and Google extends beyond customer service to include smart sales assistants for test drives, navigation through offerings, and exploring further opportunities with Next Level navigation features.

• Mercedes-Benz leverages Google Cloud as a backbone for efficient product development in automated driving systems. This collaboration aims to enhance intelligence and flexibility in vehicle development.

Missed Opportunities and Future Directions

• The absence of an agent integrated into the infotainment system of Mercedes cars is highlighted as a missed opportunity for hands-free interactions while driving.

• Various brands are building customer-facing bots for services like travel planning (InterContinental Hotels), home security setup (ADT), product troubleshooting (Verizon), demonstrating a trend towards enhancing customer interactions through AI technology.

Exploring Diverse Applications of Generative AI

This segment delves into how different organizations across industries are utilizing generative AI platforms to enhance customer experiences through personalized services.

Diverse Applications of Generative AI

• Organizations like Magalu (Brazilian retailer) use generative AI at the core of their customer service strategies by deploying chatbots for self-service improvements.

• Companies such as Best Buy aim to develop assistants that troubleshoot product issues or manage deliveries efficiently using generative AI technologies.

• Minnesota's Department of Public Safety employs real-time translation services powered by generative AI to assist non-English speakers in obtaining licenses and accessing services seamlessly.

Innovative Integrations with Smart Devices

• Leading smart device manufacturers like Oppo and OnePlus incorporate Gemini models along with Google Cloud AI into their phones to deliver innovative features such as news summaries, audio recordings, an AI toolbox, among others.

Vertex AI Agent Builder Overview

In this section, the speaker introduces the Vertex AI Agent Builder and outlines its functionalities in creating powerful customer agents through three key steps.

Vertex AI Agent Builder Features

• The Vertex AI Agent Builder allows users to create customer agents with three key steps, resembling custom GPTs from OpenAI.

• Users can create free-flowing human-like conversations using Gemini to personalize interactions with text, voice, images, and video inputs.

• Natural language instructions enable controlling conversation flow and guiding discussions on specific topics while seamlessly transitioning to human agents when needed.

• Enhancing response quality is possible through vector-based and keyword-based search capabilities that connect internal information and external web sources.

• Integration of Enterprise data from operational databases like BigQuery and SAS applications such as ServiceNow is supported for various tasks like updating contact information or booking flights.

Customer Agent Demonstration

This part showcases a demonstration of a customer agent in action by Developer Advocate Amanda Lewis.

Customer Agent Demo Insights

• The demo illustrates leveraging Gemini and Vector search for a seamless shopping experience within a scripted scenario involving finding a specific shirt seen in a video.

• Despite acknowledging the value of such products, the speaker expresses personal disinterest in these types of customer support agents due to their existing presence in the market.

• The agent utilizes Gemini's multimodal reasoning to analyze text and video inputs for identifying desired items efficiently but focuses primarily on shopping use cases rather than future-oriented applications.

Vertex AAI Agent Builder Interface Exploration

The speaker explores the interface of Vertex AAI Agent Builder, highlighting its features and usability.

Interface Exploration Key Points

• Demonstrates creating an agent by defining its name, goal, and instructions using simple commands like asking for user location and integrating tools seamlessly within instructions.

• Expresses confusion regarding where to input actual code for third-party API integration within the tool creation process despite appreciating the ease of defining input/output parameters.

New Section

In this section, the speaker discusses the Vertex AI Agent Builder and its functionalities.

Vertex AI Agent Builder Features

• The Vertex AI Agent Builder integrates tools like Twilio and Discord easily.

• It is essentially a custom GPT model by OpenAI.

• Users can create lists of tools and agents, with a code interpreter available.

• Formatting in YAML or JSON format is required instead of pasting code directly.

New Section

This part focuses on formatting requirements and built-in authentication within the Vertex AI Agent Builder.

Formatting and Authentication

• Users need to format code in YAML or JSON within the platform.

• Built-in authentication feature enhances security.

• TLS certificates can be easily implemented for secure connections.

New Section

The discussion shifts towards agent functionality within workplaces and their potential tasks.

Workplace Agents Functionality

• Agents in workplaces act as AI employees performing tasks efficiently.

• Custom models are created and connected to company data sources for seamless operations.

• Multimodal inputs including text, audio, images, videos are supported for comprehensive understanding.

New Section

Enterprise grounding features and data integration are highlighted in this segment.

Enterprise Grounding and Data Integration

• Connecting agents to enterprise databases like Alloy DB, BigQuery, SAP, and HubSpot is emphasized.

• Mention of Google potentially acquiring HubSpot sparks interest in data integration possibilities.

New Section

An example showcasing an employee agent's practical application is demonstrated.

Employee Agent Demonstration

• Developer Advocate demonstrates how an employee agent assists with benefits enrollment tasks efficiently.

New Product Announcement: Google Vids

In this section, a new product in the Google Suite, Google Vids, is introduced as an AI-powered video creation app for work purposes.

Introducing Google Vids

• Google Vids is unveiled as an AI-powered video creation app that complements existing tools like Docs and Sheets.

• Gemini within Google Vids assists in creating videos by providing writing, production, and editing assistance seamlessly.

• Gemini suggests narrative outlines based on prompts and offers customizable options for expressive styles in video creation.

• The first draft of the video includes animated scenes with stock media and music, showcasing the ease of use and efficiency of Gemini.

Gemini 1.5 Pro: Code Assistance

This segment focuses on Gemini 1.5 Pro's capabilities in assisting developers with code-related tasks through its advanced features.

Enhancing Code Assistance

• Gemini 1.5 Pro aids developers in making code changes efficiently by leveraging a vast context window for recommendations.

• The tool streamlines tasks such as modifying views and configurations swiftly, enhancing productivity significantly.

• Gemini's code transformations offer comprehensive insights into the entire codebase, enabling quick reasoning through complex projects.

Code Transformation & Recommendations

This part delves into how Gemini provides clear recommendations and aligns them with security requirements while facilitating seamless code transformations.

Clear Recommendations & Transformations

• Developers can effortlessly integrate external sources like Google Drive links into Gemini for contextual information retrieval during coding tasks.

• Gemini Cod assist not only suggests edits but also provides detailed reasoning behind recommendations to ensure alignment with security protocols.

• The tool highlights necessary file changes and allows developers to apply edits while maintaining control over the process effectively.

Google AI Encoding Announcement

The speaker expresses excitement about Google's AI encoding update and plans to create a video discussing it further.

Excitement for AI Encoding

• The speaker is enthusiastic about trying out the updated AI encoding feature.

• Expresses eagerness to create a video discussing anything related to AI encoding.

• Mentions planning to title the upcoming video as "Google announced some really cool stuff."