What's New in Kestra 2.0 - Here's What Changed and What You Get

Introduction to Castra 2.0

Overview of the Webinar

• The webinar introduces Castra 2.0, highlighting new features and architectural changes aimed at improving user experience.

• Kai Vena leads the discussion, with Will and Adam joining later for a live demo and community updates.

• The agenda includes topics like unified orchestration, challenges in the market, improvements in Castra 2.0, migration from version 1.x, AI integration, and community updates.

Motivation Behind Unified Orchestration

• Many organizations face increasing coordination challenges despite decades of automation; this is exacerbated by the rise of AI technologies.

• Legacy tools are becoming less flexible and more expensive due to consolidation in enterprise scheduling solutions.

• Adoption trends show a focus on cost reduction through next-generation schedulers like Castra.

Categories of Software Deployment

Different Layers of Orchestration

• Organizations need to manage various categories such as data platforms, infrastructure scheduling, applications (e.g., microservices), and business processes.

• Business processes often require human interaction for approvals or escalations within automated workflows.

Unified Control Plane Concept

• Castra provides a single control plane across all domains regardless of category definitions or additional layers like cybersecurity.

• Customers typically start using Castra in one domain but can expand its use across multiple categories effectively.

Importance of AI Integration

Future-Proofing with AI

• As organizations increasingly adopt AI technologies, having a unified control plane becomes essential for managing diverse workflows efficiently.

Current Users of Castra

Industry Adoption Examples

• Notable companies using Castra include JP Morgan (cybersecurity), Apple (machine learning), and various public sector organizations facing compliance requirements.

• Each organization utilizes Castra for different use cases tailored to their specific industry needs.

Unique Features of Castra

Architectural Advantages

• Unlike traditional orchestrators, Castra is built to be declarative, open-source under Apache 2.0 license, event-driven, real-time scalable with lightweight plugins.

Plugin Flexibility

• Users can easily integrate various tools without heavy connectors; if no plugin exists for a tool they need, creating one is straightforward.

Transitioning to Version 2.0

Major Changes in Architecture

• Version 2.0 introduces significant architectural changes that enhance scalability and flexibility while simplifying operations.

Repository & Queue Separation

• In version 2.0 users can independently choose their repository (e.g., MySQL or Elasticsearch), separating it from the queuing system (supporting Kafka or RabbitMQ).

Worker Database Decoupling

• Workers no longer access the database directly; this decouples environments enhancing security and operational efficiency.

New Features in Version 2.0

Case Management Feature

• The introduction of "cases" allows users to incorporate human interactions into orchestration flows natively within Castra for incident management purposes.

Enhanced User Experience

• Numerous new features have been added across implementation flows, UI experience enhancements, governance capabilities based on customer feedback.

Live Demo Overview

Demonstration by Will

• Will presents an overview of the redesigned UI in version 2.0 showcasing how workflows are built more intuitively than before.

Workflow Example

• A demo workflow receives data via webhook utilizing an AI agent to assess data quality before proceeding with further actions requiring human approval if necessary.

Migration Tools Available

Support for Transitioning Users

• Various migration tools are available including command line prompts that assist users in adapting existing flows to work seamlessly with version 2.0's architecture changes.

Integrating Agentic AI with Unified Orchestration

Building Workflows and Integration

• The integration of agentic AI into workflows is similar to incorporating other applications or APIs, emphasizing the ease of building these connections.

• The challenge lies in ensuring that agents can interact across various domains, including data infrastructure and business processes, while maintaining compliance and security.

• Kestra offers a unified control plane that facilitates the use of agentic AI across different domains, providing significant value through features like guardrails and access control.

Capabilities of Kestra

• Kestra's capabilities for integrating agentic AI include three main MCP (Multi-Channel Processing) functionalities designed to enhance orchestration workflows.

• Users can build flows using large language models within Kestra’s MCP interface, allowing for seamless integration with existing tools and systems.

Interaction with External Tools

• The MCP server enables users to access external tools at runtime, facilitating easy data integration from various systems into orchestration workflows.

• Kestra allows inspection of flows, executions, and logs in natural language through its MCP server, enhancing user interaction with the system.

Triggering Flows Externally

• Users can trigger Kestra orchestration flows externally via MCP, enabling agents to leverage deployed YAML files for business processes.

Community Engagement and Development

• Adam introduces himself as a new senior developer advocate at Kestra and highlights the active community surrounding the platform.

Engaging with the Community

Community Initiatives

• Adam encourages joining the Slack channel for community engagement and mentions an upcoming Paris meetup on September 24th for networking opportunities.

• Monthly meetups will be held in Paris and London to foster ongoing community interactions.

Virtual Meetups and Ambassador Program

• A virtual community meetup is scheduled for October 1st to discuss initiatives and updates regarding Kestra's development.

• An ambassador program is introduced as a way for community members to deepen their involvement while gaining visibility within the ecosystem.

Overview of Kestra 2.0 Features

Migration Experience

• Kai shares his positive experience migrating his instance to version 2.0 without issues or outages during production.

New Features Discussion

• Questions arise about demos related to creating ingestion pipelines; more content on utilizing AI within workflows is promised soon.

Open Source vs. Enterprise Versions

Version Changes

• There are no significant changes between open-source Apache 2.0 license versions compared to enterprise editions; both can migrate easily without additional requirements.

Improvements in Regex Integration

• Enhancements have been made regarding regex expressions in version 2.0, simplifying conditions setup for task triggers compared to previous versions.

Overall Summary:

The discussion emphasizes how Kestra integrates agentic AI into workflows through a unified control plane while fostering an engaged community around its development efforts. Key features of version 2.0 focus on improved migration experiences, enhanced functionality within the platform itself, and ongoing support through community initiatives.