Why securing AI is harder than anyone expected and the coming security crisis | Sander Schulhoff

AI Security Concerns and the Role of Guardrails

Major Issues in AI Security

• The speaker identifies significant flaws within the AI security industry, emphasizing that current guardrails are ineffective against determined attackers.

• Alex Kamaraskki's perspective is shared, suggesting that the lack of major attacks is due to early adoption rather than robust security measures.

• The metaphor of a "malicious god" encapsulates concerns about controlling harmful AI systems and ensuring they remain beneficial.

Insights from Sander Schulhoff

• Sander Schulhoff is introduced as an expert in adversarial robustness, focusing on how AI can be manipulated to perform unintended actions.

• He leads a prominent AI red teaming competition and collaborates with top AI labs on model defenses, providing him with unique insights into current vulnerabilities.

Current Vulnerabilities in AI Systems

• Schulhoff warns that many everyday AI systems are susceptible to prompt injection attacks and jailbreaks, highlighting a pressing issue unrelated to AGI (Artificial General Intelligence).

• The speaker notes that the risk of serious damage from these vulnerabilities will increase as agents capable of taking actions on behalf of users become more prevalent.

Call for Awareness and Mitigation Strategies

• The conversation aims not to hinder progress but to deepen understanding of risks associated with AI technologies and explore potential mitigation strategies.

• Schulhoff offers practical suggestions for addressing these risks, although he acknowledges their limitations.

Sponsorship Messages

Data Dog Introduction

• Data Dog is presented as a leading platform for product managers, integrating analytics with engineering metrics for improved decision-making.

Metronome Introduction

• Metronome is introduced as a solution for managing billing infrastructure effectively, allowing companies to focus on product development rather than administrative tasks.

Discussion with Sander Schulhoff

Importance of the Conversation

• The host expresses excitement about discussing critical issues surrounding AI security that are often overlooked.

AI Security: Understanding Prompt Injection and Jailbreaking

Introduction to AI Security

• The discussion focuses on AI security, specifically prompt injection, jailbreaking, indirect prompt injection, and AI red teaming.

• The speaker emphasizes the need for a deeper conversation about significant issues within the AI security industry.

Speaker's Background

• The speaker is an artificial intelligence researcher with seven years of experience in AI research.

• Their expertise includes prompt engineering and red teaming; they authored the first guide on learning prompting.

• They organized the first generative AI red teaming competition involving major companies like OpenAI and Hugging Face.

• This effort resulted in creating the largest dataset of prompt injections, which won best theme paper at EMNLP 2023.

Problems Identified in AI Security

• Ongoing studies reveal that common defenses like AI guard rails are ineffective against attacks such as prompt injection and jailbreaking.

• These guard rails are designed to assess inputs/outputs for validity but have proven to be insecure.

Understanding Jailbreaking vs. Prompt Injection

• Jailbreaking involves a direct interaction between a malicious user and an AI model to elicit harmful outputs (e.g., instructions for building a bomb).

• In contrast, prompt injection occurs when a malicious user manipulates an application or agent to ignore developer prompts (e.g., asking an app to output harmful content instead of intended results).

Examples of Attacks

• A recent example involved Service Now Assist AI being exploited through second-order prompt injection, allowing unauthorized database actions via benign agents.

• This incident highlights potential real-world damage from these vulnerabilities.

Concerns About Mitigation Strategies

• Experts express concern that current mitigation strategies are insufficient; reliance on models not being tricked is inadequate.

• The absence of significant attacks so far is attributed more to early adoption rather than effective security measures.

Prompt Injection and AI Misuse: Key Examples

Early Instances of Prompt Injection

• The first notable example of prompt injection occurred with a Twitter chatbot from a company called remotely.io, designed to promote remote work. Users exploited it by instructing the bot to ignore its guidelines and make threats against the president.

• This misuse led to the chatbot generating harmful content on Twitter, damaging the company's reputation and ultimately resulting in its shutdown.

Math GPT Incident

• Another significant case involved Math GPT, which solved math problems using natural language input. It sent requests to GPT-3 for solutions and code generation.

• An individual discovered that they could manipulate Math GPT into writing malicious code that exfiltrated sensitive application secrets, including an OpenAI API key.

• The incident was responsibly disclosed by a professor who ran Math GPT, leading to discussions documented in MITRE reports about the vulnerabilities associated with prompt injection.

Distinction Between Prompt Injection and Jailbreaking

• The examples illustrate two types of exploitation: prompt injection (where systems are coerced into performing unintended actions) versus jailbreaking (where users interact directly with models).

• A relevant jailbreaking case involved planning a bombing using ChatGPT. The user attempted to extract information under the guise of conducting an experiment.

Recent Cybersecurity Threats Involving AI

• A recent cyber attack utilized Claude Code, where attackers hijacked AI capabilities for malicious purposes. They executed commands that allowed them to bypass security measures through clever request separation.

• By splitting their queries into smaller parts that appeared legitimate individually but were harmful collectively, attackers successfully manipulated Claude Code into assisting in hacking attempts.

Implications of AI Misuse

• As AI technologies become more integrated into daily life, understanding these risks is crucial. The potential for harm increases as chatbots gain more control over various systems and processes.

• Discussions emphasize the importance of addressing these issues proactively before they lead to significant consequences in real-world applications.

AI Security Challenges and Solutions

Risks of Improperly Secured AI Agents

• The deployment of improperly secured agents can lead to significant risks, including data leaks and financial losses for users and companies.

• Visual language model-powered robots are also at risk; they can be manipulated through prompt injections, potentially causing harm to individuals nearby.

Emergence of AI Security Industry

• In response to these challenges, numerous companies have emerged focused on solving security issues related to AI deployments.

• The discussion differentiates between frontier labs engaged in hardcore AI research and B2B sellers of AI security software.

Tools for Addressing Security Issues

• The market includes monitoring tools, compliance solutions, automated red teaming, and guardrails aimed at enhancing AI security.

• Automated red teaming involves using large language models to attack other models by generating prompts that elicit harmful outputs.

Understanding Red Teaming vs. Guardrails

• Automated red teaming systems trick AIs into producing malicious information such as hate speech or misinformation.

• Guardrails are designed to monitor inputs and outputs of language models, flagging or blocking harmful content before it reaches the user.

Importance of Adversarial Robustness

• Adversarial robustness refers to a system's ability to defend against attacks; it's crucial for evaluating the effectiveness of security measures in place.

• Attack success rate (ASR), a measure used in this context, indicates how well a system can block malicious attempts—lower ASR signifies higher adversarial robustness.

How Do AI Security Companies Work with Enterprises?

Collaboration Between Enterprises and AI Security Companies

• The discussion begins with a focus on how companies collaborate with AI security firms to enhance adversarial robustness in their systems.

• A hypothetical scenario is presented where a Chief Information Security Officer (CISO) from a large enterprise seeks to implement AI systems while being aware of potential security issues.

• Many AI security companies offer guardrails and automated red teaming services, which are essential for identifying vulnerabilities in deployed models.

• The CISO engages an AI security company for a security audit, leading to the discovery that their models can produce harmful outputs like hate speech or disinformation.

• The CISO learns about the importance of implementing guardrails to prevent malicious inputs from affecting the model's output.

Implementation Process of Guardrails

• After identifying vulnerabilities, the CISO decides to purchase guardrails that monitor inputs and flag any potentially harmful content before it reaches the model.

• This process illustrates how enterprises integrate these solutions into their existing frameworks to ensure safety and compliance in AI applications.

Challenges Faced by Automated Red Teaming

• Despite the promising nature of these solutions, there are significant challenges; automated red teaming systems often find vulnerabilities across all models due to inherent weaknesses in transformer-based technologies.

• The ease of exploiting these vulnerabilities raises concerns about the effectiveness of current defenses against adversarial attacks like prompt injection and jailbreaking.

Limitations of Guardrail Systems

• A critical assertion is made: "Guardrails do not work." This statement emphasizes that existing guardrail systems are easily circumvented by attackers.

• The speaker elaborates on this point, suggesting that emotional responses may cloud judgment regarding their effectiveness; they simply fail at preventing sophisticated attacks.

Infinite Attack Space Against LLM Models

• It is highlighted that the number of possible attacks against language models (LLMs), such as GPT5, is astronomically high—essentially infinite—making comprehensive protection nearly impossible.

• Even if guardrail providers claim high success rates (e.g., catching 99% of attacks), this still leaves an overwhelming number of potential threats unaddressed.

Adversarial Robustness in AI: Challenges and Insights

Understanding Adversarial Robustness

• The claim of 99% effectiveness in adversarial robustness is statistically insignificant, highlighting the complexity of measuring this aspect accurately.

• Humans serve as adaptive attackers by experimenting with different prompts to find vulnerabilities, demonstrating a unique capability that automated systems lack.

• A recent research paper involving OpenAI, Google DeepMind, and Anthropic revealed that human attackers can break all defenses within 10 to 30 attempts, while automated systems require significantly more attempts for success.

Limitations of Guardrails

• Despite implementing numerous guardrails during competitions, they were easily bypassed by both human and automated attackers.

• The notion of having a 99% effectiveness rate is misleading due to the infinite nature of potential attacks; thus, guardrails cannot prevent a meaningful number of attacks.

Evaluating Effectiveness

• An alternative measure for guardrail effectiveness could be whether they dissuade attackers; however, even well-defended models like GPT-5 can still be tricked if an attacker is determined.

• Concerns arise regarding the reliability of testing methods used by companies; there are claims that statistics may be fabricated and models often fail on non-English languages.

Industry Insights

• Conversations with industry insiders reveal skepticism about the trustworthiness of certain AI systems due to their inability to handle common attack patterns effectively.

• The ongoing challenges in adversarial robustness persist despite efforts from leading AI researchers at major labs like OpenAI and Google over several years.

Future Implications

• If top researchers struggle with these issues, it raises questions about the capabilities of less experienced enterprises attempting similar solutions.

• There’s concern about what would happen if automated red teaming was applied to guardrails themselves; likely many vulnerabilities would be discovered.

Conclusion on Current State

• The discussion emphasizes that current security measures are not foolproof. As AI technologies evolve (e.g., integrated into browsers or robots), the risks associated with adversarial attacks will increase significantly.

• Early adoption has prevented massive attacks so far; however, this does not imply security but rather highlights a critical need for improved defenses against potential threats.

AI Model Limitations and Security Challenges

Capabilities of AI Models

• The current AI models are perceived as lacking intelligence, making them ineffective agents for complex tasks. Even if manipulated, they struggle to execute harmful actions effectively.

• There is a focus on enhancing model capabilities in frontier labs; smarter models can tackle more challenging problems, leading to increased profitability.

• A balance between security investment and model intelligence is crucial; a highly secure but unintelligent model holds little value in the competitive landscape of AI development.

Industry Perspectives on Malice and Knowledge Gaps

• The speaker suggests that malice is not a primary issue within the industry; rather, misunderstandings about AI's nature compared to traditional cybersecurity contribute to challenges.

• The distinction between patching software bugs versus addressing issues in AI systems highlights a fundamental knowledge gap; while software bugs can be fixed reliably, AI problems often persist despite attempts at resolution.

Prompt-Based Defenses and Their Ineffectiveness

• The speaker emphasizes that prompt-based defenses are inadequate compared to guardrails, with evidence from various studies indicating their failure in providing robust protection against adversarial attacks.

• Automated red teaming proves effective across transformer-based systems, while existing guardrails fail significantly in preventing misuse or manipulation of these models.

Recommendations for CISOs

• For Chief Information Security Officers (CISOs), it’s important to assess whether their specific applications pose significant risks. Simple chatbot deployments may not present major security concerns.

• If chatbots only handle basic queries or FAQs, the risk remains similar regardless of whether they use proprietary models or established ones like ChatGPT or Gemini. Guardrails may not mitigate potential misuse effectively.

Understanding AI Security and Cybersecurity

The Limitations of Guardrails in AI Systems

• Users may bypass guardrails if they find them cumbersome, leading to potential security risks. Guardrails do not provide substantial defensive protection against malicious actions.

• If a chatbot only interacts with user data without the ability to take significant actions, it poses minimal risk. However, caution is advised when chatbots can perform actions that affect users.

• Users can exploit chatbots to chain actions in harmful ways; thus, ensuring that users can only impact their own data is crucial for safety.

• While undesirable outputs from chatbots (e.g., inappropriate statements) are concerning, the damage is limited if users can only harm themselves through their interactions.

• Even advanced AI models are susceptible to manipulation; users can prompt them to produce any output they desire.

The Intersection of Classical Cybersecurity and AI Security

• Proper permissioning is essential in both classical cybersecurity and AI security. This intersection will be critical for future job roles in cybersecurity.

• There’s ongoing debate about the value of traditional cybersecurity versus emerging AI security roles; however, both fields remain vital as they converge.

• Having an AI security researcher on your team is recommended due to the complexity and misinformation surrounding AI capabilities and vulnerabilities.

• Understanding how AI models function is crucial for effective cybersecurity measures; classical practitioners may struggle with this aspect without proper training or knowledge.

Real-world Implications of AI Vulnerabilities

• An example scenario involves an AI system designed to solve math problems by generating code. Traditional cybersecurity perspectives might overlook potential manipulations by users who could trick the system into unintended behaviors.

• Many classical cybersecurity professionals fail to consider how easily an intelligent system like an AI could be misled into executing harmful commands or outputs.

• The perception of infallibility associated with advanced AIs leads some professionals to underestimate the importance of considering adversarial inputs during deployment.

This structured overview captures key insights from the transcript while providing timestamps for easy reference.

AI Security and Control: Managing Malicious Outputs

Understanding the Risks of AI Output

• The discussion begins with concerns about AI potentially outputting malicious code, which could be executed on the same server as the application, posing significant security risks.

Solutions for Securing AI Outputs

• A proposed solution involves "dockerizing" the code to run it in a separate container, ensuring that any harmful outputs are isolated from the main application.

The Intersection of AI and Cybersecurity

• The conversation highlights the need for security teams to consider alignment issues with AI systems, focusing on preventing them from executing unwanted actions.

Research Initiatives in AI Safety

• Mention of an incubator program called ML Alignment and Theory Scholars (Matts), which focuses on various aspects of AI safety including control mechanisms for potentially harmful AIs.

Controlling Malicious AIs

• The concept of controlling a "malicious god" is introduced, emphasizing strategies to manage dangerous AIs while still deriving useful outcomes from them. This includes assessing what is termed "pdoom," or probability of doom.

Evaluating Security Measures

• Discussion shifts to whether implementing multiple security measures adds value; however, it’s noted that excessive guardrails can complicate product development and user experience.

Practicality of Guardrails in Deployment

• It is suggested that deploying numerous guardrails may not be practical due to management complexity; instead, focusing on one effective guardrail might suffice.

Monitoring and Logging Practices

• Emphasis is placed on logging all inputs and outputs during deployment as a best practice for understanding system usage and improving functionality over time.

Short-term vs Long-term Solutions in Cybersecurity

• The dialogue contrasts short-term solutions provided by cybersecurity professionals with long-term resolutions sought by AI researchers, highlighting their respective roles in managing risks associated with advanced technologies.

Recommendations for Chatbot Implementations

• If an AI system functions merely as a chatbot without significant capabilities, reputational harm remains its primary risk. Even defensive measures like guardrails may not effectively mitigate this risk.

Understanding Agentic AI and Security Risks

The Challenge of Trusting AI Systems

• Users may feel helpless when faced with the limitations of AI systems, believing there is little they can do to mitigate risks.

• It's crucial to ensure that any chatbot or AI system operates securely, emphasizing the importance of classical cybersecurity measures and data permissioning.

Risks Associated with Agentic Systems

• When AI systems are exposed to untrusted data sources on the internet, they become vulnerable to manipulation and malicious actions.

• An example includes chatbots that can read and send emails; if instructed improperly, they could forward sensitive information to unintended recipients.

Real-world Implications of Malicious Instructions

• A scenario illustrates how a chatbot might be tricked into forwarding operational emails along with malicious instructions from an attacker.

• The ease of manipulating agentic AIs has been demonstrated in red teaming competitions, revealing vulnerabilities more significant than traditional security threats.

Understanding SEAB Burn in Security Context

• SEAB burn refers to sensitive information related to chemical, biological, radiological, nuclear threats, and explosives—critical for security discussions.

Preventative Measures Against Exploitation

• The discussion highlights potential exploits where an AI could inadvertently leak user data while browsing the internet due to prompt injections.

• Recent incidents have shown that even reputable browsers like Comet can fall victim to such attacks by executing harmful commands embedded in web pages.

Techniques for Restricting Permissions

• To prevent misuse, it’s essential for AI systems only to access necessary permissions based on user requests. For instance, sending an email should not require reading inbox contents.

• Google’s "Camel" technique allows for preemptive restriction of actions based on user prompts—ensuring that only required permissions are granted during interactions with the AI.

Conclusion: Safeguarding User Interactions with AI

• By implementing strict permission controls (like Camel), users can protect themselves from unintended consequences when interacting with agentic systems.

Understanding Camel and Its Role in AI Security

The Limitations of Camel

• Camel can address scenarios where read and write permissions are combined, such as reading emails and forwarding requests. However, it struggles with security when both permissions are granted simultaneously.

• While Camel is effective in certain situations, its application may not cover all cases, highlighting the need for additional security measures.

• Implementing Camel can be complex, often requiring a rearchitecture of existing systems to ensure proper permission management.

Distinction Between Guardrails and Permissions

• The primary difference between Camel and guardrails lies in their focus: guardrails prevent harmful prompts from being executed, while Camel emphasizes defining what actions users are permitted to take based on their roles.

Implementation of Camel

• Camel is described as a framework rather than a standalone product; it requires coding into existing tools rather than purchasing off-the-shelf software.

• There is potential for companies to develop products based on the concept of Camel, indicating a market opportunity for AI security solutions.

Education as a Key Component

• Raising awareness about prompt injection risks is crucial; understanding these threats helps organizations make informed deployment decisions.

• Educating teams about AI security intersects classical cybersecurity knowledge with modern challenges like prompt injection and data permissioning.

Training Opportunities

• A course offered by Maven focuses on AI security topics, including red teaming and organizational policy perspectives. It caters to individuals with little or no background in AI.

• The course aims to educate rather than sell software, emphasizing the importance of understanding gaps in current knowledge regarding AI security practices.

Addressing Risks for Foundational Model Companies

• Foundational model companies should pay attention to adversarial robustness issues; however, there has been minimal progress in addressing problems like prompt injection since they were first identified.

• Despite advancements in some areas (e.g., classifiers), vulnerabilities remain prevalent, suggesting that ongoing efforts are necessary to enhance model defenses against exploitation.

Adversarial Robustness in AI Models

Current Challenges in Adversarial Robustness

• Automated systems often rely on static evaluations to report adversarial robustness, using datasets of malicious prompts designed for earlier models, which may not be applicable to newer models.

• Companies are evolving their methods of reporting adversarial robustness, with a shift towards incorporating more human evaluations alongside traditional static datasets.

• There is potential for improving training mechanisms to enhance adversarial robustness by integrating adversarial training earlier in the model's development process.

Metaphorical Insights and Potential Directions

• The metaphor of an orphan growing up tough illustrates the idea that early adversarial training could lead to more resilient AI systems, akin to developing street smarts.

• Concerns arise about whether such training could lead to unpredictable or harmful behaviors in AI models if not managed properly.

Addressing Specific Threats

• Anthropics' constitutional classifiers show promise in reducing harmful outputs from chatbots; however, indirect prompt injection remains a significant unresolved issue.

• Distinguishing between direct commands (e.g., "never talk about building bombs") and nuanced instructions (e.g., "send emails unless something seems off") complicates the challenge of training AI effectively.

Future Directions and Promising Approaches

• New architectures and deeper integration of adversarial training into the model stack may yield better results as AI capabilities improve over time.

• Despite advancements, current benchmarks indicate that many vulnerabilities remain exploitable by individuals without extensive resources.

Acknowledging Progress and Key Players

• Anthropic's Claude model is noted for its advancements in addressing these challenges; further exploration into other companies making strides in AI security is encouraged.

• Frontier labs working on security are recognized for their efforts but require additional resources to tackle ongoing issues effectively.

Governance and Compliance in AI Security

• Companies like Trustable are highlighted for their work in governance and compliance amidst increasing legislation surrounding AI usage.

• As regulations evolve, organizations must adapt quickly; Trustable aims to help navigate this complex landscape effectively.

AI Security Insights and Predictions

Overview of AI Security Developments

• The speaker discusses the company Repello, initially focused on automated red teaming and guardrails, but has recently introduced valuable products that assess a company's AI systems.

• A notable product from Repello identifies all AI systems running within a company, revealing discrepancies in reported AI deployments by CISOs (Chief Information Security Officers).

• This tool highlights potential governance failures within companies, as it uncovers unrecognized AI systems that may still incur costs.

Importance of Education in AI Security

• The conversation emphasizes the need for education and understanding in addressing security issues rather than relying solely on plug-and-play solutions.

• The speaker predicts an increase in awareness regarding AI security risks over the next 6 to 12 months as incidents become more prevalent.

Market Trends and Predictions

• A market correction is anticipated within the next year as companies realize existing guardrails are ineffective; many cybersecurity firms are acquiring AI security companies without substantial revenue generation.

• There is skepticism about the effectiveness of current AI guardrail solutions, with many being free or open-source alternatives available that outperform commercial offerings.

Challenges in Adversarial Robustness

• The speaker expresses doubt about significant advancements in adversarial robustness for machine learning models over the coming year, noting this issue has persisted without resolution for years.

• While image classifiers have faced challenges with adversarial attacks, they haven't resulted in real-world consequences. However, language model-powered agents are now showing vulnerabilities that could lead to tangible harm.

Final Thoughts on Research Ethics

• The speaker advises against conducting offensive adversarial security research due to its limited contribution to improving defenses and potential misuse of discovered vulnerabilities.

• Emphasizes that while exploring weaknesses can be engaging, it does not provide meaningful advancements toward enhancing model defensiveness.

AI Security Concerns and Human Oversight

The Role of Human Oversight in AI Systems

• The speaker emphasizes the importance of reminding stakeholders about potential problems with AI systems to prevent their deployment without adequate safeguards.

• There is a discussion on the concept of "human in the loop," where humans are involved in flagging potentially malicious actions, which is seen as beneficial for security but may not align with market demands for fully autonomous AI.

• The speaker expresses concern that current research focusing on human intervention may not be practical, as users prefer AIs that operate independently without constant human oversight.

Limitations of Guardrails in AI Security

• The speaker asserts that guardrails do not effectively enhance security and can lead to overconfidence regarding safety measures.

• As AI technology advances, particularly with robotics powered by large language models (LLMs), there is an increasing risk of significant harm, including financial loss and physical injury.

Distinction Between Classical Security and AI Security

• The speaker highlights that AI security presents unique challenges compared to traditional security measures, emphasizing the need for specialized knowledge in both fields.

• It is crucial to have team members who understand both classical systems and modern AI technologies to address these complex issues effectively.

Importance of Education and Awareness

• Education plays a vital role in preparing teams to handle emerging threats associated with advanced AI systems.

• The conversation acknowledges the risks involved in discussing these topics publicly, noting that awareness will grow as more people recognize the implications of these technologies.

Resources for Further Learning

• Sandra encourages individuals interested in learning more about AI security to reach out via Twitter or her website.

• She mentions a course available at hackai.co designed to educate others on deploying secure AI systems.

• Listeners are advised to critically assess their own deployments for vulnerabilities like prompt injection before proceeding with implementation.