HackerOne Blog

8 AI Cybersecurity Concepts Every CISO Must Know: Insights from Anthropic and AWS

HackerOne Team
Figures Standing in a Group

Creating a secure digital space while embracing AI innovation can be difficult, so we had experts weigh in. During a webinar hosted by professionals at Anthropic, AWS, and HackerOne, they broke down key AI cybersecurity terms and strategies that security leaders should understand in this evolving environment.

These must-know terms, along with tips from specialists in the webinar, are vital to staying ahead of the risks.

Key Terms Every Security Professional Must Know

1. Prompt Injection 

  • A major vulnerability in AI systems where malicious inputs can manipulate model outputs, potentially bypassing safety protocols to extract sensitive information. 

2. AI Red Teaming 

  • A proactive security assessment methodology where experts simulate potential attacks to discover weaknesses in AI systems before malicious actors do.

During the webinar, Jason Clinton, CEO at Anthropic, points to issues like geo-breaking and prompt injections as threats AI red teaming is particularly effective at spotting.

3. Model Control Protocol (MCP)

  • MCP, an open standard developed by Anthropic, defines a universal interface for connecting AI models to external services and data sources. Instead of custom one-off integrations, MCP offers a plug-and-play framework.

4. Tool Poisoning

  • This attack involves embedding malicious instructions into datasets or tools integrated with AI systems, thus corrupting system functionality.

In the talk, Vishwa Gopinath Kurakundi, Partner Solutions Architect (Gen AI) at Amazon Web Services, tied attacks like tool poisoning and tool shadowing to situations when AI applications interface with MCPs.

“With MCP, you are not just enabling the communication. But you're also defining the communication. So that introduces a few other security challenges,” Gopinath Kurakundi said. “[Tool poisoning] occurs when a potential attacker manipulates the information or description of the tools, which will then lead the AI agent to take unintended actions.”

5. AI Jailbreak 

  • Strategies used to subvert an AI system's safety features, allowing unintended or harmful actions to take place. 

Michiel Prins, Co-founder and Senior Director of Product Management at HackerOne, said during the webinar that it’s effective to leverage the skills of security researchers with and without AI security expertise in red-teaming exercises.

“I've seen some really clever jailbreaks or bypasses to protections very smart people have put in place to just get defeated by someone who thinks very creatively, was never educated in AI, and was probably their 1st time probing an AI system,” Prins said. “So we really like to mix these perspectives of fresh eyes and expert eyes.”

How Anthropic’s Jailbreak Challenge Put AI Safety Defenses to the Test

6. RAG (Retrieval-Augmented Generation) 

  • A process of generating the output of an AI model by referencing a pre-determined, authoritative knowledge base outside of it’s training sources.

Dylan Souvage, Partner Solutions Architect at Amazon Web Services, shared a case where a client wanted to externalize an app with confidential data to a customer-facing product. He notes that the accuracy of the RAG data referenced by AI tool the should be confirmed to improve outputs and avoid sharing unauthorized data.

“Is that data already sanitized? Is it cleaned? Does it have proper tenancy? Is it something that can be extended?” he said. “These are things to think about when you're empowering your apps with that data.”

7. Vulnerability Scanner for AI Models 

  • Automated tools designed to uncover exploitable weaknesses in AI systems, ensuring deployment security. 

Prins explained that the potent combination of security researchers using AI tools to scale their research is an evolution in the vulnerability scanning practice.

8. Autonomous Agents 

  • Autonomous agents are AI-driven systems capable of making decisions and executing actions independently to achieve specific objectives. These agents operate within defined parameters, utilizing advanced algorithms, real-time data, and machine learning to adapt dynamically to changing environments.

Clinton predicts a future where these autonomous agents become virtual employees: a named agent who is onboarded, that is completely autonomous, has an episodic memory, and learns and adapts to its environment.

“My advice to folks right now is, if you have individual contributors on your team who have never managed before, start working on their management skills now,” Clinton said, “because they're going to be managing fleets of agents and fleets of virtual employees in the future.”

You can see Hai, HackerOne's AI security agent, in action on our own bug bounty program now.

Want to Go Deeper? Hear from the Experts Themselves

As AI systems become more powerful and embedded across your organization, understanding the risks, and how to mitigate them, is no longer optional. This list just scratches the surface. To hear directly from the experts featured here and dive deeper into these key concepts, watch the full on-demand webinar: Deploy GenAI with Confidence: Security Best Practices from Anthropic, AWS & HackerOne.

AI