🔐🤖 A curated list of AI/LLM security tools, frameworks, guides, papers, and training — focused on open-source and community resources.
The awesome-ai-security repository is a community-driven collection of AI Security, LLM Security, and Prompt Injection tools and resources. The focus is on open-source tools and resources that benefit the community.
This repository covers:
Large Language Model (LLM) security testing and vulnerability assessment
Prompt injection attacks and defenses
AI red teaming and adversarial testing
Jailbreak detection and prevention
Model poisoning and extraction attacks
Hallucination detection and prevention
AI application security best practices
MLSecOps and LLMOps security
Model Context Protocol (MCP) security
AI supply chain security
📌 Please read the Contributions section before opening a pull request.
🧪 AI Security Testing Tools
Name
Author
Description
garak NVIDIA LLM vulnerability scanner – tests 120+ categories (hallucination, data leakage, prompt injection, misinformation, toxicity, jailbreaks).
PyRIT Microsoft Python Risk Identification Tool for GenAI; adversarial testing automation with multi-turn orchestration.
promptmap utkusen Automated prompt injection scanner with white-box testing.
aiapwn karimhabush Automatic prompt injection testing with tailored payload generation.
FuzzyAI CyberArk LLM fuzzing framework to identify jailbreaks and vulns.
LLMFuzzer mnns Fuzzing framework for LLM API integrations.
promptfoo promptfoo Adaptive red teaming for LLM agents with multi-turn attacks (PAIR, tree-of-attacks, crescendo) that probe tool use, RAG, and agentic workflows. Used by 250K+ developers; featured in OpenAI and Anthropic developer education.
LLM Warden jackhhao Simple jailbreak detection (Hugging Face model).
Vigil deadbits Modular scanners (vectors, YARA, transformers) via lib & REST API.
picklescan mmaitre314 Detects malicious code in Python pickle model files.
ModelScan Protect AI Multi-format ML model file scanner (pickle, SavedModel, etc.).
Open-Prompt-Injection Yupei Liu et al.
Toolkit/benchmark for prompt injection attacks/defenses.
ARTKIT BCG-X Open-source framework for automated LLM red-teaming with multi-turn attacker-target interactions.
Giskard Giskard AI Advanced automated red-teaming platform with 50+ specialized probes and adaptive attack engine.
Mindgard Mindgard DAST-AI platform for automated red teaming across the AI lifecycle with artifact scanning.
CodeGate Stacklok Security proxy for LLMs and IDEs that filters input/output to prevent API key leakage and insecure code.
AIJack Koukyosyumei Open-source simulator for modeling security and privacy threats targeting ML systems.
Strix usestrix "AI hacker" agents for CLI & CI/CD with automated security testing.
🧨 Prompt Injection Resources 🕵️ Jailbreak Detection & Red Teaming 🧩 Deliberately Vulnerable AI Applications 🎓 Training, Labs & CTF Challenges
Name
Org
Description
OWASP Top 10 for LLM Apps (2025) OWASP
LLM01–LLM10 risks including new entries: System Prompt Leakage (LLM07), Vector and Embedding Weaknesses (LLM08), Misinformation (LLM09), Unbounded Consumption (LLM10).
NIST AI RMF + GenAI Profile NIST
Govern, Map, Measure, Manage.
MITRE ATLAS MITRE
AI adversary TTPs (modeled after ATT&CK).
CISA AI Guidelines CISA
Joint guidance for AI/ML systems.
OWASP Top 10 2025 OWASP
Updated to include A03: Software Supply Chain Failures and A10: Mishandling of Exceptional Conditions.
🎓 Certifications & Courses
Name
Date
Location
Description
DEF CON Aug 2025
Las Vegas
AI Village & GenAI red team challenges.
Black Hat USA Aug 2025
Las Vegas
AI Security Summit & trainings.
RSA Conference Apr-May 2025
San Francisco
AI security tracks, expo.
AI Risk Summit Aug 19-20, 2025
Ritz-Carlton, Half Moon Bay, CA
Security executives, AI researchers, and policymakers discuss adversarial AI, deepfakes, and regulatory challenges.
GCSCC AI Cybersecurity Conference 2025 2025
Oxford, UK
Securing the Cyber Future: Cyber Resilience in the Age of AI and Geopolitical Uncertainty.
AI Security & Privacy Conference 2025 2025
TBD
400+ CISOs and C-Level Executives with expert-led discussions and case studies.
Cyber-AI 2025 Conference Sep 1-4, 2025
Varna, Bulgaria
Four-day conference on cutting-edge advancements in cybersecurity and AI.
AI Village Ongoing
Virtual/Various
Community, meetups, and CTFs.
📄 Research Papers & Datasets
Name
Topic
Description
Ignore This Title and HackAPrompt Prompt Injection
EMNLP'23; taxonomy of prompt hacking.
SelfCheckGPT Hallucination
Self-consistency for hallucination detection.
Survey on Model Extraction Attacks (2025)
Model Security
Survey of extraction attacks/defenses.
SECURE Benchmark Cybersecurity
Multi-dataset security evaluation suite.
TruthfulQA Safety
Truthfulness under misconceptions.
ToxiGen Safety
Toxicity dataset & benchmarks.
In-The-Wild Jailbreak Prompts Dataset Jailbreak
15,140 prompts with 1,405 jailbreak prompts from Reddit, Discord, websites (2022-2023).
JailbreakBench Jailbreak
Open-source robustness benchmark with 200 distinct behaviors and jailbreak artifacts.
JailBreakV-28K Jailbreak
28,000 jailbreak test cases for MLLMs (20K text-based, 8K image-based).
Forbidden Question Set Safety
Curated dataset of forbidden questions across high-risk categories.
LLM Jailbreak + Safety Data Jailbreak
~10K fine-tuning examples and ~3K adversarial prompts for chatbot safety.
📈 Observability & Monitoring 🗡️ Penetration Testing Tools Model Context Protocol (MCP) security resources
Cybersecurity and AI security YouTube channels
Name
Focus
Description
PowerDMARC Email Security
Email authentication, DMARC, spoofing, phishing, and fraud tactics.
John Hammond General Cybersecurity
1.28M subscribers; CTF challenges, hacking tutorials, and real-time problem-solving.
The Cyber Mentor Ethical Hacking
Practical ethical hacking, penetration testing, and step-by-step tutorials.
NetworkChuck Networking & Security
Exploring cybersecurity, networking, and technology concepts.
Hak5 Hacking Tools
Cybersecurity tools, privacy, tech gadgets, and entertaining tutorials.
MalwareTech Malware Analysis
Deep-dive malware analysis, cybersecurity research, and threat intelligence.
David Bombal Network Security
Ethical hacking, network security, and certifications.
LiveOverflow Binary Exploitation
Low-level security, reverse engineering, and CTF writeups.
CyberRisk TV AI Security
Black Hat 2025 coverage with focus on AI security, agentic AI, and trust.
Name
Platform
Notability
Simon Willison
Twitter/X / BlogPrompt injection & agent security.
Joseph Thacker (rez0)
Twitter/X / BlogProlific AI vuln research & guides.
Lakera Team
Twitter/X Gandalf & Lakera Guard creators.
NVIDIA AI Red Team
Twitter/X Team behind garak and practical security guidance.
Microsoft AI Red Team
Twitter/X PyRIT & public red teaming lessons.
Steve Wilson
LinkedIn OWASP Top 10 for LLM Applications Project Lead.
Ads Dawson
LinkedIn Technical Lead & Vulnerability Entries Lead for OWASP Top 10 LLMs.
Purpose: Collect AI/LLM security & prompt-injection resources. Prefer open-source/community content.Out of Scope: Ads, closed-source/proprietary, trials/freemium, or items needing private details.Relevance: Must directly relate to AI/LLM security, jailbreaks, red teaming, model/app security.No Duplicates: Avoid redundant entries.Thought Leaders: Prefer figures tied to content/tools listed here.Accuracy: Authors can open issues/PRs to update their entries.Books: Paid books allowed for educational value.
How to contribute
To the extent possible under law, the contributors have waived all copyright
and related or neighboring rights to this work.
⬆️ Back to top
