Data Gathering Methodology

OWASP Top 10 for LLM GenAI Apps - Data Gathering Methodology

Overview

Welcome to our dedicated GitHub wiki for understanding and advancing the data-gathering methodology pertaining to OWASP's Top 10 for LLM AI Applications. As technology continues to evolve at an unprecedented pace, particularly in the domains of artificial intelligence and deep learning, securing these systems is of paramount importance. This wiki serves as a central repository for methodologies, strategies, and tools associated with understanding and prioritizing vulnerabilities in LLMs based on real-world data.

The ** Data Gathering Methodology, Mapping, Risk and Exploit ** initiative is designed to collect real-world data on vulnerabilities and risks associated with Large Language Models (LLMs). This effort supports the update of the OWASP Top 10 for LLMs list while maintaining mappings between the main cybersecurity frameworks. Through a comprehensive data collection approach, the initiative aims to improve AI security guidelines and offer insights to help organizations bolster their LLM-based systems.

##Why this Wiki? Centralized Knowledge Base: With the multifaceted nature of LLM vulnerabilities, having a one-stop solution where developers, researchers, and security experts can find and contribute to the most recent and relevant methodologies is invaluable. Collaborative Environment: GitHub offers an interactive platform where community members can collaborate, providing insights, updates, and refinements to the existing methodology. Transparency & Open Source Spirit: In line with the ethos of OWASP and the open-source community, this wiki promotes transparency in the data-gathering process, ensuring everyone has access to the best practices in vulnerability assessment. Addressing the Dynamic Nature of Threats: The field of AI security is nascent but growing rapidly. This wiki will act as a live document, continuously evolving to capture the latest threats and vulnerabilities.

##How to contribute? Join our Slack channel #team-llm-datagathering-methodology Feel free to reach out to Emmanuel – [email protected]

Methodology

1. Data Collection
   • Sources:
     • Industry reports, academic papers, vulnerability databases, and real-world exploit analysis.
     • Partner organizations contributing to vulnerability disclosures and risk assessments.
   • Approach:
     • Manual review and automated data scraping.
     • Use of standardized templates to ensure data consistency.
     • Prioritization based on impact, exploitability, and prevalence.

We are always on the look out for collaboration with organizations and individuals eager to share datasets and relevant work related to the protection of Large Language Models.

2. Data Analysis

   • Initial Review:
     • Classification of vulnerabilities based on type, origin, and potential impact.
   • Statistical Analysis:
     • Use of Python scripts to validate and analyze gathered data for accuracy and completeness.
   • Risk Scoring:
     • Application of scoring frameworks (e.g., CVSS) to rank vulnerabilities based on severity.

3. Data Validation

   • Python Code:
     • Automated validation scripts will be developed to ensure the integrity and accuracy of collected data.
   • Peer Review:
     • Involvement of cybersecurity experts for manual verification and risk assessment.

Datasets

The initiative will host a series of datasets, including:

• Vulnerability Dataset:
  • Real-world vulnerabilities affecting LLM applications.
• Exploit Dataset:
  • Documented exploits and attack techniques targeting LLMs.
• Risk Assessment Dataset:
  • Mapped risk assessments for different LLM deployments.

Mapping to Cybersecurity Frameworks

The collected data was mapped to existing security frameworks, including: 1. NIST Cybersecurity Framework CFS2.0

• Provides comprehensive guidelines for managing cybersecurity risk.
• A foundational framework for cybersecurity recognized worldwide. 2. ISO/IEC Standards
• ISO/IEC 27001 (Information Security Management)
• ISO/IEC 20547-4:2020 (Big Data Reference Architecture Security and Privacy)
• Crucial for global business compliance and establishing security controls. 3. MITRE ATT&CK
• A detailed knowledge base for understanding and defending against cyber attacks.
• Practical for threat modelling and security analysis. 4. CIS Controls
• Developed by the Centre for Internet Security, offering actionable controls.
• Well-regarded for practicality in strengthening cybersecurity defences. 5. CVEs and CWEs
• Common Vulnerabilities and Exposures (CVEs)
• Common Weakness Enumeration (CWEs)
• Essential for identifying and cataloging vulnerabilities. 6. FAIR
• Factor Analysis of Information Risk focuses on risk quantification and management.
• Helps organizations quantify cybersecurity risk in financial terms. 7. STRIDE
• A threat modelling methodology for identifying security threats.
• Often used in the early stages of software development. 8. ENISA
• The European Union Agency for Network and Information Security provides broad cybersecurity advice.
• Relevant especially for compliance and best practices in European contexts. 9. ASVS
• The Application Security Verification Standard, important for web application security.
• Provides a basis for testing and assessing web application security controls. 10. SAMM
• Software Assurance Maturity Model, useful for integrating security into software development.
• Helps in benchmarking and improving software security practices. 11. MITRE ATLAS
• Focused on adversarial behaviours and may not cover all aspects of cybersecurity management.
• Specific and detailed for threat modelling and analysis. 12. BSIMM
• Building Security In Maturity Model, a tool for measuring and improving software security initiatives.
• Best suited for software security practices within organizations. 13. OPENCRE
• A facilitator for understanding and implementing cybersecurity controls across different standards.
• Acts as a bridge between various frameworks rather than a standalone guide. 14. CycloneDX Machine Learning Software Bill of Materials (SBOM)
• Standard that provides advanced supply chain capabilities for cyber risk reduction.
• Standard capable of representing software, hardware, services, and other types of inventory.

This mapping ensures compatibility and compliance, facilitating broader adoption across organizations.

LLM Data Security Best Practices

This initiative culminated in a white paper detailing the best practices for securing data in LLM-based systems, focusing on:

• Data Architecture
• Risk mitigation strategies.
• Secure LLM deployment architectures.
• Governance models for AI security.

Due to the rapid and evolving nature of this technology always keep an eye on the new TTPs, frameworks, regulations and tools.

Additional Resources

• Data Validation Python Code Repository:
  • GitHub
• White Paper:
  • "LLM Data Security Best Practices."

---

For further details or contributions, please reach out to the OWASP Top 10 for LLM GenAI Apps team.