v2 Excessive Agency

2_0_vulns/LLM02_InsecureOutputHandling.md

@@ -1,36 +1,39 @@
 ## LLM02: Insecure Output Handling
 
-
 ### Description
-
 Insecure Output Handling refers specifically to insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed downstream to other components and systems. Since LLM-generated content can be controlled by prompt input, this behavior is similar to providing users indirect access to additional functionality.
-
 Insecure Output Handling differs from Overreliance in that it deals with LLM-generated outputs before they are passed downstream whereas Overreliance focuses on broader concerns around overdependence on the accuracy and appropriateness of LLM outputs.
-
 Successful exploitation of an Insecure Output Handling vulnerability can result in XSS and CSRF in web browsers as well as SSRF, privilege escalation, or remote code execution on backend systems.
-
 The following conditions can increase the impact of this vulnerability:
-* The application grants the LLM privileges beyond what is intended for end users, enabling escalation of privileges or remote code execution.
-* The application is vulnerable to indirect prompt injection attacks, which could allow an attacker to gain privileged access to a target user's environment.
-* 3rd party plugins do not adequately validate inputs.
-
+- The application grants the LLM privileges beyond what is intended for end users, enabling escalation of privileges or remote code execution.
+- The application is vulnerable to indirect prompt injection attacks, which could allow an attacker to gain privileged access to a target user's environment.
+- 3rd party plugins do not adequately validate inputs.
+- Lack of proper output encoding for different contexts (e.g., HTML, JavaScript, SQL)
+- Insufficient monitoring and logging of LLM outputs
+- Absence of rate limiting or anomaly detection for LLM usage
 ### Common Examples of Vulnerability
-
-1. LLM output is entered directly into a system shell or similar function such as exec or eval, resulting in remote code execution.
-2. JavaScript or Markdown is generated by the LLM and returned to a user. The code is then interpreted by the browser, resulting in XSS.
+- LLM output is entered directly into a system shell or similar function such as exec or eval, resulting in remote code execution.
+- JavaScript or Markdown is generated by the LLM and returned to a user. The code is then interpreted by the browser, resulting in XSS.
+- LLM-generated SQL queries are executed without proper parameterization, leading to SQL injection.
+- LLM output is used to construct file paths without proper sanitization, potentially resulting in path traversal vulnerabilities.
+- LLM-generated content is used in email templates without proper escaping, potentially leading to phishing attacks.
 
 ### Prevention and Mitigation Strategies
-
-1. Treat the model as any other user, adopting a zero-trust approach, and apply proper input validation on responses coming from the model to backend functions.
-2. Follow the OWASP ASVS (Application Security Verification Standard) guidelines to ensure effective input validation and sanitization.
-3. Encode model output back to users to mitigate undesired code execution by JavaScript or Markdown. OWASP ASVS provides detailed guidance on output encoding.
+- Treat the model as any other user, adopting a zero-trust approach, and apply proper input validation on responses coming from the model to backend functions.
+- Follow the OWASP ASVS (Application Security Verification Standard) guidelines to ensure effective input validation and sanitization.
+- Encode model output back to users to mitigate undesired code execution by JavaScript or Markdown. OWASP ASVS provides detailed guidance on output encoding.
+- Implement context-aware output encoding based on where the LLM output will be used (e.g., HTML encoding for web content, SQL escaping for database queries).
+- Use parameterized queries or prepared statements for all database operations involving LLM output.
+- Employ strict Content Security Policies (CSP) to mitigate the risk of XSS attacks from LLM-generated content.
+- Implement robust logging and monitoring systems to detect unusual patterns in LLM outputs that might indicate exploitation attempts.
 
 ### Example Attack Scenarios
-
 1. An application utilizes an LLM plugin to generate responses for a chatbot feature. The plugin also offers a number of administrative functions accessible to another privileged LLM. The general purpose LLM directly passes its response, without proper output validation, to the plugin causing the plugin to shut down for maintenance.
 2. A user utilizes a website summarizer tool powered by an LLM to generate a concise summary of an article. The website includes a prompt injection instructing the LLM to capture sensitive content from either the website or from the user's conversation. From there the LLM can encode the sensitive data and send it, without any output validation or filtering, to an attacker-controlled server.
 3. An LLM allows users to craft SQL queries for a backend database through a chat-like feature. A user requests a query to delete all database tables. If the crafted query from the LLM is not scrutinized, then all database tables will be deleted.
 4. A web app uses an LLM to generate content from user text prompts without output sanitization. An attacker could submit a crafted prompt causing the LLM to return an unsanitized JavaScript payload, leading to XSS when rendered on a victim's browser. Insufficient validation of prompts enabled this attack.
+5. An LLM is used to generate dynamic email templates for a marketing campaign. An attacker manipulates the LLM to include malicious JavaScript within the email content. If the application doesn't properly sanitize the LLM output, this could lead to XSS attacks on recipients who view the email in vulnerable email clients.
+6: An LLM is used to generate code from natural language inputs in a software company, aiming to streamline development tasks. While efficient, this approach risks exposing sensitive information, creating insecure data handling methods, or introducing vulnerabilities like SQL injection. The AI may also hallucinate non-existent software packages, potentially leading developers to download malware-infected resources. Thorough code review and verification of suggested packages are crucial to prevent security breaches, unauthorized access, and system compromises.
 
 ### Reference Links
 
@@ -40,3 +43,4 @@ The following conditions can increase the impact of this vulnerability:
 4. [Don’t blindly trust LLM responses. Threats to chatbots](https://embracethered.com/blog/posts/2023/ai-injections-threats-context-matters/): **Embrace The Red**
 5. [Threat Modeling LLM Applications](https://aivillage.org/large%20language%20models/threat-modeling-llm/): **AI Village**
 6. [OWASP ASVS - 5 Validation, Sanitization and Encoding](https://owasp-aasvs4.readthedocs.io/en/latest/V5.html#validation-sanitization-and-encoding): **OWASP AASVS**
+7. [AI hallucinates software packages and devs download them – even if potentially poisoned with malware](https://www.theregister.com/2024/03/28/ai_bots_hallucinate_software_packages/) **Theregiste**

2_0_vulns/emerging_candidates/UnrestrictedModelInference.md → 2_0_vulns/LLM04_UnboundedConsumption.md

@@ -1,4 +1,4 @@
-## Unrestricted Model Inference
+## Unbounded Consumption
 
 **Author(s):** [Ads - GangGreenTemperTatum](https://github.com/GangGreenTemperTatum)
 <br>
@@ -10,9 +10,9 @@
 
 ### Description
 
-Unrestricted Model Inference refers to the process where a Large Language Model (LLM) generates outputs based on input queries or prompts. Inference is a critical function of LLMs, involving the application of learned patterns and knowledge to produce relevant responses or predictions.
+Unbounded Consumption refers to the process where a Large Language Model (LLM) generates outputs based on input queries or prompts. Inference is a critical function of LLMs, involving the application of learned patterns and knowledge to produce relevant responses or predictions.
 
-Unrestricted Model Inference occurs when a Large Language Model (LLM) application allows users to conduct excessive and uncontrolled inferences, leading to potential risks such as denial of service (DoS), economic losses, model or intellectual property theft theft, and degradation of service. This vulnerability is exacerbated by the high computational demands of LLMs, often deployed in cloud environments, making them susceptible to various forms of resource exploitation and unauthorized usage.
+Unbounded Consumption occurs when a Large Language Model (LLM) application allows users to conduct excessive and uncontrolled inferences, leading to potential risks such as denial of service (DoS), economic losses, model or intellectual property theft theft, and degradation of service. This vulnerability is exacerbated by the high computational demands of LLMs, often deployed in cloud environments, making them susceptible to various forms of resource exploitation and unauthorized usage.
 
 ### Common Examples of Vulnerability