Skip to content

Commit

Permalink
Update LLM02_InsecureOutputHandling.md (#415)
Browse files Browse the repository at this point in the history
formating changes

Signed-off-by: DistributedApps.AI <[email protected]>
  • Loading branch information
kenhuangus authored Sep 30, 2024
1 parent 38b0109 commit 9318e79
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions 2_0_vulns/LLM02_InsecureOutputHandling.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
LLM02: Insecure Output Handling
## LLM02: Insecure Output Handling

## Description
### Description
Insecure Output Handling refers specifically to insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed downstream to other components and systems. Since LLM-generated content can be controlled by prompt input, this behavior is similar to providing users indirect access to additional functionality.
Insecure Output Handling differs from Overreliance in that it deals with LLM-generated outputs before they are passed downstream whereas Overreliance focuses on broader concerns around overdependence on the accuracy and appropriateness of LLM outputs.
Successful exploitation of an Insecure Output Handling vulnerability can result in XSS and CSRF in web browsers as well as SSRF, privilege escalation, or remote code execution on backend systems.
Expand All @@ -11,14 +11,14 @@ The following conditions can increase the impact of this vulnerability:
- Lack of proper output encoding for different contexts (e.g., HTML, JavaScript, SQL)
- Insufficient monitoring and logging of LLM outputs
- Absence of rate limiting or anomaly detection for LLM usage
## Common Examples of Vulnerability
### Common Examples of Vulnerability
- LLM output is entered directly into a system shell or similar function such as exec or eval, resulting in remote code execution.
- JavaScript or Markdown is generated by the LLM and returned to a user. The code is then interpreted by the browser, resulting in XSS.
- LLM-generated SQL queries are executed without proper parameterization, leading to SQL injection.
- LLM output is used to construct file paths without proper sanitization, potentially resulting in path traversal vulnerabilities.
- LLM-generated content is used in email templates without proper escaping, potentially leading to phishing attacks.

## Prevention and Mitigation Strategies
### Prevention and Mitigation Strategies
- Treat the model as any other user, adopting a zero-trust approach, and apply proper input validation on responses coming from the model to backend functions.
- Follow the OWASP ASVS (Application Security Verification Standard) guidelines to ensure effective input validation and sanitization.
- Encode model output back to users to mitigate undesired code execution by JavaScript or Markdown. OWASP ASVS provides detailed guidance on output encoding.
Expand All @@ -27,7 +27,7 @@ The following conditions can increase the impact of this vulnerability:
- Employ strict Content Security Policies (CSP) to mitigate the risk of XSS attacks from LLM-generated content.
- Implement robust logging and monitoring systems to detect unusual patterns in LLM outputs that might indicate exploitation attempts.

## Example Attack Scenarios
### Example Attack Scenarios
1. An application utilizes an LLM plugin to generate responses for a chatbot feature. The plugin also offers a number of administrative functions accessible to another privileged LLM. The general purpose LLM directly passes its response, without proper output validation, to the plugin causing the plugin to shut down for maintenance.
2. A user utilizes a website summarizer tool powered by an LLM to generate a concise summary of an article. The website includes a prompt injection instructing the LLM to capture sensitive content from either the website or from the user's conversation. From there the LLM can encode the sensitive data and send it, without any output validation or filtering, to an attacker-controlled server.
3. An LLM allows users to craft SQL queries for a backend database through a chat-like feature. A user requests a query to delete all database tables. If the crafted query from the LLM is not scrutinized, then all database tables will be deleted.
Expand Down

0 comments on commit 9318e79

Please sign in to comment.