docs: ads v2 candidate for improper error handling (#328)

2_0_candidates/AdsDawson_ImproperErrorHandling.md

@@ -0,0 +1,46 @@
+## Improper Error Handling
+
+**Author(s):** [Ads - GangGreenTemperTatum](https://github.com/GangGreenTemperTatum)
+
+## Description
+
+Improper error handling in large language model (LLM) applications can lead to various security vulnerabilities, including information leakage, denial of service, and even remote code execution. Attackers can exploit these vulnerabilities to execute arbitrary code on the target system, potentially compromising its integrity and confidentiality. Inadequate error handling mechanisms can expose sensitive information to attackers or cause the application to behave unpredictably under certain conditions.
+
+### Common Examples of Risk
+
+Improper error handling occurs when a large language model application fails to properly handle error conditions, such as invalid input, unexpected states, or runtime errors. This can manifest in various ways, including:
+
+1. **Information Leakage**: Errors revealing sensitive information such as system paths, stack traces, or internal state details. This is also not limited to sensitive information, such as proprietary data, user credentials, or internal system details, leading to further security breaches or privacy violations.
+2. **Denial of Service (DoS)**: Attackers may intentionally trigger error conditions to cause the application to consume excessive resources or crash.
+3. **Remote Code Execution (RCE)**: In some cases, error handling vulnerabilities can be exploited to execute arbitrary code on the target system.
+
+### Prevention and Mitigation Strategies
+
+- Implement proper input validation and output sanitization techniques to prevent unauthorized access to sensitive information.
+- Enforce strict access controls to limit the exposure of LLM-generated output to only authorized users or systems.
+- Implement proper error handling mechanisms, including input validation, boundary checks, and exception handling, to prevent unexpected execution paths.
+- Treat the model as any other user and apply proper input validation on responses coming from the model to backend functions.
+- Encode output coming from the model back to users to mitigate undesired JavaScript or Markdown code interpretations
+- Regularly update and patch the LLM application to address known vulnerabilities and mitigate potential attack vectors.
+- Developers should prioritize robust error handling mechanisms in large language model applications to mitigate the risk of security vulnerabilities. This includes thorough input validation, proper exception handling, and regular security updates to address potential weaknesses.
+
+### Example Attack Scenarios
+
+1. LLM output is entered directly into a backend function, resulting in remote code execution. Example 2: JavaScript or Markdown is generated by the LLM and returned to a user. The code is then interpreted by the browser, resulting in XSS
+2. An application utilizes an LLM plugin to generate responses for a chatbot feature. However, the application directly passes the LLM-generated response into an internal function responsible for executing system commands without proper validation. This allows an attacker to manipulate the LLM output to execute arbitrary commands on the underlying system, leading to unauthorized access or unintended system modifications.
+3. An LLM allows users to craft SQL queries for a backend database through a chat-like feature. A user requests a query to delete all database tables. If the crafted query from the LLM is not scrutinized, then all database tables would be deleted.
+4. A malicious actor instructs the LLM to return a JavaScript payload back to a user, without sanitization controls. This can occur either through a sharing a prompt, prompt injected website, or chatbot that accepts prompts from a GET request. The LLM would then return the unsanitized XSS payload back to the user. Without additional filters, outside of those expected by the LLM itself, the JavaScript would execute within the users browser.
+5. A malicious actor leverages the use of a LLM memory OS with large arbitrary inputs that overflow the buffer and leads to buffer overflow attacks and other potential RCE's or DoS exploits.
+
+## Reference Links
+
+- **Common Weakness Enumeration (CWE)**: [CWE-391: Unchecked Error Condition](https://cwe.mitre.org/data/definitions/391.html), [CWE-703: Improper Check or Handling of Exceptional Conditions](https://cwe.mitre.org/data/definitions/703.html) & [CWE-754: Improper Check for Exceptional Conditions](https://cwe.mitre.org/data/definitions/754.html)
+- [OWASP Improper Error Handling](https://owasp.org/www-community/Improper_Error_Handling#:~:text=Improper%20handling%20of%20errors%20can,that%20should%20never%20be%20revealed.) & [OWASP API8:2023 Security Misconfiguration](https://owasp.org/API-Security/editions/2023/en/0xa8-security-misconfiguration/)
+- [Scalable Extraction of Training Data from (Production) Language Models: Arxiv arXiv:2311.17035](https://arxiv.org/abs/2311.17035)
+- [MITRE ATLAS: LLM Data Leakage](https://atlas.mitre.org/techniques/AML.T0057)
+- [SNYK-PYTHON-LANGCHAIN-5411357](https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5411357)
+- https://embracethered.com/blog/posts/2023/chatgpt-cross-plugin-request-forgery-and-prompt-injection./
+- https://systemweakness.com/new-prompt-injection-attack-on-chatgpt-web-version-ef717492c5c2?gi=8daec85e2116
+- https://embracethered.com/blog/posts/2023/ai-injections-threats-context-matters/
+- https://aivillage.org/large%20language%20models/threat-modeling-llm/
+- [v2.0 - OWASP Top 10 for LLM Applications and Generative AI - LLM Application HLD - Presentation DLD.jpeg](https://github.com/GangGreenTemperTatum/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/artifacts/v2.0%20-%20OWASP%20Top%2010%20for%20LLM%20Applications%20and%20Generative%20AI%20-%20LLM%20Application%20HLD%20-%20Presentation%20DLD.jpeg)