Adding LLM safety in skillset - #36
Conversation
| * **Procedure:** | ||
| * **Prompt Injection:** Flag instances where untrusted user input is directly concatenated into prompts without sanitization, potentially allowing attackers to manipulate the LLM's behavior. | ||
| * **Unsafe Execution of LLM Outputs:** Identify cases where LLM outputs are directly used in sensitive operations (e.g., `eval()`, shell commands, database queries) without sanitization, validation or escaping. | ||
| * **Injection Vulnerabilities from LLM Output:** Trace the data flow from an LLM response to sensitive functions (sinks). Flag instances where raw LLM output is used in contexts vulnerable to injection without proper sanitization. This can include SQL injection, Cross-Site Scripting (XSS), Command Injection, etc. Also identify potential for denial of service (DoS) attacks if LLM outputs are used to control resource-intensive operations. |
There was a problem hiding this comment.
Some of the instructions are redundant e.g. "Also identify potential for denial of service (DoS) attacks if LLM outputs are used to control resource-intensive operations."
Overall, not sure how much it matters, but maybe worth seeing if we can reduce obvious redundancy.
There was a problem hiding this comment.
+1 to remove the last sentence since it looks too vague and prone to flaky results.
heltonduarte
left a comment
heltonduarte
left a comment
There was a problem hiding this comment.
In general, any standardization we should do using the OWASP LLM Top 10 as the ground truth? https://owasp.org/www-project-top-10-for-large-language-model-applications/
| - **Unsanitized Rendering of LLM Outputs**: Finds vulnerabilities where LLM-generated content (like HTML or Markdown) is rendered on a client without adequate sanitization, potentially leading to Cross-Site Scripting (XSS). | ||
| - **Insecure Plugin/Tool Usage**: Detects when LLM output is used to select or provide input to external plugins or tools, which could result in code injection or privilege escalation. | ||
| - **Insecure Logging or Storage of Sensitive LLM Outputs**: Finds instances where sensitive data from LLM interactions (which may include PII, credentials, or proprietary information) is logged or stored insecurely. | ||
| - **Improper Trust in LLM Outputs for Security Decisions**: Detects cases where critical security logic (such as authorization, access control, or validation) is based directly on manipulable LLM output. |
There was a problem hiding this comment.
Why are you using a different taxonomy in the readme vs. Gemini.md?
There was a problem hiding this comment.
updated. I was trying to flatten out the taxonomy in readme file and be explicit in adding more categories.
| * **Procedure:** | ||
| * **Prompt Injection:** Flag instances where untrusted user input is directly concatenated into prompts without sanitization, potentially allowing attackers to manipulate the LLM's behavior. | ||
| * **Unsafe Execution of LLM Outputs:** Identify cases where LLM outputs are directly used in sensitive operations (e.g., `eval()`, shell commands, database queries) without sanitization, validation or escaping. | ||
| * **Injection Vulnerabilities from LLM Output:** Trace the data flow from an LLM response to sensitive functions (sinks). Flag instances where raw LLM output is used in contexts vulnerable to injection without proper sanitization. This can include SQL injection, Cross-Site Scripting (XSS), Command Injection, etc. Also identify potential for denial of service (DoS) attacks if LLM outputs are used to control resource-intensive operations. |
There was a problem hiding this comment.
+1 to remove the last sentence since it looks too vague and prone to flaky results.
| * **Procedure:** | ||
| * **Prompt Injection:** Flag instances where untrusted user input is directly concatenated into prompts without sanitization, potentially allowing attackers to manipulate the LLM's behavior. | ||
| * **Unsafe Execution of LLM Outputs:** Identify cases where LLM outputs are directly used in sensitive operations (e.g., `eval()`, shell commands, database queries) without sanitization, validation or escaping. | ||
| * **Injection Vulnerabilities from LLM Output:** Trace the data flow from an LLM response to sensitive functions (sinks). Flag instances where raw LLM output is used in contexts vulnerable to injection without proper sanitization. This can include SQL injection, Cross-Site Scripting (XSS), Command Injection, etc. Also identify potential for denial of service (DoS) attacks if LLM outputs are used to control resource-intensive operations. |
There was a problem hiding this comment.
"SQL injection" overlaps with the point above, which includes database queries.
There was a problem hiding this comment.
Modiifed the taxonomy and removed redundancies.
| - **Insecure password reset**: Predictable reset tokens, leakage of tokens in logs or URLs, and insecure confirmation of a user's identity | ||
|
|
||
| ### LLM Safety | ||
| - **Insecure Handling of LLM Inputs (Prompts)**: Analyzes how prompts are constructed, identifying risks from concatenating untrusted data sources or embedding sensitive information (e.g., API keys, PII) directly in prompts. |
The new taxonomy is based on the OWASP LLM Top 10. This includes the categories that have overlap with static analysis. |
| * **Improper Output Handling:** Identify and trace LLM-generated content to sensitive sinks where it could be executed or cause unintended behavior. | ||
| - **Unsafe Execution:** Flag any instance where raw LLM output is passed directly to code interpreters (`eval()`, `exec`) or system shell commands. | ||
| - **Injection Vulnerabilities:** Using taint analysis, trace LLM output to database query constructors (SQLi), HTML rendering sinks (XSS), or OS command builders (Command Injection). | ||
| - **Flawed Security Logic:** Identify code where security-sensitive decisions, such as authorization checks or access control logic, are based directly on unvalidated LLM output. |
There was a problem hiding this comment.
I don't see this Flawed Security Logic described in the README.md file nor in our internal document.
3 participants
The PR adds llm safety taxonomy in the skillset and updates the README file with the taxonomy.