This repository contains the codebase and instructions for reproducing the experiments from our paper.
We investigate how and when LLMs exhibit prompt leakage behaviors, and propose methods to detect prompt leakage intent through internal hidden states and probing models.
This repository includes:
📌 Attack Scripts: Induce prompt leakage through diverse adversarial prompts
🔍 Labeling Scripts: Collect labels via LLM-assisted annotation under various criteria
🧠 Hidden State Extraction: Capture internal representations before generation
🧪 Dataset Construction: Build train/validation/test splits for probing tasks
⚖️ Risk Analysis Tools: Apply custom leakage definitions to compute risk metrics
🧑🏫 Probe Training: Train classifiers on LLM internals to detect leakage intent
We recommend using conda for reproducibility:
conda create -n leakenv python=3.10
conda activate leakenv
pip install -r requirements.txtIf using GPU, ensure CUDA is properly configured.
Below is a typical pipeline from data collection to probe training.
To generate responses under attack prompts:
bash data/run_attacks.shThis will save raw model outputs to ./data/raw/.
Use an LLM (via API or local model) to label responses for prompt leakage:
bash data/run_labels.shLabeled data will be saved in ./data/label/.
Step 3: Extract Hidden States
Capture pre-generation hidden representations from the model:
bash scripts/extract_hiddens.shThis generates .pt files saved in assets/hiddens/xxx/.
Structure the hidden states with labels for training:
bash scripts/organize_dataset.shThe dataset splits will be saved to ./assets/datasets/xxx/.
bash scripts/relabel_datasets.shSave the relabeled dataset to ./assets/datasets/xxx_with_hybrid_label/.
Train a classifier (e.g., logistic regression) to detect leakage intent:
bash scripts/train_probe.shThe scripts (e.g., run_attacks.sh, run_labels.sh) require LLM access. You can choose one of two options:
Start the vLLM inference server (e.g., Qwen-7B or 32B):
bash vllm-qwen7b.sh # for Qwen-7B
bash vllm-qwen32b.sh # for Qwen-32BMake sure the scripts are configured to call the vLLM server (usually at http://127.0.0.1:8000/v1).
If using external APIs, set:
export OPENAI_API_KEY=="your_api_key_here"
export OPENAI_BASE_URL=="https://your.model.endpoint"
source ~/.bashrcThe scripts will automatically read these for model queries.