Knowledge Regularized Negative Feature Tuning of Vision-Language Models

  

Wenjie Zhu^1,2 | Yabin Zhang³ | Xin Jin² | Wenjun Zeng² | Lei Zhang¹

¹The Hong Kong Polytechnic University, ²Eastern Institute of Technology, Ningbo, ³Stanford University.

⭐ If KRNFT is helpful to your images or projects, please help star this repo. Thanks! 🤗

🔥 News

• 25 July, 2025: Our paper has been accepted by ACM MM 2025. You can easily reproduce our results with Bash.

⚙️ Dependencies and Installation

Follow OpenOOD to set up the environment, or use our provided instructions below.

pip install git+https://github.com/YBZH/OpenOOD-VLM

📷 Datasets

We also follow OpenOOD to manage the training and testing datasets. If you only use our evaluator, the benchmarks for evaluation will be automatically downloaded by the evaluator (again check out this tutorial). If you would like to also use OpenOOD-VLM for training, you can get all data with our downloading script. Note that ImageNet-1K training images should be downloaded from its official website.

Besides datasets used in OpenOOD, we also provide evaluation on some popular OOD datasets iNaturalist, SUN, Places, and Texture curated by Huang et al. 2021. Please follow instruction from the this repository to download the subsampled datasets where semantically overlapped classes with ImageNet-1k are removed.

Our codebase accesses the datasets from ./data/ and pretrained models from ./results/checkpoints/ by default.

├── ...
├── data
│   ├── benchmark_imglist
│   ├── images_classic
│   └── images_largescale
├── openood
├── results
│   ├── checkpoints
│   └── ...
├── scripts
├── main.py
├── ...

Supported Datasets for Out-of-Distribution Detection

• BIMCV (A COVID X-Ray Dataset)

  Near-OOD: CT-SCAN, X-Ray-Bone;
  Far-OOD: MNIST, CIFAR-10, Texture, Tiny-ImageNet;
  

• MNIST

  Near-OOD: NotMNIST, FashionMNIST;
  Far-OOD: Texture, CIFAR-10, TinyImageNet, Places365;
  

• CIFAR-10

  Near-OOD: CIFAR-100, TinyImageNet;
  Far-OOD: MNIST, SVHN, Texture, Places365;
  

• CIFAR-100

  Near-OOD: CIFAR-10, TinyImageNet;
  Far-OOD: MNIST, SVHN, Texture, Places365;
  

• ImageNet-200

  Near-OOD: SSB-hard, NINCO;
  Far-OOD: iNaturalist, Texture, OpenImage-O;
  Covariate-Shifted ID: ImageNet-C, ImageNet-R, ImageNet-v2;

• ImageNet-1K

  Near-OOD: SSB-hard, NINCO;
  Far-OOD: iNaturalist, Texture, OpenImage-O;
  Covariate-Shifted ID: ImageNet-C, ImageNet-R, ImageNet-v2;

• ImageNet-1K Traditional Four Datasets

  Far-OOD: iNaturalist, SUN, Places, Texture;
  Covariate-Shifted ID: ImageNet-C, ImageNet-R, ImageNet-v2;

🚀 Training/Testing

We provide training and evaluation scripts for all the methods we support in scripts folder.