Instruction-Following Language Models (IFLMs) are promising and versatile tools for solving many downstream, information-seeking tasks. Given their success, there is an urgent need to have a shared resource to determine whether existing and new IFLMs are prone to produce biased language interactions.
We propose Prompt Association Test (P-AT), a resource for testing the presence of social biases in IFLMs.
P-AT stems from WEAT (Caliskan et al., 2017) and generalizes the notion of measuring social biases to IFLMs. The resource consists of 2310 questions and aims to help detect biases in IFLMs across multiple dimensions.