Shortcut Learning in Financial Text Mining: Exposing the Overly Optimistic Performance Estimates of Text Classification Models under Distribution Shift
2022 IEEE International Conference on Big Data (IEEE BigData 2022)
December 17-20, 2022 (Osaka, Japan)
In recent years, many cases of deep neural networks failing dramatically when faced with adversarial or real-world examples have been reported. Such failures, which are quite hard to detect, are often related to a generalization problem known as shortcut learning. Yet, with state-of-the-art transformer models now being ubiquitous in financial text mining, one cannot help but wonder how reliable the results conveyed in the ever-growing literature genuinely are. Against this background, we expose, in this work, how vulnerable contemporary financial text mining approaches are to shortcut learning. Focussing on the common learning task of financial sentiment classification, we assess, using two entity-based sampling strategies and our publicly-available dataset, the discrepancies between i.i.d. and o.o.d. performance estimates of four transformer models. Our results reveal that o.o.d. performance estimates are consistently weaker than those of their i.i.d. counterparts, with the error rate increasing by as much as 29.7%, thus, demonstrating how this issue can, when overlooked, lead to misleading evaluations. Moreover, we show how additional preprocessing steps, such as entity removal and vocabulary filtering, can help reduce the effects of shortcut learning by filtering out entity-related linguistic cues.
Coming soon...