Multimodal Recommender Systems

A curated list of awesome multimodal recommendation resources (code first: messy code is better than no code). Please contact us if your work is not in the list.
- Last updated: 2023/08/22 (MM'23 Partially Updated)

• ❓ 作者未回应提供源代码请求。如大家有相关源代码或可以复现，请联系我。
• ❌ 不可复现
• ❤️ 希望大家一起把 多模态推荐 这个领域做扎实！

@article{zhou2023comprehensive,
      title={A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions}, 
      author={Hongyu Zhou and Xin Zhou and Zhiwei Zeng and Lingzi Zhang and Zhiqi Shen},
      year={2023},
      journal={arXiv preprint arXiv:2302.04473},
}

A Unified Multimodal Recommendation Framework, $\text{MMRec}$, which currently integrates:

• 10+ multimodal models
• Various general CF models

Survey

• A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions - Hongyu Zhou, Xin Zhou, Zhiwei Zeng, Lingzi Zhang, Zhiqi Shen. arxiv, Feb 2023
• Recommender Systems Leveraging Multimedia Content - Yashar Deldjoo, Markus Schedl, Paolo Cremonesi, Gabriella Pasi. ACM Computing Surveys, Sep 2020

Table of Contents

• General
• Modality-specific Recommendation
  • Text-based Recommendation
  • Image-based Recommendation
• Public Datasets

General

• IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT -Junchen Fu, Xuri Ge, Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, Jie Wang, Joemon M Jose. SIGIR, April 2024 | [pdf] [code]

• Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima - Shanshan Zhong, Zhongzhan Huang, Daifeng Li, Wushao Wen, Jinghui Qin, Liang Lin. WWW, Feb 2024 | [pdf] | [code]

• LGMRec: Local and Global Graph Learning for Multimodal Recommendation - Zhiqiang Guo, Jianjun Li, Guohui Li, Chaoyang Wang, Si Shi, Bin Ruan. AAAI, Dec 2023 | [pdf] | [code]

• A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation - Xin Zhou. MM, Oct 2023 | [pdf] | [code]

• Multi-View Graph Convolutional Network for Multimedia Recommendation - Penghang Yu, Zhiyi Tan, Guanming Lu, Bing-Kun Bao. MM, Oct 2023 | [pdf] | [code]

• Pareto Invariant Representation Learning for Multimedia Recommendation - Shanshan Huang, Haoxuan Li, Qingsong Li, Chunyuan Zheng, Li Liu. MM, Oct 2023 | [pdf]

• Enhancing Dyadic Relations with Homogeneous Graphs for Multimodal Recommendation - Hongyu Zhou, Xin Zhou, Lingzi Zhang, Zhiqi Shen. ECAI, Oct 2023 | [pdf] | [code]

• Prompt-based and weak-modality enhanced multimodal recommendation - Xue Dong, Xuemeng Song, Minghui Tian, Linmei Hu. Information Fusion, Sep 2023 | [code]

• Bootstrap Latent Representations for Multi-modal Recommendation - Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, Feijun Jiang. WWW, May 2023 | [pdf] | [code]

• Multi-Modal Self-Supervised Learning for Recommendation - Wei Wei, Chao Huang, Lianghao Xia, Chuxu Zhang. WWW, May 2023 | [pdf] | [code]

• Self-supervised Learning for Multimedia Recommendation - Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, Tat-Seng Chua. TMM, Jun 2022 | [code]

• Latent Structure Mining with Contrastive Modality Fusion for Multimedia Recommendation - Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Mengqi Zhang, Shu Wu, Liang Wang. arxiv, Mar 2022 (Extended version of LATTICE which has been published in MM21) | [pdf] | [code]

• DualGNN: Dual Graph Neural Network for Multimedia Recommendation - Qifan Wang, Yinwei Wei, Jianhua Yin, Jianlong Wu, Xuemeng Song, Liqiang Nie. IEEE TMM, Dec 2021 | [code]

• Mining Latent Structures for Multimedia Recommendation - Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, Liang Wang. MM, Oct 2021 | [pdf] | [code]

• Pre-training Graph Transformer with Multimodal Side Information for Recommendation - Yong Liu, Susen Yang, Chenyi Lei, Guoxin Wang, Haihong Tang, Juyong Zhang, Aixin Sun and Chunyan Miao. MM, Oct 2021 | [pdf] | [code]

• Why Do We Click: Visual Impression-aware News Recommendation - Jiahao Xun, Shengyu Zhang, Zhou Zhao, Jieming Zhu, Qi Zhang, Jingjie Li, Xiuqiang He, Xiaofei He, Tat-Seng Chua, Fei Wu. MM, Oct 2021 | [pdf] | [code]

• Multi-Modal Variational Graph Auto-Encoder for Recommendation Systems - Jing Yi and Zhenzhong Chen. IEEE TMM, Sep 2021 | [code] ❌

• Recommendation by Users' Multimodal Preferences for Smart City Applications - Cai Xu , Ziyu Guan , Member, IEEE, Wei Zhao , Quanzhou Wu , Meng Yan, Long Chen and Qiguang Miao. IEEE TII, Jul 2021 | [code]

• Adversarial Training Towards Robust Multimedia Recommender System - Jinhui Tang, Xiaoyu Du, Xiangnan He, Fajie Yuan, Qi Tian and Tat-Seng Chua. IEEE TKDE, May 2020 | [pdf] | [code]

• MGAT: Multimodal Graph Attention Network for Recommendation - Zhulin Tao, Yinwei Wei, Xiang Wang, Xiangnan He, Xianglin Huang, Tat-Seng Chua. Information Processing & Management, Apr 2020 | [code]

• Graph-Refined Convolutional Network for Multimedia Recommendation with Implicit Feedback - Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Tat-Seng Chua. MM, Oct 2020 [pdf] | [code]

• User Diverse Preference Modeling by Multimodal Attentive Metric Learning - Fan Liu, Zhiyong Cheng, Changchang Sun, Yinglong Wang, Liqiang Nie, Mohan Kankanhalli. MM, Oct 2019 | [pdf] | [code]

• MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video - Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, Tat-Seng Chua. MM, Oct 2019 | [pdf] | [code]

• Breaking Isolation: Multimodal Graph Fusion for Multimedia Recommendation by Edge-wise Modulation - Feiyu Chen, Junjie Wang, Yinwei Wei, Hai-Tao Zheng, Jie Shao. MM, Oct 2022

• Learning Hybrid Behavior Patterns for Multimedia Recommendation - Zongshen Mu, Yueting Zhuang, Jie Tan, Jun Xiao, Siliang Tang. MM, Oct 2022 ❓

• Multi-modal Graph Contrastive Learning for Micro-video Recommendation - Zixuan Yi, Xi Wang, Iadh Ounis, Craig Macdonald. SIGIR, Jul 2022 ❓

• MM-Rec: Visiolinguistic Model Empowered Multimodal News Recommendation - Chuhan Wu, Fangzhao Wu, Tao Qi, Chao Zhang, Yongfeng Huang, Tong Xu. SIGIR, Jul 2022

• Disentangled Multimodal Representation Learning for Recommendation - Fan Liu, Zhiyong Cheng, Huilin Chen, Anan Liu, Liqiang Nie, Mohan Kankanhalli. arxiv, Mar 2022 | [pdf]

• A two-stage embedding model for recommendation with multimodal auxiliary information - Juan Ni, Zhenhua Huang, Yang Hu, Chen Lin. Information Sciences, Jan 2022

• Multi-modal Knowledge Graphs for Recommender Systems - Rui Sun, Xuezhi Cao, Yan Zhao, Junchen Wan, Kun Zhou, Fuzheng Zhang, Zhongyuan Wang and Kai Zheng. CIKM, Oct 2020 [pdf]

• User-Video Co-Attention Network for Personalized Micro-video Recommendation -Shang Liu, Zhenzhong Chen, Hongyi Liu, Xinghai Hu. WWW, May 2019

• Personalized Fashion Recommendation with Visual Explanations based on Multimodal Attention Network - Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, Hongyuan Zha. SIGIR, Jul 2019

• Multimodal Representation Learning for Recommendation in Internet of Things - Zhenhua Huang, Xin Xu, Juan Ni, Honghao Zhu, and Cheng Wang. IEEE IoTJ, Sep 2019

• GraphCAR: Content-aware Multimedia Recommendation with Graph Autoencoder - Qidi Xu, Fumin Shen, Li Liu, Heng Tao Shen. SIGIR, Jun 2018

Modality-specific Recommendation

Text-based Recommendation

Review-based

• Counterfactual Review-based Recommendation - Kun Xiong, Wenwen Ye, Xu Chen, Yongfeng Zhang, Wayne Xin Zhao, Binbin Hu, Zhiqiang Zhang, Jun Zhou. CIKM, Nov 2021 | [pdf] | [code]

• Reviews Meet Graphs: Enhancing User and Item Representations for Recommendation with Hierarchical Attentive Graph Neural Network - Chuhan Wu, Fangzhao Wu, Tao Qi, Suyu Ge, Yongfeng Huang, Xing Xie. EMNLP-IJCNLP, Nov 2019 | [pdf] | [code]

• NRPA: Neural Recommendation with Personalized Atention - Hongtao Liu, Fangzhao Wu, Wenjun Wang, Xianchen Wang, Pengfei Jiao, Chuhan Wu, Xing Xie. SIGIR, Jul 2019 | [pdf] | [code]

• Recommendation Based on Review Texts and Social Communities: A Hybrid Model - Zhenyan Ji, Huaiyu Pi, Wei Wei, Bo Xiong, Marcin Woźniak, Robertas Damasevicius. IEEE Access, Feb 2019 | | [code]

• A Context-Aware User-Item Representation Learning for Item Recommendation - Libing Wu, Cong Quan, Chenliang Li, Qian Wang, Bolong Zheng, Xiangyang Luo. TOIS, Jan 2019 | [pdf] | [code]

• ANR: Aspect-based Neural Recommender - Jin Yao Chin, Kaiqi Zhao, Shafiq Joty, Gao Cong. CIKM, Oct 2018 | [pdf] | [code]

• PARL: Let Strangers Speak Out What You Like - Libing Wu, Cong Quan, Chenliang Li, Donghong Ji. CIKM, Oct 2018 | [code]

• Multi-Pointer Co-Attention Networks for Recommendation - Yi Tay, Anh Tuan Luu, Siu Cheung Hui. KDD, Jul 2018 | [pdf] | [code]

• Neural Attentional Rating Regression with Review-level Explanations - Chong Chen, Min Zhang, Yiqun Liu, Shaoping Ma. WWW, Apr 2018 | [pdf] | [code]

• Interpretable Convolutional Neural Networks with Dual Local and Global Attention for Review Rating Prediction - Sungyong Seo, Jing Huang, Hao Yang, Yan Liu. RecSys, Aug 2017 | [pdf] | [code]

• Joint Deep Modeling of Users and Items Using Reviews for Recommendation - Lei Zheng, Vahid Noroozi, Philip S. Yu. WSDM, Feb 2017 | [pdf] | [code]

• Convolutional Matrix Factorization for Document Context-Aware Recommendation - Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, Hwanjo Yu. RecSys, Sep 2016 | [pdf] | [code]

• Review-Aware Neural Recommendation with Cross-Modality Mutual Attention - Songyin Luo, Xiangkui Lu, Jun Wu, Jianbo Yuan. CIKM, Oct. 2021

• Learning Hierarchical Review Graph Representations for Recommendation - Yong Liu, Susen Yang, Yinan Zhang, Chunyan Miao, Zaiqing Nie, Juyong Zhang. IEEE TKDE, Apr 2021 | [pdf]

• Improving Explainable Recommendations by Deep Review-Based Explanations - Sixun Ouyang, Aonghus Lawlor. IEEE Access, Apr 2021 | [pdf]

• Neural Unified Review Recommendation with Cross Attention - Hongtao Liu, Wenjun Wang, Hongyan Xu, Qiyao Peng, Pengfei Jiao. SIGIR, Jul 2020

• DAML: Dual Attention Mutual Learning between Ratings and Reviews for Item Recommendation - Donghua Liu, Jing Li, Bo Du, Jun Chang, Rong Gao. KDD, Aug 2019

• Attentive Aspect Modeling for Review-Aware Recommendation - Xinyu Guan, Zhiyong Cheng, Xiangnan He, Yongfeng Zhang, Zhibo Zhu, Qinke Peng, Tat-Seng Chua. TOIS, Mar 2019 | [pdf]

• Coevolutionary Recommendation Model: Mutual Learning between Ratings and Reviews -Yichao Lu, Ruihai Dong, Barry Smyth. WWW, Apr 2018 | [pdf]

Title, abstract, tag

• Aligning Dual Disentangled User Representations from Ratings and Textual Content, Nhu-Thuat Tran, Hady W. Lauw. KDD, Aug 2022 | [pdf]
• Graph Neural Network for Tag Ranking in Tag-enhanced Video Recommendation - Qi Liu, Ruobing Xie, Lei Chen, Shukai Liu, Ke Tu, Peng Cui, Bo Zhang, Leyu Lin. CIKM, Oct 2020 | [pdf] | [code]
• Leveraging Title-Abstract Attentive Semantics for Paper Recommendation - Guibing Guo, Bowei Chen, Xiaoyan Zhang, Zhirong Liu, Zhenhua Dong, Xiuqiang He. AAAI, Apr 2020 | [pdf]
• Interactive resource recommendation algorithm based on tag information - Qing Xie. Feng Xiong. Tian Han. Yongjian Liu. Lin Li. Zhifeng Bao. WWW, Feb 2018
• Tag2Word: Using Tags to Generate Words for Content Based Tag Recommendation - Yong Wu, Yuan Yao, Feng Xu, Hanghang Tong, Jian Lu. CIKM, Oct 2016 | [pdf]

Image-based Recommendation

• CausalRec: Causal Inference for Visual Debiasing in Visually-Aware Recommendation - Ruihong Qiu, Sen Wang, Zhi Chen, Hongzhi Yin, and Zi Huang. MM, Oct 2021 | [pdf] | [code]

• VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback - Ruining He, Julian McAuley. AAAI, Feb 2016 | [pdf] | [code]

• Image and Video Understanding for Recommendation and Spam Detection Systems - Aman Gupta, Sirjan Kafle, Di Wen, Dylan Wang, Sumit Srivastava, Suhit Sinha, Nikita Gupta, Bharat Jain, Ananth Sankar, Liang Zhang. KDD, Aug 2020

• Exploring the Power of Visual Features for the Recommendation of Movies - Mohammad Hossein Rimaz, Mehdi Elahi, Farshad Bakhshandegan Moghadam, Christoph Trattner, Reza Hosseini, Marko Tkalčič . UMAP, Jun 2019 | [pdf]

• Visually-Aware Personalized Recommendation using Interpretable Image Representations -Charles Packer, Julian McAuley, Arnau Ramisa. arxiv, 2018 | [pdf]

• DeepStyle: Learning User Preferences for Visual Recommendation -Qiang Liu, Shu Wu, Liang Wang. SIGIR, Aug 2017 | [pdf]

• Where to Go Next for Recommender Systems? ID-vs. Modality-based recommender models revisited -Zheng Yuan, Fajie Yuan, Yu Song. SIGIR, July 2023 | [pdf] [code]

• Exploring Adapter-based Transfer Learning for Recommender Systems: Empirical Studies and Practical Insights -Junchen Fu, Fajie Yuan, Yu Song. WSDM, March 2024 | [pdf] [code]

Public Datasets (Link verified ✔️)

• [Amazon product data 2014] (http://jmcauley.ucsd.edu/data/amazon/links.html)
  • Features processed version of Baby/Sports/Elec etc.
• [Amazon product data 2018] (https://nijianmo.github.io/amazon/index.html)
• [PixelRec] (https://github.com/westlake-repl/PixelRec)
• [MicroLens] (https://github.com/westlake-repl/MicroLens)
• [NineRec] (https://github.com/westlake-repl/NineRec)