README.md

awesome-mmps

Corpus of resources for multimodal machine learning with physiological signals.

Any additions, corrections, or concerns please submit an issue. For additions to the list, please provide the relevant information. Thank you! :)

---

Table of Contents

• Publications and Preprints

  • EEG 🧠 + X
  • ECG 🫀 + X
  • EDA 💦 + X
  • Eye Movement 👁️ + X
  • EOG 👁️ + X
  • EMG 💪 + X
  • Other 🪧 + X

• Datasets

• Laboratories

• Citation

Publications-and-Preprints

EEG 🧠 + X

• Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition, IEEE TCDS 2021; [EEG, Eye Movement, Peripheral Physiological Signals, ECG]
• Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis, ICONIP 2018; [EEG, Eye Movement]
• Correlated Attention Networks for Multimodal Emotion Recognition, IEEE BIBM 2018; [EEG, Eye Movement]
• Investigating Sex Differences in Classification of Five Emotions from EEG and Eye Movement Signals, IEEE EMBC 2019; [EEG, Eye Movement]
• Can Brain Signals Reveal Inner Alignment with Human Languages?, EMNLP Findings 2023; [EEG, Language]
• Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification, AAAI 2022; [EEG, Language]
• Decoding EEG Brain Activity for Multi-Modal Natural Language Processing, Frontiers in Human Neuroscience 2021; [EEG, Eye Movement, Language]
• Advancing NLP with Cognitive Language Processing Signals, arxiv 2019; [EEG, Language, Gaze]
• Recognition of emotions using multimodal physiological signals and an ensemble deep learning model, Computer Methods and Programs in Biomedicine 2017; [EEG, Peripheral Physiological Signals]
• Multi-modal emotion analysis from facial expressions and electroencephalogram, Computer Vision and Image Understanding 2016; (EEG, Facial Expressions)
• An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals, Sensors 2022; [EEG, ECG, GSR]
• Emotion recognition based on multi-modal physiological signals and transfer learning, Frontiers in Neuroscience 2022; [EEG, RSP, GSR, Photo-Plethysmograph]
• Multimodal Affective States Recognition Based on Multiscale CNNs and Biologically Inspired Decision Fusion Model, IEEE Transactions on Affective Computing 2023; [EEG, Peripheral Physiological Signals]
• A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG, Journal of Neural Engineering 2017; [EEG, EOG]
• Multimodal Emotion Recognition Using Deep Neural Networks, ICONIP 2017; [EEG, Eye Movement]
• EmotionMeter: A Multimodal Framework for Recognizing Human Emotions, IEEE Transactions on Cybernetics 2019; [EEG, Eye Movement]
• A Deep Learning Architecture for Temporal Sleep Stage Classification Using Multivariate and Multimodal Time Series, IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; [EEG, EOG, EMG]
• Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels, ACM 2023; [EEG, Eye Movement]
• Emotion Transformer Fusion: Complementary Representation Properties of EEG and Eye Movements on Recognizing Anger and Surprise, IEEE BIBM 2023; [EEG, Eye Movement]
• Multimodal Physiological Signals Fusion for Online Emotion Recognition, ACM Multimedia 2023; [EEG, ECG, GSR]
• Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition, ACM Multimedia 2023; [EEG, Eye Movement, GSR, Respiration, ECG]
• Stress Classification by Multimodal Physiological Signals Using Variational Mode Decomposition and Machine Learning, Hindawi Journal of Healthcare Engineering 2021; [ECG, EEG]
• Language Mapping using tEEG and EEG Data with Convolutional Neural Networks, IEEE 2022; [EEG, tEEG]
• Understanding language-elicited EEG data by predicting it from a fine-tuned language model, NAACL 2019; [EEG, ERP, Language]
• Validation and Interpretation of a Multimodal Drowsiness Detection System Using Explainable Machine Learning, Computer Methods and Programs in Biomedicine 2023; [EEG, ECG, EOG]
• A Classification Model for Sensing Human Trust in Machines Using EEG and GSR, ACM Transactions on Interactive Intelligent Systems 2018; [EEG, GSR]
• Evaluating the Stressful Commutes Using Physiological Signals and Machine Learning Techniques, IEEE 3ICT 2022; [HR, BP, EEG]
• Review of Machine and Deep Learning Techniques in Epileptic Seizure Detection using Physiological Signals and Sentiment Analysis, ACM 2022; [iEEG, EEG]
• ECG and EEG based detection and multilevel classification of stress using machine learning for specified genders: A preliminary study, PLoS One 2023; [EEG, ECG]
• Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Models, CHIL 2023; [EEG, ECG, PSG]
• Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
• An Efficient Multimodal Emotion Identification Using FOX Optimized Double Deep Q-Learning, Wireless Personal Communications 2023; [GSR, ECG, and EEG]
• EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer, arxiv 2024; [EEG, Text]
• Are EEG-to-Text Models Working?, arxiv 2024; [EEG, Text]
• Aligning Semantic in Brain and Language: A Curriculum Contrastive Method for Electroencephalography-to-Text Generation, IEEE Transactions on Neural Systems and Rehabilitation Engineering 2023, [EEG, Text]
• A Multitask Framework for Emotion Recognition Using EEG and Eye Movement Signals with Adversarial Training and Attention Mechanism, IEEE BIBM 2023; [EEG, Eye Movement]
• Temporal-Spatial Prediction: Pre-Training on Diverse Datasets for EEG Classification, ICASSP 2024; [EEG]
• Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion Recognition, arxiv 2024; [EEG]
• EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model, arxiv 2024; [EEG]
• AC-CfC: An attention-based convolutional closed-form continuous-time neural network for raw multi-channel EEG-based emotion recognition, Biomedical Signal Processing and Control 2024; [EEG]
• EEG-GPT: Exploring Capabilities of Large Language Models for EEG Classification and Interpretation, arxiv 2024; [EEG, Language]
• Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI, ICLR 2024; [EEG, Language]
• Naturalistic Emotion Recognition Using EEG and Eye Movements, Springer Nature Singapore 2023; [EEG, Eye Movement]
• EmotionKD: A Cross-Modal Knowledge Distillation Framework for Emotion Recognition Based on Physiological Signals, ACM 2023; [EEG, GSR]
• Negative emotion recognition using multimodal physiological signals for advanced driver assistance systems, Artificial Life and Robotics 2023; [ECG, EDA, EEG, and fNIRS]
• Physiological Fusion Net: Quantifying Individual VR Sickness with Content Stimulus and Physiological Response, IEEE International Conference on Image Processing (ICIP) 2019; [Vision, EEG, ECG, GSR]
• DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation, Neurips 2023; [EEG, Language]
• DreamDiffusion: Generating High-Quality Images from Brain EEG Signals, arxiv 2023; [EEG, Vision]
• Multi-Signal Reconstruction Using Masked Autoencoder From EEG During Polysomnography, IEEE BCI 2023; [EEG, EOG, Chin EMG, Event Makers]
• Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network, IOP 2020; [EOG, EMG, GSR, RSP, BVP, TMP, EEG]
• Comparative analysis of physiological signals and electroencephalogram (EEG) for multimodal emotion recognition using generative models, IEEE STSIVA 2014; [EEG, EOG, EMG, GSR, HR]
• Large Language Models for Time Series: A Survey, arxiv 2024; [EEG, ECG, Text, Vision]
• An Exploratory Study of Multimodal Physiological Data in Jazz Improvisation Using Basic Machine Learning Techniques, arxiv 2024; [EDA, EEG]
• Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition, ICASSP 2024; [EEG, Eye Movement]
• Functional Emotion Transformer for EEG-Assisted Cross-Modal Emotion Recognition, ICASSP 2024; [EEG, Eye Movement]
• EEG-Transformer: Self-attention from transformer architecture for decoding EEG of imagined speech, IEEE BCI 2022; [EEG, Speech]
• Towards Voice Reconstruction from EEG during Imagined Speech, AAAI 2023; [EEG, Speech]
• Toward Fully-End-to-End Listened Speech Decoding from EEG Signals, InterSpeech 2024; [EEG, Speech]
• Enhancing Affective Representations Of Music-Induced Eeg Through Multimodal Supervision And Latent Domain Adaptation, ICASSP 2022; [EEG, Audio]
• Scaling Law in Neural Data: Non-Invasive Speech Decoding with 175 Hours of EEG Data, arxiv 2024; [EEG, Speech]
• Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals, IEEE ICASSP Workshops 2023; [EEG, ECG, GSR]
• EEG-Based Multimodal Emotion Recognition: A Machine Learning Perspective, IEEE Transactions on Instrumentation and Measurement 2024; [EEG, ECG, EMG]
• SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals, ICML 2024; [ECG, EEG, EOG, EMG, Respiratory Sensors]
• A generative foundation model for five-class sleep staging with arbitrary sensor input, arxiv 2024; [EEG, EOG, EMG]
• NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals, arxiv 2024; [EEG, Text]
• Explainable Artificial Intelligence on Biosignals for Clinical Decision Support, KDD 2024; [EEG, ECG]
• Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective, arxiv 2024; [EEG, ECG, Audio, Text, Image]
• EEG-Language Modeling for Pathology Detection, arxiv 2024; [EEG, Text]
• An ASR-based Hybrid Approach for Auditory Attention Decoding, Lund University Publications 2024; [EEG, Audio]
• SEE: Semantically Aligned EEG-to-Text Translation, arxiv 2024; [EEG, Text]
• Neural Speech Tracking in EEG: Integrating Acoustics and Linguistics for Hearing Aid Users, Lund University Publications 2024; [EEG, Audio]
• LLM-enhanced Multi-teacher Knowledge Distillation for Modality-Incomplete Emotion Recognition in Daily Healthcare, IEEE Journal of Biomedical and Health Informatics 2024; [EEG, GSR]
• Nested Deep Learning Model Towards A Foundation Model for Brain Signal Data, arxiv 2024; [EEG, MEG]
• EEG Emotion Copilot: Pruning LLMs for Emotional EEG Interpretation with Assisted Medical Record Generation, arxiv 2024; [EEG, Text]
• Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs), arxiv 2024; [EEG, Text]
• EEGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training, OpenReview 2024; [EEG, Text]
• ARIEL: Brain-Computer Interfaces meet Large Language Models for Emotional Support Conversation, Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization 2024; [EEG, Text]
• Promoting cross-modal representations to improve multimodal foundation models for physiological signals, Medical Foundation Models (AIM-FM) workshop at NeurIPS 2024; [EEG, EMG, EOG, ECG]
• DPD (DePression Detection) Net: a deep neural network for multimodal depression detection, Health Information Science and Systems 2024; [EEG, Speech]
• Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation, PLOS Computational Biology 2024; [EEG, Speech]
• Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement, arxiv 2024; [EEG, IMU, EMG, EOG, ECG]
• Position: Addressing Ethical Challenges and Safety Risks in GenAI-Powered Brain-Computer Interfaces, GenAI for Health Workshop @ NeurIPS 2024; [EEG, Text]
• A Hybrid Artificial Intelligence System for Automated EEG Background Analysis and Report Generation, IEEE Journal of Biomedical and Health Informatics 2024; [EEG, Text]
• PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals, IEEE International Conference on Biomedical and Health Informatics 2024; [EEG, EMG, EOG]
• Decoding text from electroencephalography signals: A novel Hierarchical Gated Recurrent Unit with Masked Residual Attention Mechanism, Engineering Applications of Artificial Intelligence 2024; [EEG, Text]
• Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion, arxiv 2024; [EEG, fMRI, MEG, Image]
• A Survey of LLMs on Biosignal Applications, arxiv 2024; [ECG, PPG, EEG, IMU, Text]

ECG 🫀 + X

• Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition, IEEE TCDS 2021; [EEG, Eye Movement, Peripheral Physiological Signals, ECG]
• Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?, EACL Findings 2023; [ECG, Language]
• Converting ECG Signals to Images for Efficient Image-text Retrieval via Encoding, Machine Learning for Health 2023; [ECG, Image, Language]
• An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals, Sensors 2022; [EEG, ECG, GSR]
• Automated detection of panic disorder based on multimodal physiological signals using machine learning, ETRI Journal 2022; [ECG, EDA, Respiration, Peripheral Temperature]
• ECGBERT: Understanding Hidden Language of ECGs with Self-Supervised Representation Learning, arxiv 2023; [ECG, Language]
• ECG Language Processing (ELP): A new technique to analyze ECG signals, Computer Methods and Programs in Biomedicine; [ECG, Language]
• Robust Patient Information Embedding and Retrieval Mechanism for ECG Signals, IEEE Access 2020; [ECG, Language]
• Multimodal Physiological Signals and Machine Learning for Stress Detection by Wearable Devices, IEEE MEMEA 2022; [EDA, ECG, PPG]
• Multimodal Physiological Signals Fusion for Online Emotion Recognition, ACM Multimedia 2023; [EEG, ECG, GSR]
• Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition, ACM Multimedia 2023; [EEG, Eye Movement, GSR, Respiration, ECG]
• Automated detection of panic disorder based on multimodal physiological signals using machine learning, ETRI Journal 2023; [ECG, EDA, RESP, PT]
• Stress Classification by Multimodal Physiological Signals Using Variational Mode Decomposition and Machine Learning, Hindawi Journal of Healthcare Engineering 2021; [ECG, EEG]
• Validation and Interpretation of a Multimodal Drowsiness Detection System Using Explainable Machine Learning, Computer Methods and Programs in Biomedicine 2023; [EEG, ECG, EOG]
• A Novel Algorithm for Movement Artifact Removal in ECG Signals Acquired from Wearable Systems Applied to Horses, PLoS One 2015; [ECG, Accelerometer]
• Automatic stress detection in car drivers based on non-invasive physiological signals using machine learning techniques, Neural Computing and Applications 2023; [ECG, EMG, GSR, Respiration Rate]
• Hierarchical extreme puzzle learning machine-based emotion recognition using multimodal physiological signals, Biomedical Signal Processing and Control 2023; [EDA, ECG, TEMP, RESP, EMR]
• Machine Learning–Enabled Multimodal Fusion of Intra-Atrial and Body Surface Signals in Prediction of Atrial Fibrillation Ablation Outcomes, Journal of American Heart Association 2022; [ECG, iECG]
• ECG and EEG based detection and multilevel classification of stress using machine learning for specified genders: A preliminary study, PLoS One 2023; [EEG, ECG]
• Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Models, CHIL 2023; [EEG, ECG, PSG]
• Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
• ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Text, arxiv 2024; [ECG, Text]
• MEIT: Multi-Modal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation, arxiv 2024; [ECG, Text]
• An Efficient Multimodal Emotion Identification Using FOX Optimized Double Deep Q-Learning, Wireless Personal Communications 2023; [GSR, ECG, and EEG]
• A Framework for Cognitive Load Recognition Based on Machine Learning and Multimodal Physiological Signals by Wearable Sensors, IEEE PRML 2023; [ECG, EDA, and PPG]
• A Household Multimodal Physiological Signals Monitoring System, IEEE 2023; [ECG, lLEMG, rLEMG, pBCG]
• ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram, NeurIPS 2023; [ECG, Text]
• Multimodal 12-lead ECG data classification using multi-label DenseNet for heart disease detection, IEEE DSIT 2022; [HRV, ECG, EHR]
• Time Synchronization of Multimodal Physiological Signals through Alignment of Common Signal Types and Its Technical Considerations in Digital Health, J Imaging 2022; [ECG]
• Single-modal and multi-modal false arrhythmia alarm reduction using attention-based convolutional and recurrent neural networks, PLoS One 2024; [PPG, ABP, ECG]
• Negative emotion recognition using multimodal physiological signals for advanced driver assistance systems, Artificial Life and Robotics 2023; [ECG, EDA, EEG, and fNIRS]
• Position Paper: What Can Large Language Models Tell Us about Time Series Analysis, arxiv 2024, [ECG, Text]
• Large Language Model-informed ECG Dual Attention Network for Heart Failure Risk Prediction, arxiv 2024; [ECG, Text]
• Frozen Language Model Helps ECG Zero-Shot Learning, MIDL 2023; [ECG, Text]
• Zero-Shot ECG Diagnosis with Large Language Models and Retrieval-Augmented Generation, ML4H 2023; [ECG, Text]
• Physiological Fusion Net: Quantifying Individual VR Sickness with Content Stimulus and Physiological Response, IEEE International Conference on Image Processing (ICIP) 2019; [Vision, EEG, ECG, GSR]
• Text-to-ECG: 12-Lead Electrocardiogram Synthesis conditioned on Clinical Text Reports, ICASSP 2023; [ECG, Language]
• Large Language Models for Time Series: A Survey, arxiv 2024; [EEG, ECG, Text, Vision]
• Integrating multimodal information in machine learning for classifying acute myocardial infarction, Physiological Measurement 2023; [ECG, Text]
• Multimodal explainable artificial intelligence identifies patients with non-ischaemic cardiomyopathy at risk of lethal ventricular arrhythmias, Scientific reports 2024; [MRI, ECG]
• Multimodal Estimation Of Change Points Of Physiological Arousal During Driving, ICASSP 2023; [ECG, EDA, EMG, HR, BVP]
• Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals, IEEE ICASSP Workshops 2023; [EEG, ECG, GSR]
• Deep learning for ECG classification: A comparative study of 1D and 2D representations and multimodal fusion approaches, Biomedical Signal Processing and Control 2024; [ECG, Image]
• Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement, ICML 2024; [ECG ,Text]
• SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals, ICML 2024; [ECG, EEG, EOG, EMG, Respiratory Sensors]
• ECG Heartbeat Classification Using Multimodal Fusion, IEEE Access 2021; [ECG, Image]
• Multimodal ChatGPT-4V for Electrocardiogram Interpretation: Promise and Limitations, JMIR 2024; [ECG, Image]
• Multimodal ECG heartbeat classification method based on a convolutional neural network embedded with FCA, Nature Scientific Reports 2024; [ECG, Image]
• Large-scale Training of Foundation Models for Wearable Biosignals, ICLR 2024; [PPG, ECG]
• ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis, arxiv 2024; [ECG, Text]
• CardioGPT: An ECG Interpretation Generation Model, IEEE Access 2024; [ECG, Text]
• Harnessing the Power of ChatGPT in Cardiovascular Medicine: Innovations, Challenges, and Future Directions, preprint 2024; [ECG, Text]
• The potential for large language models to transform cardiovascular medicine, The Lancet Digital Health 2024; [ECG, Text]
• Explainable Artificial Intelligence on Biosignals for Clinical Decision Support, KDD 2024; [EEG, ECG]
• Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective, arxiv 2024; [EEG, ECG, Audio, Text, Image]
• Electrocardiogram Report Generation and Question Answering via Retrieval-Augmented Self-Supervised Modeling, arxiv 2024; [Text, ECG]
• Foundation Models for Cardiovascular Disease Detection via BioSignals from Digital Stethoscopes, Research Square 2024; [PCG, ECG]
• From Hospital to Portables: A Universal ECG Foundation Model Built on 10+ Million Diverse Recordings, arxiv 2024; [Text, ECG]
• Electrocardiogram–Language Model for Few-Shot Question Answering with Meta Learning, arxiv 2024; [Text, ECG]
• Promoting cross-modal representations to improve multimodal foundation models for physiological signals, Medical Foundation Models (AIM-FM) workshop at NeurIPS 2024; [EEG, EMG, EOG, ECG]
• Foundation Models in Electrocardiogram: A Review, arxiv 2024; [ECG, Text]
• C-MELT: Contrastive Enhanced Masked Auto-Encoders for ECG-Language Pre-Training, arxiv 2024; [ECG, Text]
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals, arxiv 2021, [ECG, Text]
• Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement, arxiv 2024; [EEG, IMU, EMG, EOG, ECG]
• Precision of artificial intelligence in paediatric cardiology multimodal image interpretation, Cambridge University Press 2024; [ECG, Text]
• A Survey of LLMs on Biosignal Applications, arxiv 2024; [ECG, PPG, EEG, IMU, Text]
• HeartBERT: A Self-Supervised ECG Embedding Model for Efficient and Effective Medical Signal Analysis, arxiv 2024; [ECG, Text]

EDA 💦 + X

• A Multimodal Music Recommendation System with Listeners' Personality and Physiological Signals, ACM/IEEE JCDL 2020; [Heart Rate, EDA, IBI, Skin Temperature, BVP]
• Stress Detection with Deep Learning Approaches Using Physiological Signals, IoT Technologies for Healthcare 2020; [EDA, BVP]
• Multimodal Physiological Signals and Machine Learning for Stress Detection by Wearable Devices, IEEE MEMEA 2022; [EDA, ECG, PPG]
• Automated detection of panic disorder based on multimodal physiological signals using machine learning, ETRI Journal 2023; [ECG, EDA, RESP, PT]
• Transformer-Based Physiological Feature Learning for Multimodal Analysis of Self-Reported Sentiment, ICMI 2022; [EDA, BVP, HR, TEMP]
• Hierarchical extreme puzzle learning machine-based emotion recognition using multimodal physiological signals, Biomedical Signal Processing and Control 2023; [EDA, ECG, TEMP, RESP, EMR]
• A Hybrid Deep Learning Emotion Classification System Using Multimodal Data, Sensors 2023; [EDA, IBI, TEMP, Text, Audio]
• Exploring the Potential of Multimodal Emotion Recognition for Hearing-Impaired Children Using Physiological Signals and Facial Expressions, ICMI Companion 2023; [ST, EDA, BVP, ACC, HR, Vision]
• Performance Exploration of RNN Variants for Recognizing Daily Life Stress Levels by Using Multimodal Physiological Signals, ICMI 2023; [EDA, ST, ACC, HR]
• Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition, IEEE Transactions On Affective Computing 2023; [EDA, BVP, TEMP]
• Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
• A Framework for Cognitive Load Recognition Based on Machine Learning and Multimodal Physiological Signals by Wearable Sensors, IEEE PRML 2023; [ECG, EDA, and PPG]
• Stress appraisal in the workplace and its associations with productivity and mood: Insights from a multimodal machine learning analysis, PLoS One 2024; [EDA, ST, BVP, Accelerometer]
• Effects of Physiological Signals in Different Types of Multimodal Sentiment Estimation, IEEE AC 2023; [Language, EDA, Audio, Vision]
• Negative emotion recognition using multimodal physiological signals for advanced driver assistance systems, Artificial Life and Robotics 2023; [ECG, EDA, EEG, and fNIRS]
• An Exploratory Study of Multimodal Physiological Data in Jazz Improvisation Using Basic Machine Learning Techniques, arxiv 2024; [EDA, EEG]
• Multimodal Estimation Of Change Points Of Physiological Arousal During Driving, ICASSP 2023; [ECG, EDA, EMG, HR, BVP]

Eye Movement 👁️ + X

• Investigating Sex Differences in Classification of Five Emotions from EEG and Eye Movement Signals, IEEE EMBC 2019; [EEG, Eye Movement]
• Decoding EEG Brain Activity for Multi-Modal Natural Language Processing, Frontiers in Human Neuroscience 2021; [EEG, Eye Movement, Language]
• Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis, ICONIP 2018; [EEG, Eye Movement]
• Correlated Attention Networks for Multimodal Emotion Recognition, IEEE BIBM 2018; [EEG, Eye Movement]
• Multimodal Emotion Recognition Using Deep Neural Networks, ICONIP 2017; [EEG, Eye Movement]
• EmotionMeter: A Multimodal Framework for Recognizing Human Emotions, IEEE Transactions on Cybernetics 2019; [EEG, Eye Movement]
• Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels, ACM 2023; [EEG, Eye Movement]
• Emotion Transformer Fusion: Complementary Representation Properties of EEG and Eye Movements on Recognizing Anger and Surprise, IEEE BIBM 2023; [EEG, Eye Movement]
• Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition, ACM Multimedia 2023; [EEG, Eye Movement, GSR, Respiration, ECG]
• Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
• Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models, arxiv 2024; [fMRI, Eye Movement]
• A Multitask Framework for Emotion Recognition Using EEG and Eye Movement Signals with Adversarial Training and Attention Mechanism, IEEE BIBM 2023; [EEG, Eye Movement]
• Naturalistic Emotion Recognition Using EEG and Eye Movements, Springer Nature Singapore 2023; [EEG, Eye Movement]
• Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition, ICASSP 2024; [EEG, Eye Movement]
• Functional Emotion Transformer for EEG-Assisted Cross-Modal Emotion Recognition, ICASSP 2024; [EEG, Eye Movement]
• EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals, NeurIPS 2024; [EEG, fMRI, Video, Text]

EOG 👁️ + X

• Machine-Learning-Based Detection of Craving for Gaming Using Multimodal Physiological Signals: Validation of Test-Retest Reliability for Practical Use, Sensors 2019; [PPG, GSR, EOG]
• A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG, Journal of Neural Engineering 2017; [EEG, EOG]
• A Deep Learning Architecture for Temporal Sleep Stage Classification Using Multivariate and Multimodal Time Series, IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; [EEG, EOG, EMG]
• Validation and Interpretation of a Multimodal Drowsiness Detection System Using Explainable Machine Learning, Computer Methods and Programs in Biomedicine 2023; [EEG, ECG, EOG]
• Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning., IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]
• Multi-Signal Reconstruction Using Masked Autoencoder From EEG During Polysomnography, IEEE BCI 2023; [EEG, EOG, Chin EMG, Event Makers]
• Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network, IOP 2020; [EOG, EMG, GSR, RSP, BVP, TMP, EEG]
• Comparative analysis of physiological signals and electroencephalogram (EEG) for multimodal emotion recognition using generative models, IEEE STSIVA 2014; [EEG, EOG, EMG, GSR, HR]
• SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals, ICML 2024; [ECG, EEG, EOG, EMG, Respiratory Sensors]
• A generative foundation model for five-class sleep staging with arbitrary sensor input, arxiv 2024; [EEG, EOG, EMG]
• Promoting cross-modal representations to improve multimodal foundation models for physiological signals, Medical Foundation Models (AIM-FM) workshop at NeurIPS 2024; [EEG, EMG, EOG, ECG]
• Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement, arxiv 2024; [EEG, IMU, EMG, EOG, ECG]
• PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals, IEEE International Conference on Biomedical and Health Informatics 2024 2024; [EEG, EMG, EOG]

EMG 💪 + X

• A Deep Learning Architecture for Temporal Sleep Stage Classification Using Multivariate and Multimodal Time Series, IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; [EEG, EOG, EMG]
• Automatic stress detection in car drivers based on non-invasive physiological signals using machine learning techniques, Neural Computing and Applications 2023; [ECG, EMG, GSR, Respiration Rate]
• Multi-Signal Reconstruction Using Masked Autoencoder From EEG During Polysomnography, IEEE BCI 2023; [EEG, EOG, Chin EMG, Event Makers]
• Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network, IOP 2020; [EOG, EMG, GSR, RSP, BVP, TMP, EEG]
• Comparative analysis of physiological signals and electroencephalogram (EEG) for multimodal emotion recognition using generative models, IEEE STSIVA 2014; [EEG, EOG, EMG, GSR, HR]
• Multimodal Estimation Of Change Points Of Physiological Arousal During Driving, ICASSP 2023; [ECG, EDA, EMG, HR, BVP]
• SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals, ICML 2024; [ECG, EEG, EOG, EMG, Respiratory Sensors]
• A generative foundation model for five-class sleep staging with arbitrary sensor input, arxiv 2024; [EEG, EOG, EMG]
• Embracing Large Language and Multimodal Models for Prosthetic Technologies, arxiv 2024; [EMG, Image, Video, Text, Audio]
• EMG-to-Speech: Direct Generation of Speech From Facial Electromyographic Signals, IEEE/ACM Transactions on Audio, Speech, and Language Processing 2024; [EMG, Speech]
• Promoting cross-modal representations to improve multimodal foundation models for physiological signals, Medical Foundation Models (AIM-FM) workshop at NeurIPS 2024; [EEG, EMG, EOG, ECG]
• Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement, arxiv 2024; [EEG, IMU, EMG, EOG, ECG]
• PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals, IEEE International Conference on Biomedical and Health Informatics 2024 2024; [EEG, EMG, EOG]

Other 🪧 + X

• Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging, ICML Workshop on Machine Learning for Multimodal Healthcare Data 2023; [CMR, Language]
• Biometric Recognition Using Multimodal Physiological Signals, IEEE 2019; [Heart Rate, Breathing Rate, Palm Electrodermal Activity, Perinasal Perspitation]
• Video-based multimodal spontaneous emotion recognition using facial expressions and physiological signals, CVPR Workshop 2022; [Image, iPPG, Heart Rate]
• Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems, Journal of Medical and Biological Engineering 2020; [Respiratory Belt, Photoplethysmography, Fingertip Temperature]
• Every word counts: A multilingual analysis of individual human alignment with model attention, AACl 2022; [Eye-Tracking, Language]
• Early Life Stress Detection Using Physiological Signals and Machine Learning Pipelines, Biology 2023; [Heart Rate, GSR]
• Comparative Analysis of Emotion Classification Based on Facial Expression and Physiological Signals Using Deep Learning, Applied Sciences 2022; [Facial Expressions, HRV]
• Towards Autonomous Physiological Signal Extraction From Thermal Videos Using Deep Learning, ICMI 2023; [HR, Respiration Rate, Temperature]
• MAD: Multi-Alignment MEG-to-Text Decoding, arxiv 2024; [MEG, Text]
• BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification, InterSpeech 2024; [Respiratory Sound, Text]
• Brant: Foundation Model for Intracranial Neural Signal, NeurIPS 2023; [iEEG]
• BrainBERT: Self-supervised representation learning for intracranial recordings, ICLR 2023; [iEEG]
• Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark, arxiv 2024; [Time Series Agnostic]
• TOTEM: TOkenized T ime Series EMbeddings for General Time Series Analysis, arxiv 2024; [Time Series Agnostic]
• SIM-CNN: Self-Supervised Individualized Multimodal Learning for Stress Prediction on Nurses Using Biosignals, medRxiv 2023; [EDA, HR, ST, Interbeat Interval, BVP, Accelerometer]
• Interpretation of Intracardiac Electrograms Through Textual Representations, CHIL 2024; [EGM, Language]
• Multimodal Risk Prediction with Physiological Signals, Medical Images and Clinical Notes, medRxiv 2023; [Vision, Language, EHR]
• Do Self-Supervised Speech and Language Models Extract Similar Representations as Human Brain?, ICASSP 2024; [ECoG, Speech, Text]
• Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism, IEEE Transactions on Cybernetics 2023; [CNS, ANS]
• Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data, CHIL 2024; [Time Series Agnostic]
• BIOT: Biosignal Transformer for Cross-data Learning in the Wild, Neurips 2023; [Time Series Agnostic]
• PhysioMTL: Personalizing Physiological Patterns using Optimal Transport Multi-Task Regression, CHIL 2022; [Time Series Agnostic]
• Transformers in biosignal analysis: A review, Elsevier Information Fusion 2024; [Time Series Agnostic]
• Towards a Personal Health Large Language Model, arxiv 2024; [Time Series Agnostic]
• Event-Based Contrastive Learning for Medical Time Series, MLHC 2024; [Time Series Agnostic]
• PaPaGei: Open Foundation Models for Optical Physiological Signals, arxiv 2024; [PPG]
• Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data, arxiv 2024; [Time Series Agnostic]
• COCOA: Cross Modality Contrastive Learning for Sensor Data, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2022; [Time Series Agnostic]
• Language Models Still Struggle to Zero-shot Reason about Time Series, arxiv 2024; [Time Series Agnostic]
• TimeSeriesExam: A time series understanding exam, NeurIPS 2024 Time Series in the Age of Large Models Workshop; [Time Series Agnostic]

Datasets

• SEED; [EEG, Eye Movement, Video]
• SEED-IV; [EEG, Eye Movement, Video]
• SEED-VIG; [EEG, EOG]
• SEED-V; [EEG, Eye Movement, Video]
• SEED-GER; [EEG, Eye Movement, Video]
• SEED-FRA; [EEG, Eye Movement, Video]
• ZuCo 2.0; [EEG, Language, Eye Movement]
• DEAP; [EEG, Peripheral Physiological Signals, Video]
• K-EmoCon; [EEG, Peripheral Physiological Signals, Video, Audio]
• AMIGOS; [EEG, ECG, GSR, Video]
• Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels; [EEG, Eye Movement]
• Dataset of Speech Production in intracranial Electroencephalography; [Intracranial EEG, Language]
• Temple University Hospital EEG Seizure Corpus; [EEG, Artifacts]
• ECG & EEG Stress Features; [EEG, ECG]
• A multi-modal open dataset for mental-disorder analysis; [EEG, Language]
• Sleep-EDF; [EEG, EOG, Chin EMG, Event Makers]
• Neurosity EEG Dataset; [EEG]
• ECG-QA; [ECG, Text]
• A Large and Rich EEG Dataset for Modeling Human Visual Object Recognition; [EEG, Image]
• MIMIC-IV-ECG: Diagnostic Electrocardiogram Matched Subset; [ECG, EHR, Text]
• EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy; [EEG, EMG]
• EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More; [EEG, Image, Text]
• Alljoined1 - A dataset for EEG-to-Image decoding; [EEG, Image]
• MindBigData; [EEG, Image]
• Human EEG recordings for 1,854 concepts presented in rapid serial visual presentation streams; [EEG, Image]
• EEG-ImageNet: An Electroencephalogram Dataset and Benchmarks with Image Visual Stimuli of Multi-Granularity Labels; [EEG, Image]
• The Dutch EEG Speech Register Corpus; [EEG, Audio]
• EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts; [EEG, Audio, Video]
• IMPRESSION; [ECG, BVP, GSR, Video, Audio, Eye Gaze]
• EEVR; [PPG, EDA, Video]

Laboratories

• Neural Interfacing Lab @ Maastricht University
• BCMI Lab @ Shanghai Jiao Tong University
• Centre for Language Technology @ University of Copenhagen
• Safe AI Lab @ Carnegie Mellon University
• NeuroMechatronics Lab @ Carnegie Mellon University
• Interactive Computing Lab @ KAIST
• Laboratory for the Neural Mechanisms of Attention @ UC Davis
• HUman Bio-behavioral Signals Lab @ Texas A&M University
• Computational Intelligence & Neural Engineering Lab @ Hanyang University
• Bioelectronics Laboratory @ Incheon National University
• Healthy ML @ MIT
• Auton Lab @ Carnegie Mellon University
• Biomedical Functional Imaging and Neuroengineering Laboratory @ Carnegie Mellon University
• Laboratoire des Systèmes Perceptifs @ École Normale Supérieure
• Brain and AI Team @ Meta
• Computational Arrhythmia Research Laboratory @ Stanford University
• Wu Tsai Institute @ Yale University
• Shah Lab @ Stanford University
• Edward Yoonjae Choi's Lab @ KAIST
• Scalable Health LAbs @ Rice University
• Pattern Recognition & Machine Learning Lab @ Korea University
• Signal Analysis and Interpretation Lab @ USC
• DataLearning Group @ Imperial College London
• Tison Lab @ UCSF
• Rajpurkar Lab @ Harvard University
• Wang Lab @ University of Toronto
• MAILAB @ Korea University
• Health AI @ Google
• Health AI @ Apple
• NIAI Lab @ Hanyang University
• Mobile Systems Research Lab @ University of Cambridge
• Aaqib Saeed's Lab @ Eindhoven University of Technology
• Multisensory Intelligence Research Group @ MIT
• Behavioral Data Science Lab @ University of Washington

Citation

If you found this repository helpful in your research, please cite the following:

@software{Han_A_corpus_of_2024,
author = {Han, William},
month = apr,
title = {{A corpus of resources for Multimodal Learning with Physiological Signals}},
url = {https://github.com/willxxy/awesome-mmps},
version = {2.0.4},
year = {2024}
}