Skip to content

[COMMSENG'24, TMI'24] Interactive Computer-Aided Diagnosis using LLMs

License

Notifications You must be signed in to change notification settings

zhaozh10/ChatCAD

Repository files navigation

[COMMSENG'24, TMI'24] Interactive Computer-Aided Diagnosis using LLMs

This repo includes official implementations of ChatCAD and ChatCAD+

Paper

[Nature COMMSENG] ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models by Sheng Wang, Zihao Zhao, Xi Ouyang, Tianming Liu, Qian Wang, Dinggang Shen

(a) Overview of our proposed strategy. The image is processed by various networks to generate diverse outputs, which are then transformed into text descriptions. The descriptions, served as a link between visual and linguistic information, are combined as inputs to a large language model (LLM). With its ability to reason and its knowledge of the medical field, the LLM can provide a condensed report. (b) Interactive explanations and medical advice from ChatCAD.

[IEEE TMI] ChatCAD+: Towards a Reliable and Universal Interactive CAD using LLMs by Zihao Zhao*, Sheng Wang*, Jinchen Gu*, Yitao Zhu*, Lanzhuju Mei, Zixu Zhuang, Zhiming Cui, Qian Wang, Dinggang Shen

Overview of our proposed ChatCAD+ system. (a) For patients seeking a diagnosis, ChatCAD+ generates reliable medical reports based on the input medical image(s) by referring to local report database. (b) Additionally, for any inquiry from patients, ChatCAD+ retrieves related knowledge from online database and lets large language model generate reliable response.

Introduction

This repository provides the official implementation of some components of ChatCAD+:

  • Modality identification Open in Colab
  • Chinese version Interactive CAD of Chest X-rays
  • LLM-based knowledge retrieval
  • An easy-deploy local web ui based on Gradio

Resources

  • We would like to thank Merck Manual Professional who make all these medical knowledge public, we sorted their website for easier usage: here
  • A BART-based model that has the capability to translate chest X-ray reports into Chinese well [link]

Usage

weights&others

  • R2GenCMN: r2gcmn_mimic-cxr.pth and annotation.json
  • PCAM weights: JFchexpert.pth
  • Place annotation.json under ./r2g/ and pre-trained weights under ./weights/
  • For template retrieval system, please download MIMIC-CXR reports from official website and organize them into a dictionary, save as report_en_dict.json under the ./

You can either find them from original repository or dowload from Google Drive

Deploy local web ui

  • pip install -r requirements.txt
  • implement web.py and load your openai api-key


- Would like some diagnostic results? upload image via left panel --> wait for your report


- ChatCAD+ will answer your question with a reference from Merck Manucal Professional


Citation

If you find our work useful, please consider giving a star ⭐ and citation.

@article{wang2023chatcad,
  title={Chatcad: Interactive computer-aided diagnosis on medical image using large language models},
  author={Wang, Sheng and Zhao, Zihao and Ouyang, Xi and Wang, Qian and Shen, Dinggang},
  journal={arXiv preprint arXiv:2302.07257},
  year={2023}
}

@article{zhao2023chatcad,
      title={ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs},
      author={Zihao Zhao and Sheng Wang and Jinchen Gu and Yitao Zhu and Lanzhuju Mei and Zixu Zhuang and Zhiming Cui and Qian Wang and Dinggang Shen},
      journal={arXiv preprint arXiv:2305.15964},
      year={2023},
}

Acknowledgment

Our implementation (including coming version) is based on the following codebases. We gratefully thank the authors for their wonderful works.

R2GenCMN, PCAM, CSNet.

About

[COMMSENG'24, TMI'24] Interactive Computer-Aided Diagnosis using LLMs

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages