Initial commit

Add files via upload Microsoft mandatory file remove zip file Code from BiomedParse Update README.md adding figure scripts improve readme Add files via upload Add inference code Update readme Add figure plots and readme ignore plots ignor large file ignore result Configure Git LFS and add large files Track large JSON files with LFS adding back results Add demo dataset Cleaned up a conda environment Update config with correct normalization Conda environment setup without MPI fixed typo fix target_dist.json fix target_dist.json Fix conda environment for clip Change utils folder to utilities folder Medical image preprocessing Update README with dataset list Fix formatting issue Update README.md Adds Usage and License Notices Update README.md Fix for compatible with numpy Update README.md
microsoft · Nov 16, 2024 · e45760b · e45760b
1 parent 9c00c75
commit e45760b
Show file tree

Hide file tree

Showing 35 changed files with 321 additions and 152 deletions.
diff --git a/README.md b/README.md
@@ -1,58 +1,46 @@
 # **BiomedParse**
-:grapes: \[[Read Our arXiv Paper](https://arxiv.org/abs/2405.12971)\] &nbsp; :apple: \[[Check Our Demo](https://microsoft.github.io/BiomedParse/)\]
+This is the official repository for "A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities"
+\[[Read Our arXiv Paper](https://arxiv.org/abs/2405.12971)\] \[[Check Our Demo](https://microsoft.github.io/BiomedParse/)\]
 
 ## Installation
+```sh
+git clone https://github.com/microsoft/BiomedParse.git
+```
 
 ### Conda Environment Setup
-Create a new conda environment
+#### Option 1: Directly build the conda environment
+Under the project directory, run
 ```sh
-conda create -n biomedparse python=3.9
+conda env create -f environment.yml
+```
+
+#### Option 2: Create a new conda environment from scratch
+```sh
+conda create -n biomedparse python=3.9.19
 conda activate biomedparse
 ```
 
 Install Pytorch
 ```sh
 conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
 ```
-In case there is issue with detectron2 installation in the following, make sure your pytorch version is compatible with CUDA version on your machine at https://pytorch.org/.
+In case there is issue with detectron2 installation, make sure your pytorch version is compatible with CUDA version on your machine at https://pytorch.org/.
 
 Install dependencies
 ```sh
 pip install -r assets/requirements/requirements.txt
-pip install -r assets/requirements/requirements_custom.txt
-```
-
-### Install Docker
-
-In order to make sure the environment is set up correctly, we use run BiomedParse on a Docker image. Follow these commands to install Docker on Ubuntu:
-
-```sh
-sudo apt update
-sudo apt install apt-transport-https ca-certificates curl software-properties-common
-curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
-sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
-sudo apt update
-apt-cache policy docker-ce
-sudo apt install docker-ce
 ```
 
-## Prepare Docker Environment
-
-Specify the project directories in `docker/README.md`.
-
-Run the following commands to set up the Docker environment:
-
-```sh
-bash docker/docker_build.sh
-bash docker/docker_run.sh
-bash docker/setup_inside_docker.sh
-source docker/data_env.sh 
-```
+## Dataset
+BiomedParseData was created from preprocessing publicly available biomedical image segmentation datasets. Check a subset of our processed datasets on HuggingFace: https://huggingface.co/datasets/microsoft/BiomedParseData. For the source datasets, please check the details here: [BiomedParseData](assets/readmes/DATASET.md). As a quick start, we've samples a tiny demo dataset at biomedparse_datasets/BiomedParseData-Demo
 
-## Dataset Description and Preparation
+## Model Checkpoints
+We host our model checkpoints on HuggingFace here: https://huggingface.co/microsoft/BiomedParse. Please expect future updates of the model as we are making it more robust and powerful based on feedbacks from the community. We recomment using the latest version of the model. Please put the model in the pretrained folder when runing the code.
 
+## Finetune on Your Own Data
+While BiomedParse can take in arbitrary image and text prompt, it can only reasonably segment the targets that it has learned during pretraining! If you have a specific segmentation task that the latest checkpint doesn't do well, here is the instruction on how to finetune it on your own data. 
 ### Raw Image and Annotation
-For each dataset, put the raw image and mask files in the following format
+BiomedParse expects images and ground truth masks in 1024x1024 PNG format. For each dataset, put the raw image and mask files in the following format
 ```
 ├── biomedparse_datasets
     ├── YOUR_DATASET_NAME
@@ -61,7 +49,6 @@ For each dataset, put the raw image and mask files in the following format
         ├── test
         └── test_mask
 ```
-
 Each folder should contain .png files. The mask files should be binary images where pixels != 0 indicates the foreground region.
 
 ### File Name Convention
@@ -81,8 +68,12 @@ One image can be associated with multiple masks corresponding to multiple target
 [TARGET] is the name of the target with spaces replaced by '+'. E.g. "tube" or "chest+tube". Make sure "_" doesn't appear in [TARGET].
 
 ### Get Final Data File with Text Prompts
-In biomedparse_datasets/create-customer-datasets.py, specify YOUR_DATASET_NAME.
-Once the create-custom-coco-dataset script is run, the dataset folder should be of the following format
+In biomedparse_datasets/create-customer-datasets.py, specify YOUR_DATASET_NAME. Run the script with
+```
+cd biomedparse_datasets
+python create-customer-datasets.py
+```
+After that, the dataset folder should be of the following format
 ```
 ├── dataset_name
         ├── train
@@ -97,28 +88,24 @@ Once the create-custom-coco-dataset script is run, the dataset folder should be
 In datasets/registration/register_biomed_datasets.py, simply add YOUR_DATASET_NAME to the datasets list. Registered datasets are ready to be added to the training and evaluation config file configs/biomed_seg_lang_v1.yaml. Your training dataset is registered as biomed_YOUR_DATASET_NAME_train, and your test dataset is biomed_YOUR_DATASET_NAME_test.
 
 
-## Training
-
-To train the model using the example BiomedParseData-Demo, run:
+## Train BiomedParse
+To train the BiomedParse model, run:
 
 ```sh
 bash assets/scripts/train.sh
 ```
+This will continue train the model using the training datasets you specified in configs/biomed_seg_lang_v1.yaml
 
-### Customizing Training Settings
-See Training Parameters section for example.
-
-## Evaluation
-
-To evaluate the model on the example BiomedParseData-Demo, run:
-
+## Evaluate BiomedParse
+To evaluate the model, run:
 ```sh
 bash assets/scripts/eval.sh
 ```
+This will continue evaluate the model on the test datasets you specified in configs/biomed_seg_lang_v1.yaml. We put BiomedParseData-Demo as the default. You can add any other datasets in the list.
 
-## Inference
+## Run Inference
 Example inference code is provided in `example_prediction.py`. We provided example images in `examples` to load from. Model checkpoint is provided in `pretrained` to load from. Model configuration is provided in `configs/biomedparse_inference.yaml`.
-
+ 
 ### Model Setup
 ```sh
 from PIL import Image
@@ -170,69 +157,32 @@ Detection and recognition inference code are provided in `inference_utils/output
 
 
 
-## Reproducing Results
-To reproduce the exact results presented in the paper, use the following table of parameters and configurations:
+<!-- ### Install Docker
 
-| Configuration Parameter     | Description                              | Value                              |
-|-----------------------------|------------------------------------------|------------------------------------|
-| Data Directory              | Path to the dataset                      | `/path/to/data/`                   |
-| Pre-trained Model Checkpoint| Path to the pre-trained model checkpoint | `pretrained/biomed_parse.pt`    |
-| Training Script             | Script used for training                 | `assets/scripts/train.sh`          |
-| Evaluation Script           | Script used for evaluation               | `assets/scripts/eval.sh`           |
-| Inference Script            | Script for running inference             | `example_prediction.py` |
-| Environment Variables       | Required environment variables           | See below                          |                     |
-| Configuration File          | Configuration file for the model         | `configs/biomed_seg_lang_v1.yaml` |
-| Training Parameters         | Additional training parameters           | See below                          |
-| Output Directory            | Directory to save outputs                | `outputs/`                         |
+In order to make sure the environment is set up correctly, we use run BiomedParse on a Docker image. Follow these commands to install Docker on Ubuntu:
 
-### Environment Variables
 ```sh
-export DETECTRON2_DATASETS=data/
-export DATASET=data/
-export DATASET2=data/
-export VLDATASET=data/
-export PATH=$PATH:data/coco_caption/jre1.8.0_321/bin/
-export PYTHONPATH=$PYTHONPATH:data/coco_caption/
-export OMPI_ALLOW_RUN_AS_ROOT=1
-export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
-#export WANDB_KEY=YOUR_WANDB_KEY # Provide your wandb key here
-```
-
-### Training Parameters
-```sh
-CUDA_VISIBLE_DEVICES=0,1,2,3 mpirun -n 4 python entry.py train \
-    --conf_files configs/biomedseg/biomed_seg_lang_v1.yaml \
-    --overrides \
-    FP16 True \
-    RANDOM_SEED 2024 \
-    BioMed.INPUT.IMAGE_SIZE 1024 \
-    MODEL.DECODER.HIDDEN_DIM 512 \
-    MODEL.ENCODER.CONVS_DIM 512 \
-    MODEL.ENCODER.MASK_DIM 512 \
-    TEST.BATCH_SIZE_TOTAL 4 \
-    TRAIN.BATCH_SIZE_TOTAL 4 \
-    TRAIN.BATCH_SIZE_PER_GPU 1 \
-    SOLVER.MAX_NUM_EPOCHS 20 \
-    SOLVER.BASE_LR 0.00001 \
-    SOLVER.FIX_PARAM.backbone False \
-    SOLVER.FIX_PARAM.lang_encoder False \
-    SOLVER.FIX_PARAM.pixel_decoder False \
-    MODEL.DECODER.COST_SPATIAL.CLASS_WEIGHT 1.0 \
-    MODEL.DECODER.COST_SPATIAL.MASK_WEIGHT 1.0 \
-    MODEL.DECODER.COST_SPATIAL.DICE_WEIGHT 1.0 \
-    MODEL.DECODER.TOP_SPATIAL_LAYERS 10 \
-    MODEL.DECODER.SPATIAL.ENABLED True \
-    MODEL.DECODER.GROUNDING.ENABLED True \
-    LOADER.SAMPLE_PROB prop \
-    FIND_UNUSED_PARAMETERS True \
-    ATTENTION_ARCH.SPATIAL_MEMORIES 32 \
-    MODEL.DECODER.SPATIAL.MAX_ITER 0 \
-    ATTENTION_ARCH.QUERY_NUMBER 3 \
-    STROKE_SAMPLER.MAX_CANDIDATE 10 \
-    WEIGHT True \
-    RESUME_FROM pretrained/biomed_parse.pt
+sudo apt update
+sudo apt install apt-transport-https ca-certificates curl software-properties-common
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
+sudo apt update
+apt-cache policy docker-ce
+sudo apt install docker-ce
 ```
 
+## Prepare Docker Environment
+
+Specify the project directories in `docker/README.md`.
 
+Run the following commands to set up the Docker environment:
 
+```sh
+bash docker/docker_build.sh
+bash docker/docker_run.sh
+bash docker/setup_inside_docker.sh
+source docker/data_env.sh 
+``` -->
 
+## Usage and License Notices
+The model described in this repository is provided for research and development use only. The model is not intended for use in clinical decision-making or for any other clinical use, and the performance of the model for clinical use has not been established. You bear sole responsibility for any use of this model, including incorporation into any product intended for clinical use.
diff --git a/assets/readmes/DATASET.md b/assets/readmes/DATASET.md
@@ -0,0 +1,42 @@
+# **BiomedParseData**
+
+BiomedParseData was created from preprocessing publicly available biomedical image segmentation datasets. 
+
+These datasets are provided pre-formatted for convenience. For additional information about the datasets or their licenses, please reach out to the owners:
+| Dataset                               | URL |
+|---------------------------------------|-----|
+| amos22                                | [https://amos22.grand-challenge.org/](https://amos22.grand-challenge.org/) |
+| MSD (Medical Segmentation Decathlon)  | [http://medicaldecathlon.com/](http://medicaldecathlon.com/) |
+| KiTS23                                | [https://github.com/neheller/kits23](https://github.com/neheller/kits23) |
+| BTCV                                  | [https://www.synapse.org/#!Synapse:syn3193805/wiki/217790](https://www.synapse.org/#!Synapse:syn3193805/wiki/217790) |
+| COVID-19 CT                           | [https://www.kaggle.com/datasets/andrewmvd/covid19-ct-scans](https://www.kaggle.com/datasets/andrewmvd/covid19-ct-scans) |
+| LIDR-IDRI                             | [https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI) |
+| ACDC                                  | [https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html](https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html) |
+| M&Ms                                  | [https://www.ub.edu/mnms/](https://www.ub.edu/mnms/) |
+| PROMISE12                             | [cite https://doi.org/10.1016/j.media.2013.12.002](https://doi.org/10.1016/j.media.2013.12.002) |
+| LGG                                   | [https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation](https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation) |
+| COVID-QU-Ex                           | [https://www.kaggle.com/datasets/anasmohammedtahir/covidqu](https://www.kaggle.com/datasets/anasmohammedtahir/covidqu) |
+| QaTa-COV19                            | [https://www.kaggle.com/datasets/aysendegerli/qatacov19-dataset](https://www.kaggle.com/datasets/aysendegerli/qatacov19-dataset) |
+| SIIM-ACR Pneumothorax Segmentation    | [https://www.kaggle.com/datasets/vbookshelf/pneumothorax-chest-xray-images-and-masks](https://www.kaggle.com/datasets/vbookshelf/pneumothorax-chest-xray-images-and-masks) |
+| Chest Xray Masks and Labels Dataset   | [https://datasetninja.com/chest-xray](https://datasetninja.com/chest-xray) |
+| COVID-19 Radiography Database         | [https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography-database](https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography-database) |
+| CAMUS                                 | [https://www.creatis.insa-lyon.fr/Challenge/camus/index.html](https://www.creatis.insa-lyon.fr/Challenge/camus/index.html) |
+| BUSI                                  | [https://scholar.cu.edu.eg/?q=afahmy/pages/dataset](https://scholar.cu.edu.eg/?q=afahmy/pages/dataset) |
+| FH-PS-AOP                             | [https://zenodo.org/records/7851339#.ZEH6eHZBztU](https://zenodo.org/records/7851339#.ZEH6eHZBztU) |
+| CDD-CESM                              | [https://www.cancerimagingarchive.net/collection/cdd-cesm/](https://www.cancerimagingarchive.net/collection/cdd-cesm/) |
+| PolypGen                              | [https://www.synapse.org/#!Synapse:syn26376615/wiki/613312](https://www.synapse.org/#!Synapse:syn26376615/wiki/613312) |
+| NeoPolyp                              | [https://www.kaggle.com/c/bkai-igh-neopolyp/data](https://www.kaggle.com/c/bkai-igh-neopolyp/data) |
+| ISIC 2018                             | [https://challenge2018.isic-archive.com/task1/](https://challenge2018.isic-archive.com/task1/) |
+| UwaterlooSkinCancer                   | [Skin Cancer Detection \| Vision and Image Processing Lab \| University of Waterloo](https://uwaterloo.ca) |
+| OCT-CME                               | [https://www.kaggle.com/datasets/zeeshanahmed13/intraretinal-cystoid-fluid](https://www.kaggle.com/datasets/zeeshanahmed13/intraretinal-cystoid-fluid) |
+| REFUGE                                | [https://bitbucket.org/woalsdnd/refuge/src](https://bitbucket.org/woalsdnd/refuge/src) |
+| G1020                                 | [https://www.dfki.uni-kl.de/g1020](https://www.dfki.uni-kl.de/g1020) |
+| DRIVE                                 | [https://drive.grand-challenge.org/](https://drive.grand-challenge.org/) |
+| GlaS                                  | [https://warwick.ac.uk/fac/cross_fac/tia/data/glascontest/](https://warwick.ac.uk/fac/cross_fac/tia/data/glascontest/) |
+| PanNuke                               | [https://jgamper.github.io/PanNukeDataset/](https://jgamper.github.io/PanNukeDataset/) |
+| FUMPE                                 | [https://figshare.com/collections/FUMPE/4107803/1](https://figshare.com/collections/FUMPE/4107803/1) |
+| TotalSegmentator                      | [https://github.com/wasserth/TotalSegmentator](https://github.com/wasserth/TotalSegmentator) |
+| BraTS2023                             | [https://www.synapse.org/#!Synapse:syn51156910/wiki/621282](https://www.synapse.org/#!Synapse:syn51156910/wiki/621282) |
+| AbdomenCT-1K                          | [https://github.com/JunMa11/AbdomenCT-1K](https://github.com/JunMa11/AbdomenCT-1K) |
+| US Simulation & Segmentation          | [https://www.kaggle.com/datasets/ignaciorlando/ussimandsegm](https://www.kaggle.com/datasets/ignaciorlando/ussimandsegm) |
+| CDD-CESM                              | [https://www.cancerimagingarchive.net/collection/cdd-cesm/](https://www.cancerimagingarchive.net/collection/cdd-cesm/) |
diff --git a/assets/requirements/requirements.txt b/assets/requirements/requirements.txt
@@ -7,14 +7,14 @@ scikit-learn==1.3.1
 pandas==2.0.3
 timm==0.4.12
 numpy==1.26.4
-einops==0.7.0
+einops==0.8.0
 fvcore==0.1.5.post20221221
 transformers==4.34.0
 sentencepiece==0.1.99
 ftfy==6.1.1
 regex==2023.10.3
 nltk==3.8.1
-#mpi4py==3.1.5
+mpi4py==3.1.5
 vision-datasets==0.2.2
 cython==3.0.2
 pycocotools==2.0.7
@@ -29,11 +29,9 @@ kornia==0.7.0
 deepspeed==0.10.3
 #wandb==0.15.12
 infinibatch==0.1.1
+open-clip-torch==2.26.1
+git+https://github.com/MaureenZOU/detectron2-xyz.git
 #gradio==3.42.0
-# --extra-index-url https://download.pytorch.org/whl/cu124
-# torch
-# torchvision
-# torchaudio
 #torch==2.3.1 #2.0.1 
 #torchvision==0.15.2 
 #torchaudio==2.0.2

diff --git a/assets/requirements/requirements_custom.txt b/assets/requirements/requirements_custom.txt
@@ -1,6 +1,6 @@
 git+https://github.com/cocodataset/panopticapi.git
 git+https://github.com/openai/CLIP.git
-git+https://github.com/arogozhnikov/einops.git
+#git+https://github.com/arogozhnikov/einops.git
 #git+https://github.com/facebookresearch/detectron2.git
 git+https://github.com/MaureenZOU/detectron2-xyz.git
 #git+https://github.com/openai/whisper.git
diff --git a/datasets/build.py b/datasets/build.py
@@ -48,7 +48,7 @@
                          InteractiveEvaluator,
 )
 from modeling.utils import configurable
-from utils.distributed import get_world_size
+from utilities.distributed import get_world_size
 
 class JointLoader(torchdata.IterableDataset):
     """

diff --git a/datasets/dataset_mappers/biomed_dataset_mapper.py b/datasets/dataset_mappers/biomed_dataset_mapper.py
@@ -18,7 +18,7 @@
 from detectron2.data import MetadataCatalog
 from pycocotools import mask as coco_mask
 
-from utils import prompt_engineering
+from utilities import prompt_engineering
 from modeling.language import build_tokenizer
 from modeling.language.misc import text_noun_with_prompt_all
 from modeling.utils import configurable

diff --git a/datasets/evaluation/classification_evaluation.py b/datasets/evaluation/classification_evaluation.py
@@ -11,8 +11,8 @@
 
 from detectron2.evaluation.evaluator import DatasetEvaluator
 
-from utils.misc import AverageMeter
-from utils.distributed import get_world_size
+from utilities.misc import AverageMeter
+from utilities.distributed import get_world_size
 
 
 @torch.no_grad()

diff --git a/datasets/evaluation/segmentation_evaluation.py b/datasets/evaluation/segmentation_evaluation.py
@@ -13,7 +13,7 @@
 from detectron2.utils.comm import all_gather, is_main_process
 from detectron2.utils.file_io import PathManager
 from detectron2.evaluation.evaluator import DatasetEvaluator
-from utils.distributed import synchronize
+from utilities.distributed import synchronize
 
 from ..semseg_loader import load_semseg
 

diff --git a/entry.py b/entry.py
@@ -13,7 +13,7 @@
 import random
 import numpy as np
 
-from utils.arguments import load_opt_command
+from utilities.arguments import load_opt_command
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)

diff --git a/environment.yml b/environment.yml
@@ -103,11 +103,11 @@ dependencies:
       - antlr4-python3-runtime==4.9.3
       - appdirs==1.4.4
       - black==21.4b2
-      - clip==1.0
+      - open-clip-torch==2.26.1
       - cloudpickle==3.0.0
       - cython==3.0.2
       - deepspeed==0.10.3
-      - detectron2==0.6
+      - git+https://github.com/MaureenZOU/detectron2-xyz.git
       - diffdist==0.1
       - einops==0.8.0
       - ftfy==6.1.1
@@ -129,12 +129,10 @@ dependencies:
       - omegaconf==2.3.0
       - opencv-python==4.8.1.78
       - pandas==2.0.3
-      - panopticapi==0.1
       - pathspec==0.12.1
       - pillow==9.4.0
       - portalocker==2.10.1
       - py-cpuinfo==9.0.0
-      - pyarrow==13.0.0
       - pycocotools==2.0.7
       - pydantic==1.10.18
       - pydot==3.0.1