Update README.md

microsoft · Nov 6, 2024 · b24a7c5 · b24a7c5
1 parent ccca34e
commit b24a7c5
Showing 1 changed file with 16 additions and 77 deletions.
diff --git a/README.md b/README.md
@@ -1,4 +1,5 @@
 # **BiomedParse**
+This is the official repository for "A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities"
 :grapes: \[[Read Our arXiv Paper](https://arxiv.org/abs/2405.12971)\] &nbsp; :apple: \[[Check Our Demo](https://microsoft.github.io/BiomedParse/)\]
 
 ## Installation
@@ -31,12 +32,15 @@ pip install -r assets/requirements/requirements.txt
 ```
 
 ## Dataset
-BiomedParseData was created from preprocessing publicly available biomedical image segmentation datasets. Please check the details here: [BiomedParseData](assets/readmes/DATASET.md). As a quick start, we've samples a tiny demo dataset at biomedparse_datasets/BiomedParseData-Demo
+BiomedParseData was created from preprocessing publicly available biomedical image segmentation datasets. Check a subset of our processed datasets on HuggingFace: https://huggingface.co/datasets/microsoft/BiomedParseData. For the source datasets, please check the details here: [BiomedParseData](assets/readmes/DATASET.md). As a quick start, we've samples a tiny demo dataset at biomedparse_datasets/BiomedParseData-Demo
 
+## Model Checkpoints
+We host our model checkpoints on HuggingFace here: https://huggingface.co/microsoft/BiomedParse. Please expect future updates of the model as we are making it more robust and powerful based on feedbacks from the community. We recomment using the latest version of the model. Please put the model in the pretrained folder when runing the code.
 
-## Prepare Your Own Data
+## Finetune on Your Own Data
+While BiomedParse can take in arbitrary image and text prompt, it can only reasonably segment the targets that it has learned during pretraining! If you have a specific segmentation task that the latest checkpint doesn't do well, here is the instruction on how to finetune it on your own data. 
 ### Raw Image and Annotation
-For each dataset, put the raw image and mask files in the following format
+BiomedParse expects images and ground truth masks in 1024x1024 PNG format. For each dataset, put the raw image and mask files in the following format
 ```
 ├── biomedparse_datasets
     ├── YOUR_DATASET_NAME
@@ -45,7 +49,6 @@ For each dataset, put the raw image and mask files in the following format
         ├── test
         └── test_mask
 ```
-
 Each folder should contain .png files. The mask files should be binary images where pixels != 0 indicates the foreground region.
 
 ### File Name Convention
@@ -65,8 +68,12 @@ One image can be associated with multiple masks corresponding to multiple target
 [TARGET] is the name of the target with spaces replaced by '+'. E.g. "tube" or "chest+tube". Make sure "_" doesn't appear in [TARGET].
 
 ### Get Final Data File with Text Prompts
-In biomedparse_datasets/create-customer-datasets.py, specify YOUR_DATASET_NAME.
-Once the create-custom-coco-dataset script is run, the dataset folder should be of the following format
+In biomedparse_datasets/create-customer-datasets.py, specify YOUR_DATASET_NAME. Run the script with
+```
+cd biomedparse_datasets
+python create-customer-datasets.py
+```
+After that, the dataset folder should be of the following format
 ```
 ├── dataset_name
         ├── train
@@ -87,15 +94,14 @@ To train the BiomedParse model, run:
 ```sh
 bash assets/scripts/train.sh
 ```
-
-### Customizing Training Settings
-See Training Parameters section for example.
+This will continue train the model using the training datasets you specified in configs/biomed_seg_lang_v1.yaml
 
 ## Evaluate BiomedParse
-To evaluate the model on the example BiomedParseData-Demo, run:
+To evaluate the model, run:
 ```sh
 bash assets/scripts/eval.sh
 ```
+This will continue evaluate the model on the test datasets you specified in configs/biomed_seg_lang_v1.yaml. We put BiomedParseData-Demo as the default. You can add any other datasets in the list.
 
 ## Run Inference
 Example inference code is provided in `example_prediction.py`. We provided example images in `examples` to load from. Model checkpoint is provided in `pretrained` to load from. Model configuration is provided in `configs/biomedparse_inference.yaml`.
@@ -151,73 +157,6 @@ Detection and recognition inference code are provided in `inference_utils/output
 
 
 
-## Reproducing Results
-To reproduce the exact results presented in the paper, use the following table of parameters and configurations:
-
-| Configuration Parameter     | Description                              | Value                              |
-|-----------------------------|------------------------------------------|------------------------------------|
-| Data Directory              | Path to the dataset                      | `/path/to/data/`                   |
-| Pre-trained Model Checkpoint| Path to the pre-trained model checkpoint | `pretrained/biomed_parse.pt`    |
-| Training Script             | Script used for training                 | `assets/scripts/train.sh`          |
-| Evaluation Script           | Script used for evaluation               | `assets/scripts/eval.sh`           |
-| Inference Script            | Script for running inference             | `example_prediction.py` |
-| Environment Variables       | Required environment variables           | See below                          |                     |
-| Configuration File          | Configuration file for the model         | `configs/biomed_seg_lang_v1.yaml` |
-| Training Parameters         | Additional training parameters           | See below                          |
-| Output Directory            | Directory to save outputs                | `outputs/`                         |
-
-### Environment Variables
-```sh
-export DETECTRON2_DATASETS=data/
-export DATASET=data/
-export DATASET2=data/
-export VLDATASET=data/
-export PATH=$PATH:data/coco_caption/jre1.8.0_321/bin/
-export PYTHONPATH=$PYTHONPATH:data/coco_caption/
-export OMPI_ALLOW_RUN_AS_ROOT=1
-export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
-#export WANDB_KEY=YOUR_WANDB_KEY # Provide your wandb key here
-```
-
-### Training Parameters
-```sh
-CUDA_VISIBLE_DEVICES=0,1,2,3 mpirun -n 4 python entry.py train \
-    --conf_files configs/biomedseg/biomed_seg_lang_v1.yaml \
-    --overrides \
-    FP16 True \
-    RANDOM_SEED 2024 \
-    BioMed.INPUT.IMAGE_SIZE 1024 \
-    MODEL.DECODER.HIDDEN_DIM 512 \
-    MODEL.ENCODER.CONVS_DIM 512 \
-    MODEL.ENCODER.MASK_DIM 512 \
-    TEST.BATCH_SIZE_TOTAL 4 \
-    TRAIN.BATCH_SIZE_TOTAL 4 \
-    TRAIN.BATCH_SIZE_PER_GPU 1 \
-    SOLVER.MAX_NUM_EPOCHS 20 \
-    SOLVER.BASE_LR 0.00001 \
-    SOLVER.FIX_PARAM.backbone False \
-    SOLVER.FIX_PARAM.lang_encoder False \
-    SOLVER.FIX_PARAM.pixel_decoder False \
-    MODEL.DECODER.COST_SPATIAL.CLASS_WEIGHT 1.0 \
-    MODEL.DECODER.COST_SPATIAL.MASK_WEIGHT 1.0 \
-    MODEL.DECODER.COST_SPATIAL.DICE_WEIGHT 1.0 \
-    MODEL.DECODER.TOP_SPATIAL_LAYERS 10 \
-    MODEL.DECODER.SPATIAL.ENABLED True \
-    MODEL.DECODER.GROUNDING.ENABLED True \
-    LOADER.SAMPLE_PROB prop \
-    FIND_UNUSED_PARAMETERS True \
-    ATTENTION_ARCH.SPATIAL_MEMORIES 32 \
-    MODEL.DECODER.SPATIAL.MAX_ITER 0 \
-    ATTENTION_ARCH.QUERY_NUMBER 3 \
-    STROKE_SAMPLER.MAX_CANDIDATE 10 \
-    WEIGHT True \
-    RESUME_FROM pretrained/biomed_parse.pt
-```
-
-
-
-
-
 <!-- ### Install Docker
 
 In order to make sure the environment is set up correctly, we use run BiomedParse on a Docker image. Follow these commands to install Docker on Ubuntu: