HyFormer-Net: A Synergistic CNN-Transformer for Breast Lesion Analysis

Official PyTorch implementation for "HyFormer-Net: A Synergistic CNN-Transformer with Interpretable Multi-Scale Fusion for Breast Lesion Segmentation and Classification in Ultrasound Images".

This repository provides the source code, trained models, and instructions to reproduce the results presented in our paper. HyFormer-Net is a hybrid deep learning framework designed for simultaneous breast lesion segmentation and classification, with a strong focus on quantitative interpretability and real-world clinical deployment.

---

Architecture Overview

HyFormer-Net features a dual-branch encoder that synergistically combines a CNN (EfficientNet-B3) for local feature extraction and a Transformer (Swin Transformer) for global context modeling. These features are progressively integrated via our proposed multi-scale fusion blocks. The attention-gated decoder refines segmentation boundaries while providing intrinsic interpretability.

---

Key Features & Contributions

• Hybrid CNN-Transformer Architecture: Leverages the complementary strengths of CNNs and Transformers for robust feature learning.
• Multi-Task Learning: Jointly optimizes for both segmentation and classification, improving overall performance.
• Dual-Pipeline Interpretability: Provides both quantitative and qualitative evidence of the model's reasoning process:
  • Intrinsic decoder attention aligns with ground truth masks (mean IoU: 0.86).
  • Grad-CAM visualizations highlight clinically relevant features for classification.
• Exceptional Ensemble Performance: Achieves 90.2% Dice Score and perfect 100% Malignant Recall on the BUSI dataset, rivaling expert-level performance.
• Systematic Domain Adaptation Study: Provides the first practical guidelines for adapting hybrid models to new clinical environments, showing 92.5% performance recovery with only 10% target data.

---

How to Cite

If you find this work useful in your research, please consider citing our paper:

@misc{rahman2025hyformer,
    title   = {HyFormer-Net: A Synergistic CNN-Transformer with Interpretable Multi-Scale Fusion for Breast Lesion Segmentation and Classification in Ultrasound Images}, 
    author  = {Mohammad Amanour Rahman},
    year    = {2025},
    eprint  = {2511.01013},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV},
    doi     = {10.48550/arXiv.2511.01013},
    url     = {[https://arxiv.org/abs/2511.01013](https://arxiv.org/abs/2511.01013)}
}