NVIDIA BioNeMo Framework v2.7

Updates & Improvements

• Evo2 model improvements:

  • Context, tensor and data parallelism support in the prediction endpoint as well as support for context lengths over 8192 #1123. Fixes #910 and #1048.

  • LoRA fine-tuning by @gabenavarro: #980. Note: internal CI coverage of LoRA convergence is still a work in progress; therefore, we cannot guarantee convergence.

  • Fix a 2x memory-usage issue during Evo2 generation: NVIDIA-NeMo/NeMo#14515

  • Add flash-decode support in inference: #1000

  • Update Rotary Embedding and sequence-length defaults to address incorrect checkpoint conversion: NVIDIA-NeMo/NeMo#14514

  • Improvements to tag masking in the Evo2 loss: #1008

  • Support for Spike-no-more to improve training stability: #1011

• Added a header to SCDL archives, providing improved provenance tracking and supporting future releases. It also adds tracking of AnnData API coverage in SCDL tests.
  This header stores metadata about the archive and its composite arrays, including a version; the array lengths and data types; and information about the RowFeatureIndexes. This adds the features necessary to fix #999 as well as to implement simple bit-packing of the rowptr, colptr, and data arrays. It should also make SCDL more secure, enable strict compatibility checking, and open the door to further performance improvements: #1030

• bionemo-geometric has been deprecated and removed. The molecular-featurization tooling in this package has moved to cuik-molmaker.

Known Issues

• We have removed libtiff from the container due to a known vulnerability, CVE-2025-9900. libtiff isn't directly used in any BioNeMo code; however, users might face issues with e.g. Pillow or other common image-manipulation libraries inside this container.

What's Changed

• fix esm2 finetuning tutorial by @yzhang123 in #1002
• Better masking in evo2, and default use of more robust megatron loss by @jstjohn in #1008
• aws-cli install refactor in Docker container, version bump by @trvachov in #1001
• Add ESM2 Finetuning Benchmark Configuration by @nvmvle in #964
• Add README and SpeedTest. by @edawson in #1005
• update codeowners by @yzhang123 in #1009
• update evo2 zero shot notebook by @yzhang123 in #1003
• Hyena Inference Updates to support Flash Decode by @jstjohn in #1000
• Bump NeMo by @farhadrgh in #1012
• feat: added lora to evo2 by @gabenavarro in #980
• Code for general single cell benchmarking by @polinabinder1 in #969
• Evo2 spike-no-more support by @jstjohn in #1011
• Fix typos in ESM2 finetune notebook by @balvisio in #1023
• Amoradza/rearrange by @moradza in #1025
• updating setup tools in pyproject.toml of scspeedtest by @polinabinder1 in #1033
• Clean up repo root and unused internal tooling by @pstjohn in #1037
• run mdformat and bump ruff pre-commit by @pstjohn in #1036
• Add validation tests for ESM2 fine-tuning benchmark partial-conv by @nvmvle in #1010
• Add subquadratic-ops support by @farhadrgh in #1043
• revert formatting of docs page by @pstjohn in #1044
• Refactor license check script by @pstjohn in #1045
• fix confidence MDLM sampling and add ar and margin by @nvdreidenbach in #1029
• Edawson/scdl schema by @edawson in #1030
• Entropic time scheduler by @btrentini in #1024
• add changed files to github workflow by @pstjohn in #1038
• make sure framework ci runs on schedule events by @pstjohn in #1059
• Bionemo Core version update by @polinabinder1 in #1050
• remove ./internal copy in Dockerfile by @pstjohn in #1063
• update license check and ignore .gitignore in framework ci by @pstjohn in #1062
• enable per output token likelihood prediction for evo2 by @yzhang123 in #1057
• remove extra -- from pipeline parallel command by @jwilber in #1067
• Add bionemo-recipes by @pstjohn in #1052
• remove esm2 native recipe by @pstjohn in #1068
• Update README.md by @taras-sereda in #1070
• unskip evo2 tests by @broland-hat in #1058
• add some fixes for recipes ci by @pstjohn in #1072
• add bionemo-recipes CI summary job by @pstjohn in #1073
• checkout submodules in framework changed-actions by @pstjohn in #1074
• Remove index.md markdown to prevent 3.13 requirement by @jwilber in #1076
• skip get-pr-info if we're not on a pull request branch by @pstjohn in #1079
• Release v2.7rc1 by @yzhang123 in #1056
• updating esm2 native recipe by @pstjohn in #1078
• Change NeMo to new organization NVIDIA-NeMo. by @cspades in #1089
• ESM-2 Accelerate Recipes by @pstjohn in #1080
• temporarily comment out failing checkpoint by @jwilber in #1093
• feat: Add Evo2 fine-tuning partial-conv benchmarking by @nvmvle in #1028
• pin transformers in esm2 golden values by @pstjohn in #1100
• Add ESM-2 model gradient tests by @pstjohn in #1077
• Add initial model convergence workflow for BioNeMo tests by @jwilber in #1102
• ESM-2 mfsdp recipe expanded tests by @pstjohn in #1101
• bump NeMo by @farhadrgh in #1105
• Fix infer_evo2 argparse error. by @jstjohn in #1110
• moving FAST_CI_MODE flag to pytest script by @dorotat-nv in #1111
• add emacs-nox [bnm2717] by @broland-hat in #1107
• run all recipes on a single node by @pstjohn in #1108
• fix memory error on nightly a bnm2712 by @broland-hat in #1104
• pin torch version in amplify by @pstjohn in #1112
• with zstd compression lets just run the steps individually by @pstjohn in #1117
• only xfail thd tests if cuda arch is unsupported by @pstjohn in #1118
• Update model name in L1_3B_ddp.yaml by @jwilber in #1119
• Add wandb mode arg by @jwilber in #1120
• adding shared_eden_dataloader. by @yzhang123 in #1109
• [BENCHMAKRS] fix jet evo2 pretrain for partial conv by @dorotat-nv in #1115
• add partial conv tests to esm2_accelerate recipe by @pstjohn in #1122
• make sure we initialize accelerator before model by @pstjohn in #1132
• Optimize load() by avoiding redundant hash checks and unpacking by @antonvnv in #1081
• [BENCHMARKS] fixing evo2 finetune config by @dorotat-nv in #1097
• rename nvfsdp to mfsdp globally by @pstjohn in #1137
• Moving bionemo-core loads to bionemo-scdl, so that it's fully standalone. by @polinabinder1 in #1133
• Pbinder/changes for scdl profile by @polinabinder1 in #1124
• Remove kwargs in amplify forward pass by @pstjohn in #1141
• Refactor accelerate test suite, add amplify tests by @pstjohn in #1138
• update root README.md for open file count error by @broland-hat in #1034
• Update amplify model, add loss tests by @pstjohn in #1135
• nit: fix amplify config by @pstjohn in #1150
• add gitlab dashboard trigger workflow by @jwilber in #1153
• Add a generic ViT model to BioNeMo Recipes. by @cspades in #1103
• Add THD input format support to ESM-2 model by @pstjohn in #1149
• bionemo-recipes folder restructure by @pstjohn in #1151
• Evo2 predict support for CP, TP and DP with new test coverage. by @jstjohn in #1123
• rename first two recipe deliverables by @pstjohn in #1158
• Add Unit Test for Genformer Training Script by @nvmvle in #943
• Jwilber/recipes convergence tests lepton by @jwilber in #1159
• add wandb support for any arg + step timing by @jwilber in #1161
• Update VERSION for scdl by @polinabinder1 in #1162
• add mxfp8 padding and tests for esm-2 by @pstjohn in #1160
• Update CODEOWNERS by @pstjohn in #1163
• switch to 25.08 for unit tests by @pstjohn in #1157
• Pstjohn/accelerate config fixes by @pstjohn in #1154
• moving sub-package out of test by @polinabinder1 in #1164
• update devcontainer for recipes by @pstjohn in #1165
• remove xfail flag from esm2 fp8 test by @pstjohn in #1167
• Update esm2_native_te recipe, add configurable streaming dataset and non-torchrun support by @pstjohn in #1168
• Jwilber/add esm2 configs by @jwilber in #1166
• pin megatron-fsdp and add backwards tests by @pstjohn in #1176
• add option to select lepton node_group/cluster by @jwilber in #1177
• update image in local recipe tests by @pstjohn in #1179
• Add FP8 and THD support to the esm-2 native recipe by @pstjohn in #1175
• add fp8 decorator to esm2 tests by @pstjohn in #1183
• re-organising Github CI labels and markers by @dorotat-nv in #771
• Add checkpointing documentation by @balvisio in #1042
• Adds checkpointing for ESM2 by @jomitchellnv in #1169
• Update configs to better support wandb by @jwilber in #1178
• bug template - adding instructions how to get git commit hash by @dorotat-nv in #1156
• ensure branch resolves correctly for commit_sha by @jwilber in #1186
• Consolidate perf logging and add final averaged metrics output by @pstjohn in #1184
• Refactor convergence test gh action to allow for gpu selection by @jwilber in #1187
• adds attn mask type by @jomitchellnv in #1188
• add position ids to esm2 collator by @pstjohn in #1189
• evo2_test_flaky_on_l40s [bnm2827] by @broland-hat in #1180
• [BIONEMO-2473] Added tests for Evo2 LoRA fine-tuning by @balvisio in #1060

New Contributors

• @edawson made their first contribution in #1005
• @gabenavarro made their first contribution in #980
• @moradza made their first contribution in #1025
• @btrentini made their first contribution in #1024
• @taras-sereda made their first contribution in #1070
• @antonvnv made their first contribution in #1081

Full Changelog: v2.6.3...v2.7