Create freesolv and mnsol subsets by jaclark5 · Pull Request #118 · OpenFreeEnergy/openfe-benchmarks

jaclark5 · 2026-03-26T14:38:32Z

Not all of the freesolv and mnsol datasets fit the needs of OpenFF benchmarking (e.g., outside of element coverage) using OpenFE (e.g, charged). This PR defines common subsets to be used in benchmarking in a repeatable way. For both FreeSolv and MNSol there are two subsets:

All systems that meet our filtering requirements
A down selected subset of systems with a sampling of functional groups for quick benchmarking.

This PR also updates:

mnsol-name-to-smiles.py
- standardize chemical names and smiles as long as stereochemistry isn't defined. Otherwise we were accidentally running self ASFEs which we want to filter out.
generate_freesolv_exp_data.py:
- generate 3D compatible inchikeys in the exp*.json
- filter out solvents without defined stereochemistry to avoid results that should be racemic but aren't
ligands.sdf
- Incidental to the updates in generate_freesolv_exp_data.py

The ligands_*.sdf files will be updated in a different PR where I'll check the possibility of accidentally adding hydrogen

Do not review define_freesolv_mnsol_openff_subsets.py this file has been reviewed by our science team. If you really want to take a look I'm open to suggestions, but since this defines OpenFF subsets I don't think it's the OpenFE team's responsibility

Copilot

Pull request overview

Defines repeatable OpenFF-focused subsets for FreeSolv and MNSol solvation benchmarks, and updates FreeSolv experimental data generation to use InChIKeys derived from the distributed 3D SDF structures.

Changes:

Add subset_data discovery/loading to benchmark system metadata and validate subset JSONs in tests.
Introduce data-generation scripts to produce subset_openff_filtered and subset_openff_small for FreeSolv and MNSol.
Update FreeSolv experimental JSON generation to compute (RDKit) InChI/InChIKey from ligands.sdf, and update stored experimental identifiers accordingly.

Reviewed changes

Copilot reviewed 10 out of 13 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
openfe_benchmarks/data/_benchmark_systems.py	Adds `subset*.json` discovery and exposes `subset_data` on `BenchmarkData`.
openfe_benchmarks/tests/test_benchmark_data.py	Adds subset JSON load + basic integrity checks.
openfe_benchmarks/scripts/_example_plan_asfe.py	Example ASFE planning script now filters experimental entries by a named subset.
openfe_benchmarks/data/data_generation/generate_freesolv_exp_data.py	Regenerates experimental FreeSolv JSON using InChIKeys derived from shipped SDF structures.
openfe_benchmarks/data/data_generation/define_mnsol_openff_subset.py	New script to filter/downselect MNSol into OpenFF benchmarking subsets.
openfe_benchmarks/data/data_generation/define_freesolv_openff_subset.py	New script to filter/downselect FreeSolv into OpenFF benchmarking subsets.
openfe_benchmarks/data/benchmark_systems/solvation_set/mnsol_neutral/PREPARATION_DETAILS.md	Documents new subset files and filtering/downselection criteria for MNSol.
openfe_benchmarks/data/benchmark_systems/solvation_set/freesolv/PREPARATION_DETAILS.md	Documents new subset files and filtering/downselection criteria for FreeSolv.
openfe_benchmarks/data/benchmark_systems/solvation_set/freesolv/subset_openff_small.json	Adds the down-selected “small” FreeSolv subset definition.
openfe_benchmarks/data/benchmark_systems/solvation_set/freesolv/experimental_solvation_free_energy_data.json	Updates stored solute InChI/InChIKey values to match SDF-derived identifiers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

jaclark5 · 2026-03-27T14:32:00Z

I'm also expecting feedback from @lilyminium on how the down selection takes place, since these subsets won't likely be changed once we start running.

hannahbaumann

Thanks @jaclark5 ! I've only done a partial review for now and haven't made it through the large define_freesolv_mnsol_openff_subsets.py file, will do that later.

hannahbaumann · 2026-04-22T07:56:33Z

+
+## Subsets
+
+Regenerate all subsets: run `run.sh` in `data_generation/`.


I don't think the run.sh is currently present in that directory.

I updated this!

hannahbaumann · 2026-04-22T07:59:21Z

-## Charging Solutes / Solvents
+## Partial Charges
+
+- Licensing prevented including MNSol experimental values. After properly obtaining the dataset, a user can locally generate the needed experimental data file with the [generate_mnsol_data.py](../../../data_generation/generate_mnsol_data.py) script.


This may need a separate heading since it doesn't specify anything partial charge related.

Good point, I moved it

hannahbaumann · 2026-04-22T07:59:42Z


- Licensing prevented including MNSol experimental values. After properly obtaining the dataset, a user can locally generate the needed experimental data file with the [generate_mnsol_data.py](../../../data_generation/generate_mnsol_data.py) script.
- Experimental uncertainties are set to 0.2 kcal/mol for all neutral entries, following the recommendation in the MNSol documentation.
+Regenerate all subsets: run `run.sh` in `data_generation/`.


As above, file not present.

hannahbaumann · 2026-04-22T08:01:21Z

+Regenerate all subsets: run `run.sh` in `data_generation/`.
+
+### subset_openff_filtered.json
+1587 systems retained from ~3037 MNSol database entries, covering 232 unique solutes across multiple solvent environments. Excluded: 219 charged systems; 528 entries with solvents on the skip list; 313 entries with solvents bearing disqualifying SMIRKS; 198 with missing solvent SMILES; 44 with solutes containing out-of-scope elements; 42 self-solvation entries (solute = solvent); 40 with solvents containing out-of-scope elements; 24 with solutes on the skip list; 18 with undefined solvent stereochemistry; 10 with disqualifying solute SMIRKS; 10 with undefined solute stereochemistry; and 4 with missing solute SMILES. Entries may satisfy multiple exclusion criteria; totals are not additive.


maybe add a brief explanation what the skip list is.

hannahbaumann · 2026-04-22T08:04:12Z

+    "mnsol-0019": {
+        "mnsol No.": "0019",
+        "solute_name": "4-methyl-2-pentanone",
+        "solvent_name": "4methyl2pentanone",


This means the subset small does include self-solvation entries?

0.0 ... good catch, I just when through the MNSol dataset again and standardized names and smiles to prevent duplicates.

hannahbaumann · 2026-04-22T08:16:50Z

        elif solvent_inchikey not in mol_dict:
            logger.warning(
-                f"Solvent '{solvent_inchikey}' not found in SDF; skipping network '{network_name}'"
+                f"Solvent '{solvent_inchikey}' not found in SDF; skipping network '{transformation_name}'"


Suggested change

f"Solvent '{solvent_inchikey}' not found in SDF; skipping network '{transformation_name}'"

f"Solvent '{solvent_inchikey}' not found in SDF; skipping transformation '{transformation_name}'"

… subsets

hannahbaumann

Thanks @jaclark5 , lgtm!

jaclark5 added 7 commits March 26, 2026 10:25

Create freesolv and mnsol subsets

0187a5c

Update subset

c7ebfac

Update preparation details

2681287

Add subset detection

518b9ea

Merge branch 'main' into add_solvation_subsets

df92c60

Add use of subsets into plan_asfe

aa2af3c

Update exp*freesolv*.json with 3D based inchikeys

37de099

jaclark5 marked this pull request as ready for review March 27, 2026 13:52

jaclark5 requested a review from Copilot March 27, 2026 13:52

Copilot started reviewing on behalf of jaclark5 March 27, 2026 13:52 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

Update contributing expectations

85b7bf8

jaclark5 requested review from IAlibay and jthorton and removed request for IAlibay March 27, 2026 14:06

jaclark5 and others added 2 commits March 27, 2026 10:21

Apply suggestions from code review

78bd782

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update path in defeine*subsets.py

8d35454

jaclark5 self-assigned this Mar 27, 2026

jaclark5 requested review from IAlibay and hannahbaumann March 27, 2026 14:27

jaclark5 added 6 commits March 31, 2026 11:13

Update subsets

36b4f98

Update subsets using UMAP swapping

7340aa2

Update preparation details solvation

dca6c9f

Update subsets

2a9ed59

Update _example_plan_asfe.py with subset

4fb1306

Update toolkit registry for _example_plan_asfe.py

c12a05e

hannahbaumann reviewed Apr 22, 2026

View reviewed changes

jaclark5 added 2 commits April 22, 2026 11:47

Canonicalize MNSol smiles and names

305090c

Update generate mnsol to exclude undefined stereochemistry and update…

ec48589

… subsets

jaclark5 added 3 commits April 22, 2026 14:08

Update run instructions

5423852

reformat mnsol preparation details

8d60d2b

Update explanations

3da82de

jaclark5 requested a review from hannahbaumann April 22, 2026 18:15

IAlibay reviewed May 20, 2026

View reviewed changes

Comment thread openfe_benchmarks/data/benchmark_systems/solvation_set/mnsol_neutral/PREPARATION_DETAILS.md

jaclark5 and others added 4 commits May 20, 2026 16:56

Update gitignore

a847c5a

Update subset sampling

cf307bc

Update docs to correctly reflect missing ChemEnv

a639204

Merge branch 'main' into add_solvation_subsets

8b7b228

hannahbaumann approved these changes Jun 11, 2026

View reviewed changes

jaclark5 and others added 4 commits June 16, 2026 14:50

Update subsets to remove duplicate exp data

ec15ab5

Fix SMILES

69cb77d

Merge branch 'main' into add_solvation_subsets

14c15c0

Update SDF files

d543238

jaclark5 merged commit 509bafe into main Jun 17, 2026
6 checks passed

jaclark5 deleted the add_solvation_subsets branch June 17, 2026 17:47


		## Subsets

		Regenerate all subsets: run `run.sh` in `data_generation/`.

	f"Solvent '{solvent_inchikey}' not found in SDF; skipping network '{transformation_name}'"
	f"Solvent '{solvent_inchikey}' not found in SDF; skipping transformation '{transformation_name}'"

Uh oh!

Conversation

jaclark5 commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jaclark5 commented Mar 27, 2026

Uh oh!

hannahbaumann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hannahbaumann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jaclark5 commented Mar 26, 2026 •

edited

Loading