diff --git a/README.md b/README.md index 6e6e9d5..91003dc 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,32 @@ -# Open Free Energy Benchmark Systems +# OpenFE Benchmarks -A set of benchmark systems to validate the OpenFE components. +Curated benchmark inputs and result submission conventions for OpenFE-based free-energy calculations. The repository provides remediated system inputs (proteins, ligands, ligand networks), example scripts/notebooks, and a machine-readable results schema. ## Contents -The `openfe_benchmark` repository contains: - * A file for each system in the benchmark set (e.g. `tyk2.py`). These hold - methods for creating the free energy network and the system components - necessary to calculate the benchmark. - * A [`data`](/Users/jenniferclark/bin/openfe-benchmarks/openfe_benchmarks/data/README.md) directory with a set of PDB files for each host system and SDF files - for each set of ligands for each system. - * A `scripts` directory with examples of how to set up calculations such as: - - bfe: binding free energy - - hfe: hydration free energy - - sfe: solvation free energy (requires external tools) - * `examples` a directory with notebooks for example calculations and analyses +- `openfe_benchmarks/data/` — remediated benchmark inputs (ligands, proteins, networks, preparation notes). +- `openfe_benchmarks/scripts/` — utility scripts and validation helpers. +- `openfe_benchmarks/results/` — canonical result-submission layout and schema (see `results/README.md`). +- `examples/` — runnable notebooks showing common workflows. -## Available Systems +## Quick start -The available systems are indexed in the [benchmark_system_indexing.yml](openfe_benchmarks/data/benchmark_system_indexing.yml) file. This file provides a comprehensive list of the benchmark systems along with tags representing whether they can be used for `bfe` or `sfe` calculations and whether there are cofactors in the system. These systems can also be explored in Python, as demonstrated in the notebook [`1_initializing_benchmark_data.ipynb`](examples/1_initializing_benchmark_data.ipynb). +1. Install the package. No dependencies are needed to access data. + + ```bash + pip install -e . + ``` + +2. Browse benchmarking systems with [data](openfe_benchmarks/data/README.md) +3. Set-up calculations with scripts. +4. Submit benchmarking [results](openfe_benchmarks/results/README.md) ## Additional Resources - **OpenFE Documentation**: https://docs.openfree.energy - **GUFE Documentation**: https://gufe.openfree.energy - **Example Notebooks**: See `examples/` directory for: - - `1_initializing_benchmark_data.ipynb` - How to use the benchmark systems API - - `2_benchmark_data_with_openfe.ipynb` - Using a BenchmarkData system with OpenFE tools - - `building_networks.ipynb` - Creating ligand networks \ No newline at end of file + +## License + +This project is MIT-licensed. See `LICENSE` for details. diff --git a/openfe_benchmarks/results/README.md b/openfe_benchmarks/results/README.md new file mode 100644 index 0000000..f0ebbef --- /dev/null +++ b/openfe_benchmarks/results/README.md @@ -0,0 +1,156 @@ +# OpenFE Benchmarks Results Submission Format + +Purpose +------- +This document defines the structure, required files, metadata schema, and validation expectations for a "submitted result" in this repository. The goal is to make each submission a self‑contained, machine-readable dataset that an external user or automated tool can ingest, validate and cite. + +~~See [OpenFE CLI tutorial](https://docs.openfree.energy/en/latest/tutorials/rbfe_cli_tutorial.html) for more information in how to prepare files for a submission with the plan/gather/run lifecycle and example `gather` output. For visualization and example plots that consume `dg`/`ddg` reports see the [OpenFE plotting tutorial](https://docs.openfree.energy/en/latest/tutorials/plotting.html).~~ + +Summary +-------- +Each submission is a directory under `results/` that MUST contain: a human- and machine-readable metadata YAML (`submission.yaml`), a `computational_results.json` containing `dg` and `ddg` information with units for each benchmark system, and a pointer (DOI or URL) to the full simulation archive (Alchemical Archive / Zenodo) containing the raw JSON/replicate/trajectory assets. + +Minimum Required Layout +------------------------ +results// +- `submission.yaml` # (Required) Metadata + links (see schema) +- `computational_results.json` # (Required) Ligand, dg, ddg, and minimal mbar information + +Note: heavy/binary simulation artifacts (repeat folders, openfe JSON results, trajectories, AlchemicalArchives) SHOULD be deposited in a long-term archive (Zenodo/Alchemical Archive) and referenced from `submission.yaml` rather than included in the git repo. + +submission.yaml — annotated example (required fields + guidance) +---------------------------------------------------------------- +The block below is a single, copy‑pasteable example that documents required and recommended keys inline. Use the exact key names shown. + +```yaml +# REQUIRED: unique, kebab-case identifier for this submission +submission_id: example-tyk2-2025 + +# REQUIRED: short descriptive title +title: Tyk2 RBFE benchmark — OpenFE example submission + +# REQUIRED: comprehensive description of what networks the submission calculated and what is / is not comparable to the norm. +description: This submission presents a RBFE benchmark study for JACS TYK2. The calculations employ the standard OpenFE protocol (SOMD with AMBER-based forcefield, AM1-BCC charges) using 3 independent repeats. Note: this submission focuses on relative free energies only; absolute binding free energies (DG) are derived from a reference ligand and should not be interpreted as stand-alone predictions. + +# REQUIRED: list of contributing authors (name, affiliation; ORCID optional) +authors: + - name: My Name + affiliation: Example Lab + orcid: '0000-0002-XXXX-XXXX' + +# REQUIRED: publication/submission date (ISO 8601) +date: 2025-02-10 + +# REQUIRED: OpenFE version used to produce the gathered reports +openfe_version: 0.8.3 + +# Recommended but useful: force field and partial charge descriptor +forcefield: openff-2.3.0 +partial_charges: am1bcc + +# REQUIRED: mapping of report names +reports: computational_results.json + +# REQUIRED: long-term archive pointer (at least doi or url) +archive: + doi: 10.5281/zenodo.1234567 + archive_provider: zenodo + +# REQUIRED: license for the submission (e.g. CC-BY-4.0) +license: CC-BY-4.0 + +# RECOMMENDED / OPTIONAL metadata +summary: "RBFE benchmark for TYK2 comparing protocols X and Y; includes dg and ddg summary tables." +keywords: [tyk2, rbfe, benchmark, openfe] +``` + +Expected `computational_results.json` +----------------------------------------- + +The file is a JSON object with two top-level arrays: `"DG"` and `"DDG"`. + +### Unit objects + +Both `DG`/`DG_uncertainty` and `DDG`/`DDG_uncertainty` values are encoded as a unit-aware object: + +```json +{ + "magnitude": -0.312, + "unit": "kilocalories_per_mole", + ":is_custom:": true, + "pint_unit_registry": "openff_units" +} +``` + +| key | purpose | +|-----|---------| +| `magnitude` | numerical value | +| `unit` | must be `"kilocalories_per_mole"` | +| `":is_custom:"` | always `true`; marks this as an openff-units Quantity | +| `pint_unit_registry` | always `"openff_units"`; identifies the unit registry used for deserialisation | + +--- + +### `DG` array + +One entry per ligand per system. These are the **per-ligand absolute binding free energies** derived from a network-wide Maximum Likelihood Estimate (MLE) across all DDG edges. + +```json +{ + "ligand": "ejm_50", + "DG": { "magnitude": 0.291, "unit": "kilocalories_per_mole", ":is_custom:": true, "pint_unit_registry": "openff_units" }, + "DG_uncertainty": { "magnitude": 0.099, "unit": "kilocalories_per_mole", ":is_custom:": true, "pint_unit_registry": "openff_units" }, + "system_group": "jacs_set", + "system_name": "tyk2", + "source": "MLE" +} +``` + +| key | purpose | +|-------|---------| +| `ligand` | ligand identifier matching the SDF/benchmark data | +| `DG` | estimated absolute binding free energy (kcal/mol); note this is an MLE-shifted relative quantity, not a true absolute binding free energy — its absolute value is only meaningful relative to the reference ligand within the same network | +| `DG_uncertainty` | 1-sigma propagated uncertainty from the MLE fit | +| `system_group` | benchmark set the system belongs to (e.g. `jacs_set`, `charge_annihilation_set`) | +| `system_name` | protein target (e.g. `tyk2`, `p38`) | +| `source` | estimation method; currently always `"MLE"` | + +--- + +### `DDG` array + +One entry per perturbation edge. These are the **raw pairwise relative free energies** computed directly from simulations, before any network MLE. + +```json +{ + "ligand_a": "ejm_50", + "ligand_b": "ejm_42", + "system_group": "jacs_set", + "system_name": "tyk2", + "repeats": 3, + "DDG": { "magnitude": -0.312, "unit": "kilocalories_per_mole", ":is_custom:": true, "pint_unit_registry": "openff_units" }, + "DDG_uncertainty": { "magnitude": 0.185, "unit": "kilocalories_per_mole", ":is_custom:": true, "pint_unit_registry": "openff_units" }, + "DGs_complex": [-19.161, -19.111, -18.822], + "DGs_solvent": [-18.572, -18.830, -18.756], + "Complex_smallest_mbar_overlaps": [0.134, 0.131, 0.130], + "Complex_smallest_replica_mixing": [0.089, 0.096, 0.092], + "Solvent_smallest_mbar_overlaps": [0.138, 0.138, 0.138], + "Solvent_smallest_replica_mixing": [0.109, 0.105, 0.110] +} +``` + +| key | purpose | +|-------|---------| +| `ligand_a` / `ligand_b` | the two ligands defining the perturbation edge; DDG = DG(ligand_b) − DG(ligand_a) | +| `system_group` / `system_name` | same meaning as in `DG` entries | +| `repeats` | number of independent simulation repeats; determines the length of each per-repeat array below | +| `DDG` | mean relative binding free energy across repeats (kcal/mol) | +| `DDG_uncertainty` | standard error of the mean across repeats | +| `DGs_complex` | per-repeat ΔG of the alchemical transformation in the **protein–ligand complex** (kcal/mol); length equals `repeats` | +| `DGs_solvent` | per-repeat ΔG of the alchemical transformation in the **solvent** leg (kcal/mol); length equals `repeats` | +| `Complex_smallest_mbar_overlaps` | per-repeat **minimum pairwise MBAR overlap** across all alchemical windows in the complex leg; values near 0 indicate a window spacing that is too coarse and may compromise free energy accuracy | +| `Complex_smallest_replica_mixing` | per-repeat **minimum replica-exchange acceptance rate** between neighbouring windows in the complex leg; values near 0 indicate poor mixing and potentially unconverged sampling | +| `Solvent_smallest_mbar_overlaps` | same as `Complex_smallest_mbar_overlaps` but for the solvent leg | +| `Solvent_smallest_replica_mixing` | same as `Complex_smallest_replica_mixing` but for the solvent leg | + +