Skip to content

CHANGELOG.md

Latest commit

 

History

History
69 lines (50 loc) · 22.8 KB

CHANGELOG.md

File metadata and controls

69 lines (50 loc) · 22.8 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

  • Annotation sources rebuilt on verified provider contracts (embpy.resources). New offline indexes: molecule.chebi_obo.ChebiOboIndex (ChEBI OBO, has role edges keyed by standard InChIKey), molecule.kegg_index (build_kegg_reference, KeggIndex: KEGG entries keyed by InChIKey through KEGG's own PubChem SIDs, with exact and connectivity match tiers and separate compound-pathway, drug-map and target-pathway families) and gene.reactome.ReactomeIndex (Reactome release files joined on Entrez ID, lowest-level and top-level pathways). New methods: MoleculeAnnotator.lookup_chebi_roles, lookup_kegg, resolve_pubchem_cids_by_inchikey, get_pubchem_xrefs_batch, get_chebi_compounds; GeneAnnotator.get_pathways_batch (one MyGene POST per 250 symbols, HGNC renames recorded as renamed_from); ChEMBLAnnotator.resolve_many and get_*_many batch methods; ChEMBLAnnotator(backend="sqlite", db_path=...), a read-only ChEMBL SQLite backend with the REST return shapes; CellLineAnnotator(depmap_model_csv=..., passports_models=...) and get_cellosaurus_batch. Every result carries a status, and failed requests are reported as service_error/batch_failed (or raise AnnotationServiceError), never as an empty answer.

  • Stage-1 embpy.tl.fusion: ID-aligned concatenation, reusable per-view normalization and weighting, joint PCA, score/rank ensembles, AnnData/static-store adapters, provenance and serializable fitted state.

  • Paired benchmark_fusion and benchmark_fusion_retrieval, with grouped splits, fold/entity results and a cluster-only LINCS validation workflow. Added the fusion guide and executable tutorial.

  • benchmark_embeddings accepts multiple obsm_keys, a fusion operator and groupby. PCA is now fitted inside training folds and rigorous evaluation uses nested hyperparameter search. Consequently reduce_dim no longer writes a full-data PCA to obsm, and rigorous best_params is a list of dictionaries, one per outer fold.

  • StateEmbeddingWrapper fetches the published SE-600M release when no checkpoint or model_folder is given, instead of raising ValueError("Either checkpoint or model_folder must be provided."). state was the one registry key list_available_models() advertised that load() categorically refused to run, while its weights sat public on the Hub -- every other single-cell wrapper downloads on first use. Only the three files needed to run it are fetched (config.yaml, protein_embeddings.pt, se600m_epoch16.ckpt, ~12 GB); the repo also ships an epoch4 checkpoint and safetensors variants, and pulling it whole would cost ~29 GB. huggingface_hub caches, so it is a one-time cost, and naming a local path still short-circuits the download entirely.

  • BioEmbedder.embed_cells(..., model_kwargs={"state": {"checkpoint": ...}}) forwards per-model constructor arguments to the single-cell wrapper. This is what makes the checkpoint-only models reachable at all: StateEmbeddingWrapper and StackWrapper take checkpoint on __init__ and raise from load() without it, and while get_singlecell_wrapper has always forwarded **kwargs, embed_cells had no parameter to pass and _get_or_load_singlecell_wrapper passed nothing -- so state and stack were listed by list_available_models() and could not be run through the public API on any platform. The kwargs form part of the wrapper cache key, so two checkpoints of one model no longer collide on the first one loaded. Note that the arc-state / arc-stack backends are linux-64 only; tests/embpy/models/test_singlecell_models.py::TestSingleCellModelKwargs covers the plumbing rather than a forward pass.

  • embpy.resources.ChEMBLAnnotator (src/embpy/resources/molecule/chembl.py) covers the ChEMBL_37 drug record that MoleculeAnnotator did not reach. MoleculeAnnotator used three endpoints -- molecule, activity, mechanism; the new class adds drug (clinical phase, first approval, routes of administration, availability, prodrug/orphan/first-in-class/withdrawn flags), drug_indication (MeSH + EFO indications with the phase reached per indication), drug_warning (black box warnings and withdrawals, with toxicity class, country and year), atc_class (WHO levels 1-5), molecule_synonyms (INN / USAN / trade names / research codes), metabolism (metabolites and the enzymes responsible), molecule_form (parent and salt forms), target + target_component + protein_classification (targets as gene symbols, UniProt accessions and protein families rather than bare CHEMBL accessions), similarity (Tanimoto analogs), and the full activity field set (assay description, BAO format, source document, ligand efficiency, data-validity flags). Adds derived per-target potency profiles (get_target_profile) and a selectivity summary (summarize_selectivity) that excludes cell-line "targets" before calling a compound's primary target.

  • embpy.tl.annotate_drug_perturbations(adata, column=...) -- the compound-centric counterpart to annotate_drugs. Writes 38 drug_* summary columns to .obs and the full records to .uns["chembl_annotations"], with *_at_limit companion flags and the fetch caps in .uns["chembl_annotation_limits"], following the contract GeneAnnotator already uses for capped counts.

  • MoleculeAnnotator gains the source groups "drug", "target_profile", "metabolism" and "analogs", delegating to ChEMBLAnnotator and landing the result under result["drug"]. sources="all" now includes the first three; "analogs" stays opt-in because a similarity search scans the whole database. MoleculeAnnotator.annotate_adata writes the drug_* columns alongside the existing mol_* ones.

  • embpy.store.actions.compile_action_table is now the shared headless action compiler, with explicit RESOLVED / CONTROL / UNRESOLVED status accounting and JSON sidecars.

  • embpy.resources.gene.control.ControlPolicy and status-aware embedding APIs add clearer handling for control and unresolved identifiers.

  • AlphaGenomeWrapper (Google DeepMind's cloud-API DNA model) and ScoobyWrapper (gagneurlab/scooby, single-cell-resolution DNA sequence model) are now registered in DNA_MODELS/MODEL_REGISTRY under the keys alphagenome, scooby_onek1k, scooby_neurips, and scooby_epicardioids, reachable through BioEmbedder.embed(model=...) and list_available_models() like any other model. Each is gated behind its own optional extra (embpy[alphagenome], embpy[scooby]) and standalone pixi environment (pixi install -e alphagenome / -e scooby) -- neither is pulled in by embpy[all] or the default/gpu/dev pixi environments, since AlphaGenome needs an external ALPHAGENOME_API_KEY and Scooby installs from a git URL.

Changed

  • MoleculeAnnotator.get_chebi_roles is renamed get_chebi_terms: it always returned the molecule's own ChEBI term label (OLS), never roles. get_chebi_roles stays as a deprecated alias (DeprecationWarning); use lookup_chebi_roles for roles. get_kegg_pathways keeps its list return shape but flattens lookup_kegg, which needs a KeggIndex ({"status": "index_required"} without one). get_disease_associations is documented as assay-name keywords (Active rows only, columns by name). get_cross_references queries ChEMBL last, by exact standard InChIKey, and a ChEMBL failure gives status="partial". ChEMBLAnnotator.resolve uses the RDKit InChIKey for SMILES first; flexmatch is opt-in and InChIKey-verified. CellLineAnnotator parses Cellosaurus API v2, resolves accessions first, returns ambiguous for multi-hit name searches, uses the Passports JSON:API filters, and reads DepMap from Model.csv only.

  • GeneAnnotator.get_pathways, annotate and annotate_batch no longer fill reactome from MyGene (ConsensusPathDB, not a Reactome release). Pass reactome_index=ReactomeIndex(...) to get Reactome pathways; without it the reactome list is empty. KEGG and WikiPathways still come from MyGene. Cell-line lookups (get_cellosaurus_info, get_depmap_info, get_passports_info) now return a status dict on a miss instead of {}, and DepMap needs depmap_model_csv (otherwise source_unavailable).

  • docs/notebooks/04_benchmark_models.ipynb rewritten. The old version trained probes against rng.normal(size=16) -- random noise -- and read the resulting R-squared as a model ranking, which ranked nothing. It now embeds the same cells with five single-cell models (pca, scvi, scgpt, geneformer_v2_12L, state) and scores them with two metric families: tl.compute_scib_metrics for structure preservation and tl.run_cell_eval for pairwise agreement, each plotted. Also documents the single-cell environment recipe, since no resolver-satisfiable environment holds helical, arc-state and cell-eval together.

  • embpy/__init__.py is now a PEP-562 lazy facade (audit step 1). import embpy no longer eagerly walks dt, models, pl, pp, resources, tl, or pulls embpy.embedder (3,600 lines, transformers + torch + numpy). Cold import embpy drops from minutes (lustre, observed) to milliseconds. The public API is unchanged: embpy.BioEmbedder, embpy.GeneResolver, from embpy import tl, etc. still resolve -- they load on first access via a module-level __getattr__ (PEP 562). Static analysis is preserved through a TYPE_CHECKING import block. Guarded by tests/embpy/test_boundary.py::{test_import_embpy_does_not_eagerly_load_subpackages, test_import_embpy_does_not_eagerly_load_embedder, test_lazy_bioembedder_attr_access_still_works}.

  • MODEL_REGISTRY and the three DNA species sets (HUMAN_ONLY_MODELS, MOUSE_ONLY_MODELS, MULTI_SPECIES_DNA) moved from src/embpy/embedder.py to the new src/embpy/embedder_registry/flat.py (audit step 2). The optional-dep wrapper imports (Evo, Evo2, Boltz2) and the _HAVE_* flags moved alongside them. from embpy.embedder import MODEL_REGISTRY continues to return the exact same dict object (re-export via from .embedder_registry.flat import ...). Net delta in embedder.py: -216 lines.

  • Registry split into per-modality submodules under src/embpy/embedder_registry/ (audit step 3). flat.py is now a 25-line merge of dna.py (46 entries + the three DNA species sets), protein.py (23 entries, including Boltz-2 structural), molecule.py (14 entries), text.py (5 entries), morphology.py (11 entries), singlecell.py (0 entries today; reserved for the upcoming SingleCellEmbedder facade), and api.py (7 entries). Total: 106 entries; the public MODEL_REGISTRY dict has identical keys and identical (Wrapper, path) tuples vs. the pre-split snapshot (set + value byte-equivalence; iteration order changes only because GENA-LM / NT / HyenaDNA / Caduceus -- previously appended after morphology -- now group with the other DNA entries). Optional-dep gating (EvoWrapper if _HAVE_EVO else None) stays inside the per-modality submodule that owns the entry, so importing one modality's submodule never pays the cost of loading another's optional dependency. Guarded by tests/embpy/test_registry_split.py (snapshot of 106 entries + disjoint-modality + merge-equals-flat + species-sets-in-DNA + same-object-re-export tests).

  • The [scooby] extra no longer lists scooby @ git+https://github.com/gagneurlab/scooby.git. PyPI rejects uploads with URL dependencies, so this one line would have failed the next release; the scooby project on PyPI is an unrelated tool, so it cannot be listed by name either. Install the model package next to the extra: pip install "embpy[scooby]" "scooby @ git+https://github.com/gagneurlab/scooby.git" (README, docs/technical.md and the ScoobyWrapper error messages now say so). [tool.hatch.metadata] allow-direct-references is gone too, so a URL dependency now fails the build instead of the upload. Guarded by tests/embpy/test_extras_consistency.py::TestPublishableToPyPI.

Fixed

  • Annotation uncertainty is preserved: unrecorded ChEMBL flags are None, missing phase is not preclinical, and Cellosaurus sampling site no longer fills tissue of origin. Target profiles fetch complete activity evidence by default, retain every target component and expose truncation when a limit is requested. Exact-match eligibility and a parent-family identity audit make stereochemical/isotope differences explicit so callers can gate annotation transfer. The default AnnData field drug_coconut_member replaces the misleading drug_is_natural_product; see docs/annotation_semantics.md for the changed contracts.

  • PubChem CID resolution no longer puts SMILES in the URL path (every SMILES with / or \ got HTTP 400); it sends smiles as a query parameter. KEGG lookups no longer treat a PubChem CID as a KEGG pubchem: SID (CID 5291, imatinib, used to map to C02226, citraconic acid). MyGene is no longer used as a Reactome source.

  • tl.cell_eval / tl.run_cell_eval now take metric_configs and skip_metrics, routed to MetricsEvaluator.compute rather than the constructor. All **kwargs previously went to __init__, so passing metric_configs raised MetricsEvaluator.__init__() got an unexpected keyword argument 'metric_configs' -- and since metric_configs is the only route to embed_key, the ten ANNDATA_PAIR metrics could not be pointed at an .obsm embedding at all. They silently scored .X instead, which for an embedding comparison is the wrong matrix rather than a missing feature.

  • The protein resolver now retries transient UniProt and MyGene failures, matching what the gene resolver has always done for Ensembl. _fetch_fasta_sequence turned every exception into None, and every caller above it reads None as this protein does not exist, so a connection reset part-way through a burst of per-accession FASTA requests was indistinguishable from an absent entry. All four HTTP call sites now go through a retry_with_backoff-wrapped helper that treats 4xx as permanent (not retried) and 5xx / timeouts / resets as transient. Symptom this fixes: 03_compare_embeddings.ipynb failed with ValueError: 6 target .obs_names have no embedding after UniProt dropped six of sixteen requests; the notebook now runs unchanged.

  • ProteinResolver caches fetched sequences by accession. Accessions were already cached but sequences were not, so a layer sweep re-fetched every FASTA once per layer -- 03_compare_embeddings.ipynb made 112 UniProt requests for 16 distinct proteins, which is what earned the throttling in the first place. Now 16, and the second pass over the same panel is free.

  • ProteinResolver.get_canonical_sequences_batch logs a counted warning naming the identifiers it could not resolve. A short dict was the only signal, and callers iterate what came back rather than what they asked for, so a failed fetch simply vanished and surfaced much later as a re-index error from the io layer. Same counted-drop contract as embpy.io._canon.canonicalize.

  • Static embedding tables no longer claim a pooling they never applied. genept and the other precomputed lookup tables do no forward pass and have no tokens, but _embed_static_hf_to_result recorded the caller's pooling_strategy -- which defaults to "mean" whether or not it was asked for -- so provenance reported pooling: "mean" and _default_key put it in the key: X_emb__gene__genept__pool_mean. Both were false. Now pooling=None, the same convention the cell path uses, and the key is X_emb__gene__genept.

  • Molecule embeddings could not be attached back to an AnnData indexed by a non-canonical SMILES. embed() canonicalises molecule ids on the way in (io/_canon.py, SCHEME["molecule"] == "canonical_smiles"), but build_aliases had no molecule branch, so the form the caller passed was never registered as an alias of the canonical string. to_anndata then re-indexed the target by its own .obs_names and raised ValueError: N target .obs_names have no embedding for molecules that had embedded perfectly well -- Kekule caffeine (CN1C=NC2=C1C(=O)N(C(=O)N2C)C) against its canonical Cn1c(=O)c2c(ncn2C)n(C)c1=O. The input form is now recorded under the input_smiles alias, matching what gene_symbol already does for genes, so callers no longer have to canonicalise with RDKit before calling embed(). Guarded by tests/embpy/io/test_to_anndata_attach.py::{test_non_canonical_smiles_attaches_via_input_alias, test_canonical_smiles_still_attaches_directly, test_build_aliases_records_the_input_smiles}.

  • MoleculeAnnotator.get_mechanism_of_action returned an empty list for any drug whose mechanism ChEMBL registers against a salt form. It filtered mechanism on molecule_chembl_id, but drug-level rows hang off the parent molecule: mechanism?molecule_chembl_id=CHEMBL941 (imatinib) returns zero rows because the mechanism belongs to imatinib mesylate (CHEMBL1642) and names CHEMBL941 only as its parent. Every drug-level query now resolves the molecule hierarchy and filters on parent_molecule_chembl_id, which for imatinib is the difference between 0 and 4 mechanisms, and between 52 and 134 indication rows. The failure was silent -- indistinguishable from "this compound has no curated mechanism".

  • MoleculeAnnotator.get_mechanism_of_action always returned target_name as "". It read target_name off the mechanism payload, which carries only target_chembl_id; the name is now resolved through the target endpoint.

  • MoleculeAnnotator._resolve_to_smiles resolved every name to None, so annotate() reported "Could not resolve identifier to SMILES" and every mol_* structural column came back empty for name-based input. PubChem renamed its SMILES properties without changing the request vocabulary: a request for CanonicalSMILES now answers under the key ConnectivitySMILES, and one for IsomericSMILES under SMILES, so reading back the name that was requested always missed.

  • DrugResolver._extract_smiles did not recognise PubChem's current SMILES key, so DrugResolver's direct name lookup always failed and fell through to the slower CID path. name_to_smiles_resolved("aspirin") reported source="pubchem_cid" where it should report pubchem_name. Both the old and the current property names are now accepted.

  • BioEmbedder.embed(adata, obs_column=..., output="anndata") (the README gene-perturbation Quick Start) now aligns rows on the target.obs[obs_column] labels instead of obs_names, so barcoded cells each receive their perturbation's vector rather than raising N target .obs_names have no embedding (or silently NaN-filling every row with missing="nan"). var_column does the same on the var axis. Unknown labels raise listing the labels (missing="error") or NaN-fill only their rows (missing="nan"); obs_names/var_names and the label column are left untouched. The column is threaded through route_output / to_anndata_many / to_anndata as obs_column / var_column keywords (default None, unchanged behaviour). Guarded by tests/embpy/io/test_bioembedder_embed_standardized.py::test_embed_gene_perturbations_*obs_column* and test_embed_genes_align_on_var_column_labels.

  • molformer_base failed to load on transformers 4.x (ImportError: cannot import name 'create_bidirectional_mask') because ibm/MoLFormer-XL-both-10pct updated its trust_remote_code files for transformers 5. Every trust_remote_code load (GENA-LM, Nucleotide Transformer v1/v2, NTv3, HyenaDNA, Caduceus, MolFormer) now passes revision= from the new embpy.models.hf_revisions.REMOTE_CODE_REVISIONS mapping ({hf_repo_id: commit_sha}, matched case-insensitively). Every repo except MolFormer is pinned to its main head as of 2026-10-11, so its behaviour is frozen at that date. MolFormer is pinned to the last transformers-4-compatible commit, and on transformers 4.x only: that old commit imports transformers.onnx, which transformers 5 removed, so on transformers >= 5 (e.g. the ntv3 extra) molformer_base loads from main as before. Pass revision= to a wrapper to override the pin (BioEmbedder does not forward kwargs, so this only works for directly constructed wrappers). The gated NTv3 repos are left unpinned. Guarded by tests/embpy/models/test_hf_revisions.py.

  • Gene sequences no longer vanish when Ensembl has a bad minute. _ensembl_get retried only HTTP 429, so a single 500 Server Error from rest.ensembl.org made get_dna_sequence return None and the whole call fail with no embeddings were produced (all inputs failed) -- which is how 01_embed_any_model.ipynb failed twice on the HPC. 5xx responses, connection errors and timeouts are now retried with exponential backoff through the existing retry_with_backoff helper (still honouring Retry-After); other 4xx are returned immediately. retry_with_backoff itself now honours a retry_after attribute on the raised exception. Guarded by tests/embpy/test_gene_resolver_retry.py and tests/embpy/test_retry.py; the exon-truncation tests now model a timeout that persists across retries.

  • Llama text models (llama3.1_8b, llama3.2_3b, llama3.2_1b) could load but never embed through BioEmbedder.embed. LlamaEmbeddingWrapper overrode embed() but not embed_batch(), so the inherited TextLLMWrapper.embed_batch read outputs.last_hidden_state, which AutoModelForCausalLM outputs do not have (AttributeError: 'CausalLMOutputWithPast' object has no attribute 'last_hidden_state'). Both now share one path that pools hidden_states, with last_token pooling as the default for both. Loading no longer passes device_map, which on transformers 5 required accelerate (in no extra); the model is moved with .to(device) on every device. Guarded by tests/embpy/models/test_api_models.py::TestLlamaEmbeddingWrapper*.

  • esmc_* models without the esm SDK raised NameError: name 'ESMC' is not defined (reported as a generic ModelLoadError) instead of the typed DependencyError naming the install. The import guard now binds ESMC = None and ESMCWrapper.load raises ImportError, which BioEmbedder turns into DependencyError. Guarded by tests/embpy/models/test_protein_models.py and tests/embpy/test_embedder.py.

Removed

  • pp.load_scperturb, pp.list_scperturb_datasets, pp.scperturb_info and ScPerturbDatasetCard. Downloading third-party perturbation datasets is not embpy's job -- it is a curated mirror of someone else's Zenodo record, which drifts independently of this package and has to be tracked by hand. It had also never worked: every filename in the registry was fabricated, so every call 404'd. Rather than maintain a dataset index alongside an embedding library, load the h5ad yourself and pass it to embed/embed_cells -- nothing else in embpy depended on this module.

Deprecated

  • ActionEmbeddingProvider.embed(symbols) -- replaced by embed_with_status(symbols). Scheduled for removal in 0.1.0.