Parallel memory hardening#8
Open
reece-maticebio wants to merge 24 commits into
Open
Conversation
…python modules for python scripting
The existing canonical test only locks consensus output, which depends solely on the largest observed cluster per score set. A refactor that perturbs per-score statistics (beta, p-values, observed clusters) could pass that test silently. The new test snapshots beta, p_value, observed_cut_size, and the full cluster contents per score set, run at n_jobs=1 and n_jobs=2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First load-bearing piece of the refactor. The Store protocol is the contract fan-out pipeline stages will use to write N artifacts (permuted scores, hierarchies, per-permutation stats) without committing to a memory-or-disk backend. MemoryStore lands the in-memory implementation; DiskStore + Codec implementations come in a later wave once stages are converted. Nothing in the existing pipeline consumes Store yet, so this commit is purely additive: 13 new tests for MemoryStore, all 16 prior tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the hardcoded list of .py files (and the parallel install_sources block for the storage subpackage) with a small helper script that meson calls at configure time. New files are picked up automatically; new subpackages (any directory with __init__.py) are discovered too. Caveat documented inline: meson only runs the discovery at configure time, so adding a new file still requires `pip install --no-build-isolation --no-deps -e .` to reconfigure. Editing existing files needs nothing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six typed config objects group the flat parameters currently accepted by run_pipeline into the conceptual blocks they belong to: SimilarityConfig, ScoringConfig, PermutationConfig, HierarchyConfig, ConsensusConfig, RuntimeConfig. Additive: nothing in the pipeline consumes these yet. Defaults are pinned in tests to match run_pipeline so they stay in lockstep until the flat kwargs are removed in a later wave. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
W1.7: ten 'raise Warning(...)' calls in hhio.py were latent bugs — Warning subclasses Exception, so they aborted the load instead of skipping the bad line as the 'omitted' message suggested. Converted to warnings.warn(...) so the documented behavior actually happens. W1.8: replaced six print() calls in run_pipeline with logger.info(); when verbose=True, _ensure_visible_logging() idempotently attaches a stream handler to the package logger so users still see progress. External logging configuration is now respected. Also: hierarchical_clustering.py now logs which backend (Fortran vs pure Python) is active at import. Without this, refactor work that breaks the Fortran import path would silently fall back to the slow implementation and only show up as a perf regression, not a correctness one. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 'hhio' name conflated 'hh' (HotNet, package-internal) with 'io' (which shadows stdlib io). file_io.py is the new home; all 9 internal callers now import from there. hhio.py remains as a thin shim that re-exports the same names and emits a DeprecationWarning, so external code that did `from hierarchical_hotnet.hhio import load_edge_list` keeps working through one release. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hierarchical_clustering_tests.py was a script-style benchmark living inside the installed package. Useful pieces — a naive O(n^3 log n) reference implementation of hierarchical decomposition and the canonical Tarjan-1983 example — have been ported into tests/test_clustering_against_naive.py as proper pytest tests. The timing/perf scaffolding was dropped (not a correctness check, didn't belong in unit tests). The new tests add 14 cases that exercise tarjan_HD against an independent oracle: 2 directions on the Tarjan-1983 example, plus a 12-case small fuzz on random strongly-connected graphs (n up to 13). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every compute module previously carried its own argparse machinery (get_parser/run/main + __main__ block). That coupling meant importing the library — say compute_similarity_matrix — pulled in argparse setup and file IO helpers that only the CLI wrapper needs. Done via tools/extract_cli.py (AST-aware): for each of the 7 compute modules with a library/CLI split, the CLI block + its required imports moved into hierarchical_hotnet/cli/<name>.py, and the original module had argparse plus any CLI-only file_io imports trimmed. generate_example_graph (pure CLI, no separate library API) moved wholesale via git mv. pyproject.toml console_scripts now point at the new cli locations. All 51 tests still pass; CLI smoke-tested by invoking hhnet-construct-similarity- matrix against the example data. Net result: -593 / +64 lines across the 8 affected modules; the compute files are now ~50-70% smaller and contain no CLI concerns. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DiskStore[T] pairs the Store protocol from W1.3 with a Codec[T] to give a
filesystem-backed store. Two codecs land:
- ScoreMapCodec — dict[str, float] as a gene\tscore TSV, matching what
hhnet-permute-scores writes. Reads with score_threshold=-inf so the
round-trip is lossless (the CLI's default 0.0 threshold would silently
drop negatives).
- HierarchyCodec — (T, index_to_gene) split into <key>.edges.tsv +
<key>.genes.tsv, matching hhnet-construct-hierarchy output. Existence
keys off the edges file.
Pipeline integration follows in W3.2/W3.3/W3.5; nothing wires DiskStore in
yet. Store protocol also gains __getitem__ so `dict(store)` works without
duck-typing fights.
15 new tests; suite at 63 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…3.2+W3.3) Both batch functions gain an optional out: Store | None = None parameter. When None (default), the function returns a list of N results — historical behavior, no change for existing callers. When a Store is provided, each result streams into it as it arrives (keyed by str(seed) for permutations, str(index) for hierarchies) and the populated store is returned. Peak memory in the Store path is one result plus whatever the pool's queue holds, instead of all N. The streaming uses zip(seeds, map_fn(...)) so executor.map's lazy iterator is drained rather than materialized with list(). 8 new tests cover the Store path at n_jobs=1 and n_jobs=2, and verify the Store result equals the list result for the same seeds/inputs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…4+W3.5) run_pipeline now accepts workdir: Path and reuse: bool. When workdir is set, every artifact spills to a fixed layout on disk: <workdir>/similarity_matrix.h5 <workdir>/beta.txt <workdir>/bins/<label>.tsv <workdir>/permuted_scores/<label>/<seed>.tsv <workdir>/hierarchies/<label>/<i>.edges.tsv (+ .genes.tsv) reuse=True checks each artifact path individually: existing files are read back, missing ones are computed (so interrupted runs resume naturally). reuse=False (default) always recomputes but still writes everything out. process_hierarchies' stats_input is now a generator instead of a materialized list. Combined with itertools.tee in the pipeline, the permuted hierarchies are pulled from the DiskStore one at a time during the stats computation rather than all N+1 being held in memory at once. The historical two-list API for process_hierarchies still works (lists have __len__, so the validation check still runs in that path). construct_hierarchies gains a keys= parameter so the pipeline can compute only the subset of hierarchies that aren't already on disk while still addressing them by their original positions (0..N) in the store. Six new tests cover the layout, the workdir-vs-memory parity, reuse skipping recomputation, reuse filling in a partially-missing workdir, and reuse=False overwriting. Total suite: 77 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compute primitives move into hierarchical_hotnet.core with noun-based names; the private _parallel.py becomes the public hierarchical_hotnet.parallel subpackage so anyone writing a new fan-out stage can import its pool helper. construct_similarity_matrix.py -> core/similarity.py find_permutation_bins.py -> core/bins.py construct_hierarchy.py -> core/hierarchy.py permute_scores.py -> core/permute_scores.py permute_network.py -> core/permute_network.py hierarchical_clustering.py -> core/clustering.py process_hierarchies.py -> core/process_hierarchies.py perform_consensus.py -> core/consensus.py common.py -> core/common.py _parallel.py -> parallel/__init__.py Top-level keeps __init__.py, config.py, file_io.py, hhio.py (the existing hhio deprecation shim), pipeline.py, and the Fortran extension. CLI modules stay at hierarchical_hotnet.cli.<command> — pyproject.toml console scripts unchanged. This is a clean break, no new compat shims: anyone who imported ``hierarchical_hotnet.construct_similarity_matrix`` directly (rather than the recommended ``from hierarchical_hotnet import compute_similarity_matrix``) must update. The hhio shim from the previous wave is unaffected. 77 tests still pass; CLI smoke test against example data still works. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six operations previously had scattered 'if imported_fortran_module:'
checks across core/clustering.py and core/process_hierarchies.py:
condense_adjacency_matrix, find_distinct_weights, slice_array,
strongly_connected_components_labels, threshold_edges, summarize_sizes.
They now live behind a single dispatcher at hierarchical_hotnet.backends.
The selection happens once at import time and is exposed as backends.BACKEND.
HHNET_BACKEND env var lets callers force a path:
unset / auto -> try Fortran, warn-and-fall-back to Python
fortran -> require the extension (raises on missing -- CI guard)
python -> force the pure-Python path (testing the fallback)
core/clustering.py shrinks by ~120 lines (the pure-Python SCC + the if/else
ladders all moved out). core/process_hierarchies.py's summarize_cluster_sizes
shrinks from 27 to 7 lines.
Three new tests:
- dev env actually uses Fortran (silent-fallback guard)
- HHNET_BACKEND=bogus fails loudly at import
- end-to-end pipeline parity: HHNET_BACKEND=python (subprocess) matches
the in-process Fortran result on the example data
Suite at 80 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the 8 separate hhnet-* console scripts with one ``hhnet`` entry
and 8 subcommands. Each cli/<name>.py still owns its own argparse parser;
cli/main.py is a thin dispatcher that forwards argv to the right
subcommand's main(). The subcommand's --help reports a sensible program
name ("hhnet construct-similarity-matrix" rather than the previous
"hhnet-construct-similarity-matrix").
pyproject.toml [project.scripts]: 8 entries -> 1
examples/example_commands.sh, example_commands_parallel.sh: updated
README.md: updated invocations + sentences about the CLI surface
Clean break (no shim console scripts for the old names). End-to-end
verified by running the full example_commands.sh; consensus output
matches the committed canonical fixtures. 80 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…n scope
The refactor added three user-facing features that were not in the README:
- run_pipeline(workdir=..., reuse=...) for disk-backed runs and resuming
interrupted runs. Documented with the fixed on-disk layout and a note
that the formats match the standalone hhnet-* artifacts.
- HHNET_BACKEND env var to force fortran/python dispatch (and the
backends.BACKEND constant for inspection). Documented as a table under
Setup.
- Explicit clarification that run_pipeline does NOT perform network
permutations -- only score permutations, matching the canonical
Hierarchical HotNet methodology. Network permutation is exposed but
must be invoked separately.
No code changes; 80 tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…commands.sh
A runnable script that drives the example toy dataset through hhn.run_pipeline()
the way example_commands.sh drives it through the CLI subcommands. Same
inputs, same parameters, same canonical consensus output -- two
demonstrations side-by-side:
1. In-memory run (workdir=None) -- shows the typical Python API usage
2. Disk-backed run (workdir=<tmp>) -- shows the spill-to-disk layout
and confirms parity with the in-memory result
Prints per-score stats (p_value, cut height, cluster sizes, ratios),
verifies the consensus byte-matches examples/example_consensus_nodes.tsv
and example_consensus_edges.tsv, and reports the active backend.
Useful both as a learning resource (copy-and-modify starting point for
real analyses) and as a smoke test after install.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the redundant in-memory-vs-disk-backed parity check (that property is already locked in by tests/test_pipeline_workdir.py::test_workdir_result_matches_in_memory). The script now does one thing: runs the pipeline once with workdir=..., prints per-score stats, verifies the consensus against the canonical fixtures, and lists the artifacts that landed on disk. Adds --workdir (default: examples/pipeline_output/) and --reuse flags for resuming interrupted runs. examples/pipeline_output/ is gitignored so the 600-ish artifact files never get committed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
run_pipeline now writes the final per-score outputs under
<workdir>/results/ whenever workdir is set, in the exact same file
formats hhnet process-hierarchies and hhnet perform-consensus produce:
<label>_clusters.tsv header (cut height, p-value, observed/
expected cluster sizes and ratios) + one
cluster per line, sorted by size desc
<label>_observed_sizes.tsv per-height largest-cluster sizes for the
observed hierarchy
<label>_permuted_sizes.tsv per-height min/expected/max aggregated
across permutations
<label>_sizes.pdf observed vs expected size plot
(only when plot=True)
consensus_nodes.tsv final consensus subnetwork nodes
consensus_edges.tsv final consensus subnetwork edges
The TSVs are byte-for-byte identical to what the CLI commands write
against the example data -- verified by diff against the committed
examples/results/clusters_network_1_scores_*.tsv files.
run_pipeline gains a plot: bool = False kwarg. examples/example_pipeline.py
exposes --plot. The matplotlib dep is imported lazily inside the plot
branch, so users without the [plot] extra installed are unaffected
unless they opt in.
save_pipeline_result is also exposed as a standalone helper for callers
who want to save a PipelineResult from an in-memory run to disk after
the fact.
80 tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously, compute_similarity_matrix silently consumed whatever weight column was in the input edge list. With a 2-column file (the example), load_edge_list defaults weights to 1.0 so the diffusion was effectively unweighted; with a 3-column file (e.g. STRING confidences), the input weights were folded directly into the adjacency, producing a different similarity matrix that diverges from the canonical Hierarchical HotNet methodology (which assumes an unweighted topology). Now the default is to ignore the third component of each edge tuple and treat the network as unweighted. Set use_edge_weights=True (or pass -uew/--use_edge_weights to hhnet construct-similarity-matrix, or use_edge_weights=True to run_pipeline) to opt in to the old behavior. Behavior change for users with weighted 3-column inputs: - Pre-change: weights were used silently in the diffusion. - Post-change: weights are ignored unless opted in. The example pipeline uses a 2-column edge list so the canonical golden snapshot (beta=0.050716, p-values, clusters, consensus) is unchanged. 85 tests pass (80 prior + 5 new): unit-weight inputs match between the flag states; weighted inputs diverge when opted in; the caller's edge list is not mutated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A node that appears in no edge produces an all-zero row/column in the
similarity matrix, which the strongly-connected-component code cannot
handle -- previously this surfaced as a cryptic IndexError deep inside
combined_similarity_matrix ("similarity matrix and index-gene
associations have different dimensions").
New drop_isolated_nodes(edges, index_to_gene) detects any index in
index_to_gene that is absent from every edge, removes it, and -- since
the package requires consecutive 1..N indices -- renumbers the survivors
to stay gap-free. It emits a UserWarning naming how many nodes (and which
genes) were dropped. Clean input is returned unchanged and silent.
run_pipeline now calls this automatically right after materializing the
edge list, so isolated nodes can no longer reach the algorithm. The
function is also exported from the top-level package for callers who want
to clean inputs before calling the individual step functions.
8 new tests (unit: trailing/middle/multiple isolated nodes, weight and
tuple-arity preservation, no-op on clean input; integration: an isolated
gene appended to the example inputs is dropped and the run still
completes). Suite at 93 passing; the canonical golden snapshot is
unchanged since the example network has no isolated nodes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The hierarchy fan-out passed the dense NxN similarity matrix through ProcessPoolExecutor initargs, which pickles it once per worker -- W private copies of identical, read-only data. On real PPI networks that is the dominant memory cost and the main OOM trigger. Three changes in hierarchical_hotnet.parallel: * Shared-memory matrix (parallel/sharedmem.py). shared_ndarray() copies the matrix once into a POSIX shared-memory segment; workers attach a read-only view with attach_shared_ndarray(). One physical copy regardless of worker count. The parent owns the segment (creates and unlinks it) and the pool nests inside, so every worker has detached before the unlink. Measured at 8 workers: removes (W-1)x the matrix from resident memory, scaling to ~18 GB saved on an 18k-node network. * /dev/shm preflight guard. POSIX shared memory is /dev/shm-backed on Linux; tmpfs allocates lazily, so an oversized segment is created without error and then dies with an opaque SIGBUS when written. _dev_shm_guard() checks free space first and raises an environment-specific message (Docker --shm-size, k8s emptyDir, or a host remount). * Descriptive worker-failure errors (parallel/oom.py). An OOM-killed worker surfaced as a bare BrokenProcessPool. maybe_pool now rewrites that, and a propagated MemoryError, into a message that names n_jobs and the fixes (fewer workers, or workdir= for disk-backed storage); on cgroup v2 hosts it reads memory.events to confirm the OOM. Tests: test_sharedmem.py (round-trip, read-only enforcement, cleanup, /dev/shm guard) and test_pool_errors.py (failure translation). Full suite 113 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.