Hardware-Aware Inference Optimizer

An evidence-first system that searches model optimization and serving choices for a specific evaluation set and hardware target. It uses Python, PyTorch, CUDA, ONNX, TensorRT, and Ray to perform real work—not just appear in configuration.

The customer problem

An accurate model is not automatically a good production model. The best runtime, precision, compilation mode, pruning level, and batch size change with the target device and latency budget. Hand-tuning those choices is slow, results are easy to compare incorrectly, and a single “fastest” number hides quality, throughput, and memory tradeoffs.

This project turns that work into a reproducible search:

1. inspect the target device and available runtimes;
2. generate valid optimization configurations;
3. execute them against labeled evaluation data;
4. retain the Pareto frontier across quality, latency, throughput, and memory;
5. select a configuration under explicit quality and p95-latency constraints;
6. export a local FastAPI inference bundle plus its evidence manifest.

No hosted service is deployed by this repository.

Validated result

Across three real pretrained vision models and two physical hardware targets, the selected configurations produced:

Metric	Measured result
Median batch-one latency reduction	79.616%
Median deployable artifact-size reduction	20.987%
Maximum selected task-quality loss	0.0 percentage points
Coverage	3 models × 2 physical targets

The baseline and candidate are both batch one for the latency comparison. Batching is reported separately as throughput under a 100 ms batch-p95 constraint. The system measured process RSS and peak GPU memory, but these runs do not establish a defensible memory reduction; no memory-reduction claim is made.

The complete results, raw samples, confidence intervals, environment snapshots, checksums, and excluded development run are in benchmarks/.

Architecture

flowchart LR
    A["Model + labeled evaluation set + target device"] --> B["Capability discovery"]
    B --> C["Search space<br/>quantization · pruning · compilation<br/>batching · runtime"]
    C --> D["Ray trial execution"]
    D --> E["PyTorch / CUDA"]
    D --> F["ONNX Runtime<br/>CPU or CUDA"]
    D --> G["TensorRT"]
    E --> H["Evidence recorder"]
    F --> H
    G --> H
    H --> I["Pareto frontier<br/>quality ↑ · throughput ↑<br/>latency ↓ · memory ↓"]
    I --> J["Constraint-aware selection"]
    J --> K["Local FastAPI bundle<br/>manifest + model artifact"]

Loading

What each technology does

Technology	Role
Python	CLI, configuration schema, measurement, evidence aggregation, service export
PyTorch	pretrained models, eager/compiled inference, pruning, TorchAO quantization
CUDA	synchronized accelerator execution and memory sampling on NVIDIA hardware
ONNX	portable graph export and artifact validation
ONNX Runtime	CPU and verified CUDAExecutionProvider execution
TensorRT	strongly typed FP16 engine construction and inference on the Tesla T4
Ray	isolated parallel trial scheduling with retained task evidence

Optimization search

The search space combines:

• runtime: PyTorch, ONNX Runtime, TensorRT;
• quantization: FP32/none, dynamic INT8 where valid, FP16 TensorRT;
• pruning: zero and configured unstructured sparsity levels;
• compilation: eager and supported compiler modes;
• batch size: batch one for latency, larger batches for constrained throughput;
• device/runtime controls: CPU thread count or CUDA device and provider.

Unsupported combinations are recorded rather than silently discarded. Selection first removes failed, unsupported, and non-deployable candidates. It then computes the non-dominated set: a candidate is Pareto-optimal when no other candidate is at least as good on every objective and strictly better on one. A final recommendation is chosen only from candidates that satisfy the configured quality-loss and batch-p95 constraints.

Benchmark matrix

Target	Models	Data	Runtime paths
Apple Silicon CPU, 4 threads	ResNet-18, MobileNetV3-Small, EfficientNet-B0	deterministic 256-image Imagenette/Imagewoof validation samples	PyTorch, ONNX Runtime CPU
NVIDIA Tesla T4	same three official torchvision models	same selected public samples	PyTorch CUDA, ONNX Runtime CUDA, TensorRT 11

Each configuration used 10 warmups and 50 measured iterations. Real task top-1 accuracy and macro-F1 were evaluated against public labels. Detailed CPU and T4 tables—including throughput and p95 latency—are in REAL_BENCHMARK_REPORT.md.

Clean install

Prerequisites: Python 3.11–3.14 and uv.

git clone https://github.com/johnquevedo/hardware-aware-inference-optimizer.git
cd hardware-aware-inference-optimizer
uv sync --frozen --extra dev
uv run ruff check src tests scripts
uv run pytest
uv run hwopt --help

For the real public-data benchmark:

uv run hwopt prepare-data --data-dir data/public
uv run hwopt real-benchmark --help

The preparation command downloads public archives locally. Datasets, pretrained weights, exported ONNX graphs, and TensorRT engines are intentionally not stored in Git.

GPU validation requires an NVIDIA/CUDA environment with compatible TensorRT and ONNX Runtime GPU packages. The two-hour, T4-specific Modal harness used for the evidence run is retained in scripts/modal_t4_validation.py; it creates no persistent hosted service.

Reproducibility and evidence

• benchmarks/final-evidence.json: combined, apples-to-apples result and raw-sample SHA-256 digests.
• benchmarks/real-cpu/: CPU environments, trials, Pareto sets, recommendations, and service manifests.
• benchmarks/modal-t4-20260725T2200Z/: corrected CUDA/ONNX Runtime CUDA/TensorRT evidence.
• benchmarks/modal-t4-20260725T2100Z/: retained, ineligible development run with failures and fallback evidence.
• benchmarks/gpu-cost-and-termination.json: $0.84501807 metered list-price cost and termination proof.
• EVIDENCE_LEDGER.md and RESUME_SUPPORT.md: claim-level support and limitations.

CPU CI runs linting and the test suite on a clean Ubuntu runner. Benchmark artifacts are deliberately not regenerated in CI because public datasets and GPU execution are larger, separately controlled validation jobs.

Honest limitations

• The evaluation uses deterministic 256-image subsets, not complete validation sets.
• Only one CPU machine and one NVIDIA GPU model were physically tested.
• Accuracy and macro-F1 are real task metrics, but the coverage is image classification only.
• The first T4 development pass is retained but excluded: ONNX Runtime fell back to CPU, legacy TensorRT FP16 flags failed, and NVML process matching was incorrect.
• Pruning is searched and measured, but sparsity does not guarantee lower latency on every runtime.
• Process RSS, artifact size, and GPU memory are separate metrics. The evidence does not support a memory-reduction claim.
• Exported service bundles were loaded and exercised locally; there are no external users, no production deployment, and no hosted endpoint.

Project map

src/hwopt/          optimizer, runtimes, metrics, Pareto selection, export
tests/              CPU-safe unit and integration tests
scripts/            bounded GPU validation and infrastructure safeguards
benchmarks/         versioned evidence (generated binaries excluded)
docs/assets/        benchmark and evidence screenshots
docker/             CUDA/TensorRT build context

See RECRUITING_SPEC.md for the original acceptance criteria and LICENSE for the MIT license.