Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 7 additions & 9 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@ before writing any code.
- **First runtime:** Ollama (shipped). Future: LM Studio, llama.cpp, Jan AI,
GPT4All, vLLM (added as adapters, no core changes)
- **Design style:** Clean Architecture + SOLID + Dependency Inversion
- **Status:** Greenfield. No source code yet — only docs (`PROJECT.MD`,
`Architecture.md`, `README.md`, `LICENSE`).
- **Status:** Active development. Dynamic catalog from ollama.com.

---

Expand Down Expand Up @@ -55,9 +54,9 @@ before writing any code.
(see `common/errors.py`). Every error must carry actionable context.
8. **Cross-platform by default.** Use `platformdirs` for paths, avoid
hardcoded `/` or `C:\`, avoid OS-specific syscalls without guards.
9. **NEVER generate GitHub content with a coding agent.** Issues, PRs,
commits, and release notes must be authored by a human, not composed by a
coding agent.
9. **GitHub content may be generated by coding agents.** Issues, PRs,
commits, and release notes may be composed by a coding agent, but must
follow the repo's writing rules and templates.

---

Expand Down Expand Up @@ -122,9 +121,9 @@ before writing any code.
7. **NEVER let CLI-framework objects cross into `common/`/`core`/`domain`.**
The CLI must resolve and validate every option to a plain value before any
downstream call; `configure_logging()` falls back to `INFO` on bad input.
8. **NEVER generate GitHub content with a coding agent.** Issues, PRs,
commits, and release notes must be authored by a human, not composed by a
coding agent.
8. **GitHub content may be generated by coding agents.** Issues, PRs,
commits, and release notes may be composed by a coding agent, but must
follow the repo's writing rules and templates.

---

Expand All @@ -137,7 +136,6 @@ Domain: modeldock/domain/ (pure entities, no I/O)
Ports: modeldock/ports/ (typing.Protocol interfaces)
Adapters: modeldock/adapters/ (runtimes, registry, downloaders, cache, progress)
Common: modeldock/common/ (config, logging, platform, http, errors)
Data: modeldock/data/catalog.json (bundled model registry)
```

See `Architecture.md` §2 for the full tree and §3 for per-module responsibilities.
Expand Down
44 changes: 30 additions & 14 deletions Architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ installation verification, and loading through pluggable runtime adapters.
│ Adapter / Infrastructure Layer (modeldock/adapters) │
│ - runtimes/ollama.py (first), lmstudio, llamacpp, jan, │
│ gpt4all, vllm (future) │
│ - registry/ (bundled catalog + remote registry)
│ - registry/ (dynamic Ollama catalog + bundled fallback)
│ - downloaders/ (http, ollama-native pull) │
│ - cache/ (filesystem cache) │
│ - progress/ (rich, tqdm, silent) │
Expand Down Expand Up @@ -112,8 +112,9 @@ modeldock/
│ │ │ ├── gpt4all.py
│ │ │ └── vllm.py
│ │ ├── registry/
│ │ │ ├── bundled.py # static catalog shipped in package
│ │ │ └── remote.py # optional remote registry fetch
│ │ │ ├── ollama_library.py # dynamic catalog from ollama.com (default)
│ │ │ ├── bundled.py # static fallback catalog (offline)
│ │ │ └── remote.py # optional remote registry fetch
│ │ ├── downloaders/
│ │ │ ├── http.py # generic HTTP downloader
│ │ │ └── ollama_pull.py # delegates to `ollama pull`/API
Expand Down Expand Up @@ -165,13 +166,13 @@ standard.
| `ports/` | `typing.Protocol` interfaces defining *what* the system needs from the outside world (runtime, registry, downloader, cache, progress, events). No implementation. |
| `core/` | Application services that implement use cases by composing ports. `LifecycleOrchestrator` is the brain behind `load()`. `ModelManager` is the high-level facade. |
| `adapters/runtimes/` | Concrete runtime integrations. Each implements `RuntimePort`. Ollama ships first; others are stubs implementing the interface. |
| `adapters/registry/` | Provides the searchable model catalog. `bundled.py` reads `catalog.json`; `remote.py` can refresh from a URL. |
| `adapters/registry/` | Provides the searchable model catalog. `ollama_library.py` scrapes ollama.com for the full model list with auto-detected categories/capabilities; `bundled.py` is a static fallback; `remote.py` can refresh from a URL. |
| `adapters/downloaders/` | Moves bytes. `ollama_pull.py` wraps Ollama's native pull; `http.py` is a generic fallback for runtimes that expose direct URLs. |
| `adapters/cache/` | Tracks what is installed/downloaded so we never re-fetch. Filesystem manifest + content hashing. |
| `adapters/progress/` | Pluggable progress reporters (rich, tqdm, silent) implementing `ProgressPort`. |
| `cli/` | Thin delivery layer. Translates argv → core service calls. No business logic. |
| `common/` | Config loading, logging bootstrap, platform/OS helpers, shared HTTP client, base errors. |
| `data/catalog.json` | The bundled, versioned model registry (names, aliases, categories, capabilities, sizes, default tags). |
| `data/catalog.json` | **Deprecated.** Static bundled model registry. Replaced by `OllamaLibraryRegistry` which scrapes ollama.com dynamically. Kept as offline fallback only. |

---

Expand Down Expand Up @@ -293,8 +294,8 @@ modeldock --help
explicit runtime overrides.
- **Format:** TOML for the file (human-friendly, stdlib `tomllib` in 3.11+).
- **Model:** a frozen `Settings` dataclass/pydantic model: `default_backend`,
`cache_dir`, `registry_url`, `log_level`, `progress_style`, `auto_install`,
`ollama_host`, etc.
`cache_dir`, `registry_url`, `catalog_source`, `log_level`, `progress_style`,
`auto_install`, `ollama_host`, etc.
- **Cross-platform paths:** resolved via `common/platform.py` using
`platformdirs` (the de-facto standard for user/config/cache dirs across OSes).
- **Validation:** config loaded through a validator; unknown keys warn, invalid
Expand Down Expand Up @@ -330,21 +331,35 @@ Goal: "never re-download installed models" + offline cache management.

A **searchable, versioned catalog** decoupled from any runtime.

- **Bundled catalog** (`data/catalog.json`): curated entries with
- **Dynamic catalog** (`adapters/registry/ollama_library.py`): scrapes
`ollama.com/library` for the full model list. Auto-detects `Category` and
`Capability` from model names and HTML tags. Caches results locally
(`<cache_dir>/catalog_cache.json`) with 24-hour TTL for offline use.
This is the **default** registry.
- **Bundled fallback** (`data/catalog.json`): static curated entries with
`name, aliases[], category, capabilities[], default_tag,
variants[{tag, params, size_bytes, min_ram}], description, backend_hints`.
Shipped with the package → works offline, zero-config.
Used as offline fallback when `catalog_source="bundled"` or when the
dynamic catalog fails and no cache exists.
- **`RegistryPort`** abstraction: `search(query)`, `get(ref)`,
`by_category(cat)`, `recommend(task)`, `list_all()`.
- **Implementations:**
- `BundledRegistry` — reads `catalog.json` (default).
- `OllamaLibraryRegistry` — scrapes ollama.com, auto-detects metadata,
caches locally (default).
- `BundledRegistry` — reads `catalog.json` (offline fallback).
- `RemoteRegistry` — optional fetch/refresh from a URL/JSON for community
updates without a package release.
- **Configuration:** `catalog_source` setting in `Settings` controls which
registry is used: `"auto"` (try dynamic, fallback to bundled), `"ollama"`
(dynamic only), `"bundled"` (static only).
- **Alias resolution:** `domain/alias.py` maps friendly names (`llama3`) →
canonical `ModelSpec`. Handles version shortcuts (`llama3:8b`),
capability-based lookup (`recommend("vision")`), and "auto model selection."
- **Extensibility:** new models added by editing `catalog.json` (or a remote
registry) — no code change. Future: community-contributed registry plugins.
- **Auto-detection rules:**
- **Category:** name contains `embed` → EMBEDDING; `code` → CODING;
`vision`/`llava` → VISION; `r1`/`thinking` → REASONING; default → CHAT.
- **Capabilities:** HTML pills (`tools`, `vision`, `thinking`, `audio`,
`embedding`) map to `Capability` enum values. Default: CHAT + COMPLETION.

---

Expand Down Expand Up @@ -504,7 +519,7 @@ dependency-light. Runtime-native SDKs (e.g., `ollama` Python client) are
|---|---|---|
| Clean Architecture + ports | Maximum extensibility for 6 future runtimes; testable without Ollama | More files/abstraction upfront; slight "ceremony" for a small v1 |
| Return runtime-native client from `load()` | Stay lightweight; don't reimplement inference | Less uniform cross-runtime inference API (acceptable — management is the product) |
| Bundled `catalog.json` | Works offline, zero-config, beginner-friendly | Catalog can go stale; mitigated by optional `RemoteRegistry` refresh |
| Dynamic catalog via HTML scraping | Always up-to-date with ollama.com; zero manual maintenance | Depends on ollama.com HTML structure; mitigated by 24h cache + bundled fallback |
| Entry-point plugins | Third parties extend without forking | Slightly more discovery code; stdlib `importlib.metadata` is cheap |
| `httpx` over `requests` | Streaming/resumable downloads, async-ready | Extra dep (small, well-maintained) |
| Optional runtime SDK extras | Tiny base install | User may need `pip install modeldock[ollama]` (documented) |
Expand All @@ -516,7 +531,8 @@ dependency-light. Runtime-native SDKs (e.g., `ollama` Python client) are
## 17. Future Scalability Considerations

- **New runtimes:** entry-point adapters — zero core changes (§14).
- **New models:** catalog edits / remote registry — no code change.
- **New models:** dynamic catalog scrapes ollama.com automatically; no manual
catalog edits needed. Models appear as soon as they're added to ollama.com.
- **Concurrency:** `install_category` parallel downloads; service layer already
async-ready (`httpx` async). Could add `asyncio` variants of the API
(`md.aload(...)`) later without breaking sync API.
Expand Down
38 changes: 38 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,43 @@ All notable changes to ModelDock will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).

## [0.1.3] - 2026-07-19

Dynamic catalog: replaced static `catalog.json` with live scraping of ollama.com.

### Added

- `OllamaLibraryRegistry` adapter — scrapes `ollama.com/library` for the full model list, auto-detects categories and capabilities, and caches locally for offline use
- `catalog_source` config setting (`"auto"` | `"ollama"` | `"bundled"`) to control which registry is used
- `MODELDOCK_CATALOG_SOURCE` environment variable support
- Local catalog cache (`<cache_dir>/catalog_cache.json`) with 24-hour TTL

### Changed

- `ModelManager` now defaults to `OllamaLibraryRegistry` (dynamic) instead of `BundledRegistry` (static)
- Auto-detection rules: model name patterns and HTML capability tags determine `Category` and `Capability`
- `Architecture.md` updated to reflect dynamic catalog design

### Removed

- Deleted `src/modeldock/data/catalog.json` — no longer needed
- Removed `[tool.setuptools.package-data]` from `pyproject.toml`

### Deprecated

- `BundledRegistry` is now a fallback only, used when `catalog_source="bundled"` or when the dynamic catalog fails and no cache exists

### Tests

- 32 new tests for `OllamaLibraryRegistry` (HTML scraping, auto-detection, cache, network fallback)
- BundledRegistry tests skipped when `catalog.json` is not present

### Contributors

- @himanshu231204

---

## [0.1.2] - 2026-07-19

Patch fix for `catalog.json` not being included in the installed package.
Expand Down Expand Up @@ -81,6 +118,7 @@ This project follows [Semantic Versioning](https://semver.org/):

## Links

[0.1.3]: https://github.com/OpenAgentHQ/modeldock/compare/v0.1.2...v0.1.3
[0.1.2]: https://github.com/OpenAgentHQ/modeldock/compare/v0.1.1...v0.1.2
[0.1.1]: https://github.com/OpenAgentHQ/modeldock/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/OpenAgentHQ/modeldock/releases/tag/v0.1.0
5 changes: 2 additions & 3 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ inference itself; it orchestrates runtimes (starting with Ollama).

- **Repo:** https://github.com/OpenAgentHQ/modeldock.git
- **Language:** Python 3.9–3.12 · `src/` layout · pure-Python, minimal deps
- **Status:** Greenfield (docs only; no source yet)
- **Status:** Active development. Dynamic catalog from ollama.com.

---

Expand All @@ -41,7 +41,7 @@ inference itself; it orchestrates runtimes (starting with Ollama).
3. **Extensible by design.** New runtimes = one adapter class + one entry-point
line. No core/CLI/API changes.
4. **Zero-config, beginner-friendly, cross-platform.** `platformdirs` for
paths; works offline via bundled `catalog.json`.
paths; works offline via cached catalog from ollama.com.
5. **Typed errors, library-friendly logging.** Never `basicConfig()` at import;
raise `ModelDockError` subclasses with actionable context.

Expand All @@ -56,7 +56,6 @@ Domain: modeldock/domain/ (pure entities, no I/O)
Ports: modeldock/ports/ (typing.Protocol interfaces)
Adapters: modeldock/adapters/ (runtimes, registry, downloaders, cache, progress)
Common: modeldock/common/ (config, logging, platform, http, errors)
Data: modeldock/data/catalog.json (bundled model registry)
```

**Key ports:** `RuntimePort`, `RegistryPort`, `DownloaderPort`, `CachePort`,
Expand Down
8 changes: 6 additions & 2 deletions Development.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ python -m modeldock --help

## 8. Configuration During Development

Config precedence (low → high): built-in defaults → bundled `catalog.json`
Config precedence (low → high): built-in defaults → dynamic catalog (ollama.com)
system config → user config (`~/.config/modeldock/config.toml` or
`%APPDATA%\modeldock\config.toml`) → env vars (`MODELDOCK_*`) → runtime overrides.

Expand Down Expand Up @@ -200,7 +200,11 @@ See `AGENT.md` → "Extension Checklist" for the full list.

## 10. Adding Models to the Registry

Edit `src/modeldock/data/catalog.json` (no code change). Each entry:
Model discovery is **dynamic** — scraped live from `ollama.com/library`.
New models appear automatically after catalog refresh (24h TTL cache).

To add a model to the **bundled fallback**, edit `src/modeldock/data/catalog.json`
(no code change). Each entry:

```json
{
Expand Down
37 changes: 30 additions & 7 deletions INSTRUCTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ src/modeldock/
adapters/ runtimes, registry, downloaders, cache, progress
cli/ Typer CLI, thin wrappers over core
common/ config, logging, platform, http, errors
data/ catalog.json (bundled registry)
data/ (removed — catalog is now dynamic from ollama.com)
```

**Dependency direction (never violate):**
Expand All @@ -115,8 +115,9 @@ src/modeldock/
- Library-friendly logging: no `basicConfig()` at import.
- Typed errors (`ModelDockError` subclasses) with actionable context.
- Cross-platform: `platformdirs` for paths; no hardcoded `/` or `C:\`.
- **NEVER generate GitHub content with a coding agent.** Issues, PRs, commits,
and release notes must be authored by a human, not composed by a coding agent.
- **GitHub content may be generated by coding agents.** Issues, PRs, commits,
and release notes may be composed by a coding agent, but must follow the
repo's writing rules and templates.

---

Expand Down Expand Up @@ -147,7 +148,11 @@ Targets: ruff clean, `mypy --strict` clean, bandit clean, ≥80% coverage.

## 7. Adding Models

Edit `src/modeldock/data/catalog.json`. No code change. See
Model discovery is now **dynamic** — scraped live from `ollama.com/library`.
New models appear automatically after catalog refresh (24h TTL cache).

To add a model to the **bundled fallback**, edit `src/modeldock/data/catalog.json`
(temporarily restored for offline fallback only). See
[`Development.md`](./Development.md) §10 for the entry shape.

---
Expand Down Expand Up @@ -236,12 +241,30 @@ Rules for all generated GitHub content (issues, PRs, commits, docs).
Professional maintainer. Clear, concise, technically accurate. At most one emoji
per section.

### Authorship (non-negotiable)
### Authorship

- **NEVER generate GitHub content with a coding agent.** Issues, PRs, commits,
and release notes must be written by a human, not composed by a coding agent.
- **GitHub content may be generated by coding agents.** Issues, PRs, commits,
and release notes may be composed by a coding agent, but must follow the
repo's writing rules and templates.

### Validation (before output)

Confirm: Markdown renders correctly · file paths use `/` · code blocks fenced ·
headings consistent · links formatted.

### PowerShell Escaping

When using `gh pr create` or `gh pr edit` with `--body` on Windows PowerShell,
backticks are interpreted as escape characters and special characters get mangled.
Always write the body to a temp file first, then use `--body-file`:

```bash
# Write body to file
Set-Content -Path .tmp\pr_body.md -Value "## Summary`nYour content here"

# Use file instead of inline --body
gh pr create --body-file .tmp\pr_body.md
gh pr edit 123 --body-file .tmp\pr_body.md
```

This preserves backticks, checkboxes, and special characters in the PR body.
1 change: 1 addition & 0 deletions PROJECT.MD
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,7 @@ model = load("llama3")

ModelDock automatically:

* **Scrapes a live catalog** from `ollama.com/library` (cached 24h, falls back to bundled)
* Checks whether the model is already installed
* Downloads the model if it is missing
* Displays download progress
Expand Down
11 changes: 11 additions & 0 deletions QUICKSTART.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,17 @@ modeldock config show
modeldock config set auto_install true
```

### Catalog Source

ModelDock scrapes `ollama.com/library` for a live model catalog, cached locally
for 24 hours. Set `catalog_source` in config or `MODELDOCK_CATALOG_SOURCE` env var:

| Value | Behavior |
|-------|----------|
| `auto` | Try dynamic, fallback to bundled (default) |
| `ollama` | Dynamic only — requires internet |
| `bundled` | Static catalog.json only — fully offline |

---

## Supported Runtimes
Expand Down
14 changes: 12 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ write `md.load("llama3")` and ModelDock handles the rest.
- **Smart caching** — never re-download installed models; content-addressed offline cache.
- **Extensible runtimes** — Ollama ships first; LM Studio, llama.cpp, Jan AI, GPT4All, vLLM are drop-in adapters.
- **Cross-platform** — Windows, macOS, Linux via `platformdirs`.
- **Zero-config, beginner-friendly** — works offline with a bundled catalog.
- **Zero-config, beginner-friendly** — dynamic catalog from ollama.com with offline caching.

## Quick Start

Expand Down Expand Up @@ -128,9 +128,19 @@ Domain: modeldock/domain/ (pure entities, no I/O)
Ports: modeldock/ports/ (typing.Protocol interfaces)
Adapters: modeldock/adapters/ (runtimes, registry, downloaders, cache, progress)
Common: modeldock/common/ (config, logging, platform, http, errors)
Data: modeldock/data/catalog.json (bundled model registry)
```

### Catalog Source

ModelDock scrapes `ollama.com/library` for a live model catalog, cached locally
for 24 hours. Set `catalog_source` in config or `MODELDOCK_CATALOG_SOURCE` env var:

| Value | Behavior |
|-------|----------|
| `auto` | Try dynamic, fallback to bundled (default) |
| `ollama` | Dynamic only — requires internet |
| `bundled` | Static catalog.json only — fully offline |

See [Architecture.md](Architecture.md) for the full design contract.

## Configuration
Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Send an email to **opensource@openagenthq.com** with:
- Keep ModelDock and its dependencies up to date (`pip install -U modeldock`).
- Use environment variables for any secrets; never hardcode them.
- Download models only from trusted runtimes/registries.
- Review the bundled `catalog.json` and any remote registry sources.
- Review the dynamic catalog source (`ollama.com`) and any bundled registry sources.

### For Contributors

Expand Down
Loading
Loading