Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Full release notes with details on each version: [GitHub Releases](https://githu

## 0.9.32 (unreleased)

- Feat: add an explicit `copilot-sdk` LLM backend for semantic extraction, LLM deduplication, community labels, PR triage, and raster-image attachments. It uses the official Python SDK with one persistent headless runtime and a fresh isolated session per Graphify request, defaults to the enterprise-authenticated system Copilot CLI, runs in SDK `empty` mode with no tools/MCP/persistent memory/remote sessions, and rejects every permission request. SDK state is placed under a temporary Copilot home; each request session is disconnected and permanently deleted, and uncertain cleanup discards the runtime. SDK requests automatically fall back to the existing `copilot-cli` transport unless `GRAPHIFY_COPILOT_SDK_FALLBACK=0`; Python 3.10 therefore remains supported through the fallback while the optional SDK dependency is installed only on Python 3.11+. Both Copilot backends remain explicit-only.
- Feat: add an explicit `copilot-cli` LLM backend for semantic extraction, LLM deduplication, community labels, and PR triage. It reuses the official GitHub Copilot CLI credential store and supports GHE.com enterprise SSO through `copilot login --host ...` / `COPILOT_GH_HOST`, including configurable `--model` selection. Prompts are piped over stdin from an empty temporary directory, tool and repository-agent features are restricted where the installed CLI supports those controls, remote session export is disabled when available, and requests run serially unless `GRAPHIFY_COPILOT_CLI_PARALLEL=1` is set. The backend is never auto-selected merely because the CLI is installed.
- Fix: incremental extraction and `_rebuild_code` no longer drop a file's other tier (#2333, #2334, #2336). Node/edge ownership was keyed on `source_file` alone, so a semantic re-extract deleted a doc's AST headings and a full rebuild deleted document AST nodes. Merge is now tier-aware (an AST re-extract replaces only AST nodes and keeps the semantic layer, and vice versa), the `_origin` provenance marker is backfilled on load so old graphs self-heal, and the full-rebuild drop is scoped to sources actually regenerated.
- Fix: `graphify update` preserves the graph's `directed` flag instead of rebuilding it undirected (#2342, thanks @Rishet11), so God-node / path ranking keeps its direction on both the clustered and `--no-cluster` rebuild paths.
- Fix: a numeric or otherwise non-string node id from an LLM fragment no longer aborts the build with a TypeError (#2326, thanks @Rishet11); ids are coerced consistently across nodes, edges, and hyperedges.
Expand Down
51 changes: 47 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,7 @@ Codex users also need `multi_agent = true` under `[features]` in `~/.codex/confi
| `gemini` | Google Gemini API | `uv tool install "graphifyy[gemini]"` |
| `anthropic` | Anthropic Claude API (`--backend claude`, uses `ANTHROPIC_API_KEY`) | `uv tool install "graphifyy[anthropic]"` |
| `bedrock` | AWS Bedrock (uses IAM, no API key) | `uv tool install "graphifyy[bedrock]"` |
| `copilot` | GitHub Copilot SDK backend (Python 3.11+; automatic CLI fallback) | `uv tool install --python 3.12 "graphifyy[copilot]"` |
| `azure` | Azure OpenAI Service (`--backend azure`, uses `AZURE_OPENAI_API_KEY` + `AZURE_OPENAI_ENDPOINT`) | `uv tool install "graphifyy[openai]"` |
| `sql` | SQL schema extraction | `uv tool install "graphifyy[sql]"` |
| `postgres` | Live PostgreSQL introspection (`--postgres DSN`) | `uv tool install "graphifyy[postgres]"` |
Expand Down Expand Up @@ -515,9 +516,17 @@ These are only needed for **headless / CI extraction** (`graphify extract`). Whe
| `AZURE_OPENAI_API_VERSION` | Azure API version override | optional — default `2024-12-01-preview` |
| `AZURE_OPENAI_DEPLOYMENT` or `GRAPHIFY_AZURE_MODEL` | Azure deployment name | optional — default `gpt-4o` |
| `AWS_*` / `~/.aws/credentials` | AWS Bedrock — standard credential chain | `--backend bedrock` (no API key, uses IAM) |
| `COPILOT_GH_HOST` | GitHub host inherited by the official Copilot SDK/CLI runtime, including GHE.com data-residency hosts such as `example.ghe.com` | `--backend copilot-sdk` or `copilot-cli` (recommended when multiple accounts/hosts are configured) |
| `GRAPHIFY_COPILOT_SDK_MODEL`, `GRAPHIFY_COPILOT_MODEL`, or `COPILOT_MODEL` | Model requested from the Copilot SDK; `--model` takes precedence | optional — default `auto`, subject to enterprise policy and plan availability |
| `GRAPHIFY_COPILOT_CLI_MODEL` or `COPILOT_MODEL` | Model requested from the standalone Copilot CLI backend; `--model` takes precedence | optional — default `auto` |
| `GRAPHIFY_COPILOT_SDK_CLI_PATH` or `COPILOT_CLI_PATH` | Managed system Copilot CLI executable used by the SDK's stdio transport | optional — defaults to `copilot` / `copilot.cmd` |
| `GRAPHIFY_COPILOT_SDK_USE_BUNDLED_CLI` | Use the SDK-downloaded version-pinned runtime instead of the system CLI | optional — set `1`; system CLI is the default for managed enterprise workstations |
| `GRAPHIFY_COPILOT_SDK_FALLBACK` | Permit automatic fallback from `copilot-sdk` to `copilot-cli` | optional — enabled by default; set `0` to require SDK success |
| `GRAPHIFY_COPILOT_SDK_PARALLEL` | Allow concurrent SDK sessions | optional — set `1` to opt in; serial by default |
| `GRAPHIFY_COPILOT_CLI_PARALLEL` | Allow concurrent standalone Copilot CLI subprocesses | optional — set `1` to opt in; serial by default |
| `GRAPHIFY_MAX_WORKERS` | AST parallelism thread count | optional — also `--max-workers` flag |
| `GRAPHIFY_MAX_OUTPUT_TOKENS` | Raise output cap for dense corpora | optional — e.g. `32768` for large files |
| `GRAPHIFY_API_TIMEOUT` | Per-call timeout in seconds for HTTP, claude-cli, Anthropic SDK, and Bedrock backends (default: 600) | optional — also `--api-timeout` flag |
| `GRAPHIFY_API_TIMEOUT` | Per-call timeout in seconds for HTTP, claude-cli, copilot-sdk, copilot-cli, Anthropic SDK, and Bedrock backends (default: 600) | optional — also `--api-timeout` flag |
| `GRAPHIFY_MAX_RETRIES` | How many times to retry a rate-limited (429) request before giving up (default: 6; honors `Retry-After`) | optional — raise for strict per-org limits (e.g. kimi); `0` disables |
| `GRAPHIFY_FORCE` | Force graph rebuild even with fewer nodes | optional — also `--force` flag |
| `GRAPHIFY_GOOGLE_WORKSPACE` | Auto-enable Google Workspace export | optional — set to `1` |
Expand All @@ -533,13 +542,46 @@ These are only needed for **headless / CI extraction** (`graphify extract`). Whe

---

## GitHub Copilot SDK and CLI backends

`--backend copilot-sdk` is the preferred structured integration. It uses the official Python SDK to control a persistent headless Copilot runtime over JSON-RPC, creates a fresh isolated session for each Graphify request, disconnects and permanently deletes that session after use, supports image file attachments, and automatically falls back to the existing one-shot `copilot-cli` transport when the SDK is unavailable or fails. `--backend copilot-cli` remains available as a standalone diagnostic and compatibility backend.

Install the optional SDK under Python 3.11 or newer, then authenticate the official CLI to the enterprise host:

```bash
uv tool install --python 3.12 "graphifyy[copilot]"
copilot login --host https://example.ghe.com

export COPILOT_GH_HOST=example.ghe.com
graphify extract ./docs --backend copilot-sdk --model auto
```

```powershell
$env:COPILOT_GH_HOST = "example.ghe.com"
graphify extract ./docs `
--backend copilot-sdk `
--model auto
```

Graphify defaults the SDK to the managed system `copilot` executable so it uses the same enterprise login and software-management path. Set `GRAPHIFY_COPILOT_SDK_USE_BUNDLED_CLI=1` to opt into the SDK-downloaded matching runtime. On Python 3.10, the SDK package cannot load and the explicit `copilot-sdk` backend uses `copilot-cli`; set `GRAPHIFY_COPILOT_SDK_FALLBACK=0` to require SDK success.

The SDK client runs in `mode="empty"` with a temporary working directory and temporary `COPILOT_HOME`, exposes no tools or MCP servers, rejects every permission request, disables persistent memory/infinite sessions and remote sessions, and does not configure SDK telemetry. Each request uses a unique session ID; Graphify disconnects and permanently deletes the session before returning, and discards the runtime if cleanup cannot be verified. It is never auto-selected. These controls limit local agent capabilities, but the selected Copilot model still receives the source chunks and remains governed by enterprise policy and approved-use boundaries.

SDK model precedence is `--model`, `GRAPHIFY_COPILOT_SDK_MODEL`, `GRAPHIFY_COPILOT_MODEL`, `COPILOT_MODEL`, then `auto`. Requests are serial by default; `GRAPHIFY_COPILOT_SDK_PARALLEL=1` opts into concurrency. The CLI fallback has separate `GRAPHIFY_COPILOT_CLI_MODEL` and `GRAPHIFY_COPILOT_CLI_PARALLEL` controls.

Copilot responses do not expose provider-style billing data to Graphify, so token counts are estimates and Graphify's provider-cost field is `$0`; GitHub AI credits or plan allowances can still be consumed.

See [`docs/copilot-sdk-backend.md`](docs/copilot-sdk-backend.md) for architecture, GitHub Enterprise setup, fallback behavior, image handling, security controls, and troubleshooting. See [`docs/copilot-cli-backend.md`](docs/copilot-cli-backend.md) for the standalone fallback transport.

---

## Privacy

- **Code files** — processed locally via tree-sitter. Nothing leaves your machine. A code-only corpus requires no API key — `graphify extract` runs fully offline. On a mixed repo, add `--code-only` to index just the code and skip the docs/PDFs/images that would otherwise need an LLM.
- **Video / audio** — transcribed locally with faster-whisper. Nothing leaves your machine.
- **Docs, PDFs, images** — sent to your AI assistant for semantic extraction (via the `/graphify` skill, using whatever model your IDE session runs). Headless `graphify extract` requires `GEMINI_API_KEY` / `GOOGLE_API_KEY` (Gemini), `MOONSHOT_API_KEY` (Kimi), `ANTHROPIC_API_KEY` (Claude), `OPENAI_API_KEY` (OpenAI), `DEEPSEEK_API_KEY` (DeepSeek), a running Ollama instance (`OLLAMA_BASE_URL`), AWS credentials via the standard provider chain (Bedrock - no API key needed, uses IAM), or the `claude` CLI binary (Claude Code - no API key needed, uses your Claude subscription). The `--dedup-llm` flag uses the same key.
- **Docs, PDFs, images** — sent to your AI assistant for semantic extraction (via the `/graphify` skill, using whatever model your IDE session runs). Headless `graphify extract` requires `GEMINI_API_KEY` / `GOOGLE_API_KEY` (Gemini), `MOONSHOT_API_KEY` (Kimi), `ANTHROPIC_API_KEY` (Claude), `OPENAI_API_KEY` (OpenAI), `DEEPSEEK_API_KEY` (DeepSeek), a running Ollama instance (`OLLAMA_BASE_URL`), AWS credentials via the standard provider chain (Bedrock - no API key needed, uses IAM), the `claude` CLI binary (Claude Code - no API key needed, uses your Claude subscription), or the official Copilot SDK/CLI (`--backend copilot-sdk` or `copilot-cli`, using its signed-in GitHub/Copilot account). The `--dedup-llm` flag uses the same backend.
- **Data residency** — `graphify extract` auto-detects which provider to use based on which API key is set (priority: Gemini → Kimi → Claude → OpenAI → DeepSeek → Azure → Bedrock → Ollama). For code with data-residency requirements, use `--backend ollama` (fully local) or pass an explicit `--backend` flag. Kimi (`MOONSHOT_API_KEY`) routes to Moonshot AI servers in China.
- **No telemetry**, no usage tracking, no analytics.
- **Graphify telemetry** — Graphify itself adds no telemetry, usage tracking, or analytics. External LLM providers and their CLIs retain their own service and enterprise-monitoring behavior; Graphify does not opt the Copilot SDK into telemetry, while enterprise-managed runtime settings still apply.
- **Query logging** — every `graphify query`, `graphify path`, `graphify explain`, and MCP `query_graph` call is logged to `~/.cache/graphify-queries.log` in JSON Lines format (timestamp, question, corpus, nodes returned, duration). Full subgraph responses are **not** stored by default. Set `GRAPHIFY_QUERY_LOG_DISABLE=1` to opt out, or `GRAPHIFY_QUERY_LOG=/dev/null` to silence without disabling the code path.

---
Expand Down Expand Up @@ -720,7 +762,7 @@ graphify antigravity install # .agents/rules + .agents/workflows (Google A
graphify antigravity uninstall

graphify extract ./docs # headless LLM extraction for CI (no IDE needed)
graphify extract ./docs --backend gemini # explicit backend: gemini, kimi, claude, openai, deepseek, ollama, bedrock, or claude-cli
graphify extract ./docs --backend gemini # explicit backend: gemini, kimi, claude, openai, deepseek, ollama, bedrock, azure, claude-cli, copilot-sdk, or copilot-cli
graphify extract ./docs --backend gemini --model gemini-3.1-pro-preview
graphify extract ./docs --backend ollama # local Ollama (set OLLAMA_BASE_URL / OLLAMA_MODEL) - no API key needed for loopback
OPENAI_BASE_URL=http://localhost:8080/v1 OPENAI_MODEL=my-model graphify extract ./docs --backend openai # any OpenAI-compatible server (llama.cpp, vLLM, LM Studio)
Expand All @@ -729,6 +771,7 @@ GRAPHIFY_OLLAMA_NUM_CTX=32768 graphify extract ./docs --backend ollama # overr
GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama # unload model after each chunk (saves VRAM on small GPUs)
graphify extract ./docs --backend bedrock # AWS Bedrock via IAM - no API key, uses AWS credential chain
graphify extract ./docs --backend claude-cli # route through Claude Code CLI - no API key, uses your Claude subscription
COPILOT_GH_HOST=example.ghe.com graphify extract ./docs --backend copilot-sdk --model auto # preferred SDK transport with automatic CLI fallback
graphify extract ./docs --backend azure # Azure OpenAI (set AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT)
graphify extract ./docs --max-workers 16 # AST parallelism (also GRAPHIFY_MAX_WORKERS)
graphify extract --postgres "postgresql://user:pass@host/db" # introspect live PostgreSQL schema directly
Expand Down
Loading