diff --git a/.dockerignore b/.dockerignore index 244e6034093a..0e4a88fd2fa7 100644 --- a/.dockerignore +++ b/.dockerignore @@ -14,3 +14,6 @@ node_modules .env *.md + +# Runtime data (bind-mounted at /opt/data; must not leak into build context) +data/ diff --git a/.env.example b/.env.example index f2c5769c6555..066e93f7c99b 100644 --- a/.env.example +++ b/.env.example @@ -24,6 +24,15 @@ # Optional base URL override (default: Google's OpenAI-compatible endpoint) # GEMINI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai +# ============================================================================= +# LLM PROVIDER (Ollama Cloud) +# ============================================================================= +# Cloud-hosted open models via Ollama's OpenAI-compatible endpoint. +# Get your key at: https://ollama.com/settings +# OLLAMA_API_KEY=your_ollama_key_here +# Optional base URL override (default: https://ollama.com/v1) +# OLLAMA_BASE_URL=https://ollama.com/v1 + # ============================================================================= # LLM PROVIDER (z.ai / GLM) # ============================================================================= @@ -45,6 +54,14 @@ # KIMI_BASE_URL=https://api.moonshot.cn/v1 # For Moonshot China keys # KIMI_CN_API_KEY= # Dedicated Moonshot China key +# ============================================================================= +# LLM PROVIDER (Arcee AI) +# ============================================================================= +# Arcee AI provides access to Trinity models (trinity-mini, trinity-large-*) +# Get an Arcee key at: https://chat.arcee.ai/ +# ARCEEAI_API_KEY= +# ARCEE_BASE_URL= # Override default base URL + # ============================================================================= # LLM PROVIDER (MiniMax) # ============================================================================= @@ -137,6 +154,10 @@ # Only override here if you need to force a backend without touching config.yaml: # TERMINAL_ENV=local +# Override the container runtime binary (e.g. to use Podman instead of Docker). +# Useful on systems where Docker's storage driver is broken or unavailable. +# HERMES_DOCKER_BINARY=/usr/local/bin/podman + # Container images (for singularity/docker/modal backends) # TERMINAL_DOCKER_IMAGE=nikolaik/python-nodejs:python3.11-nodejs20 # TERMINAL_SINGULARITY_IMAGE=docker://nikolaik/python-nodejs:python3.11-nodejs20 diff --git a/.envrc b/.envrc index 3550a30f2de3..45c59523cbe4 100644 --- a/.envrc +++ b/.envrc @@ -1 +1,5 @@ +watch_file pyproject.toml uv.lock +watch_file ui-tui/package-lock.json ui-tui/package.json +watch_file flake.nix flake.lock nix/devShell.nix nix/tui.nix nix/package.nix nix/python.nix + use flake diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 60a11e294f66..67a3f64aa372 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -11,6 +11,7 @@ body: **Before submitting**, please: - [ ] Search [existing issues](https://github.com/NousResearch/hermes-agent/issues) to avoid duplicates - [ ] Update to the latest version (`hermes update`) and confirm the bug still exists + - [ ] Run `hermes debug share` and paste the links below (see Debug Report section) - type: textarea id: description @@ -82,6 +83,25 @@ body: - Slack - WhatsApp + - type: textarea + id: debug-report + attributes: + label: Debug Report + description: | + Run `hermes debug share` from your terminal and paste the links it prints here. + This uploads your system info, config, and recent logs to a paste service automatically. + + If you're in an interactive chat session, you can also use the `/debug` slash command — it does the same thing. + + If the upload fails, run `hermes debug share --local` and paste the output directly. + placeholder: | + Report https://paste.rs/abc123 + agent.log https://paste.rs/def456 + gateway.log https://paste.rs/ghi789 + render: shell + validations: + required: true + - type: input id: os attributes: @@ -97,8 +117,6 @@ body: label: Python Version description: Output of `python --version` placeholder: "3.11.9" - validations: - required: true - type: input id: hermes-version @@ -106,14 +124,14 @@ body: label: Hermes Version description: Output of `hermes version` placeholder: "2.1.0" - validations: - required: true - type: textarea id: logs attributes: - label: Relevant Logs / Traceback - description: Paste any error output, traceback, or log messages. This will be auto-formatted as code. + label: Additional Logs / Traceback (optional) + description: | + The debug report above covers most logs. Use this field for any extra error output, + tracebacks, or screenshots not captured by `hermes debug share`. render: shell - type: textarea diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index 8dba7d43d544..720cc8f1f27c 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -71,3 +71,15 @@ body: label: Contribution options: - label: I'd like to implement this myself and submit a PR + + - type: textarea + id: debug-report + attributes: + label: Debug Report (optional) + description: | + If this feature request is related to a problem you're experiencing, run `hermes debug share` and paste the links here. + In an interactive chat session, you can use `/debug` instead. + This helps us understand your environment and any related logs. + placeholder: | + Report https://paste.rs/abc123 + render: shell diff --git a/.github/ISSUE_TEMPLATE/setup_help.yml b/.github/ISSUE_TEMPLATE/setup_help.yml index f13eea4a3cd0..974181b5d568 100644 --- a/.github/ISSUE_TEMPLATE/setup_help.yml +++ b/.github/ISSUE_TEMPLATE/setup_help.yml @@ -9,7 +9,8 @@ body: Sorry you're having trouble! Please fill out the details below so we can help. **Quick checks first:** - - Run `hermes doctor` and include the output below + - Run `hermes debug share` and paste the links in the Debug Report section below + - If you're in a chat session, you can use `/debug` instead — it does the same thing - Try `hermes update` to get the latest version - Check the [README troubleshooting section](https://github.com/NousResearch/hermes-agent#troubleshooting) - For general questions, consider the [Nous Research Discord](https://discord.gg/NousResearch) for faster help @@ -74,10 +75,21 @@ body: placeholder: "2.1.0" - type: textarea - id: doctor-output + id: debug-report attributes: - label: Output of `hermes doctor` - description: Run `hermes doctor` and paste the full output. This will be auto-formatted. + label: Debug Report + description: | + Run `hermes debug share` from your terminal and paste the links it prints here. + This uploads your system info, config, and recent logs to a paste service automatically. + + If you're in an interactive chat session, you can also use the `/debug` slash command — it does the same thing. + + If the upload fails or install didn't get that far, run `hermes debug share --local` and paste the output directly. + If even that doesn't work, run `hermes doctor` and paste that output instead. + placeholder: | + Report https://paste.rs/abc123 + agent.log https://paste.rs/def456 + gateway.log https://paste.rs/ghi789 render: shell - type: textarea diff --git a/.github/actions/nix-setup/action.yml b/.github/actions/nix-setup/action.yml new file mode 100644 index 000000000000..0fcd7784bc98 --- /dev/null +++ b/.github/actions/nix-setup/action.yml @@ -0,0 +1,8 @@ +name: 'Setup Nix' +description: 'Install Nix with DeterminateSystems and enable magic-nix-cache' + +runs: + using: composite + steps: + - uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22 + - uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13 diff --git a/.github/workflows/contributor-check.yml b/.github/workflows/contributor-check.yml new file mode 100644 index 000000000000..3ca4991c615f --- /dev/null +++ b/.github/workflows/contributor-check.yml @@ -0,0 +1,73 @@ +name: Contributor Attribution Check + +on: + pull_request: + branches: [main] + paths: + # Only run when code files change (not docs-only PRs) + - '*.py' + - '**/*.py' + - '.github/workflows/contributor-check.yml' + +permissions: + contents: read + +jobs: + check-attribution: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + with: + fetch-depth: 0 # Full history needed for git log + + - name: Check for unmapped contributor emails + run: | + # Get the merge base between this PR and main + MERGE_BASE=$(git merge-base origin/main HEAD) + + # Find any new author emails in this PR's commits + NEW_EMAILS=$(git log ${MERGE_BASE}..HEAD --format='%ae' --no-merges | sort -u) + + if [ -z "$NEW_EMAILS" ]; then + echo "No new commits to check." + exit 0 + fi + + # Check each email against AUTHOR_MAP in release.py + MISSING="" + while IFS= read -r email; do + # Skip teknium and bot emails + case "$email" in + *teknium*|*noreply@github.com*|*dependabot*|*github-actions*|*anthropic.com*|*cursor.com*) + continue ;; + esac + + # Check if email is in AUTHOR_MAP (either as a key or matches noreply pattern) + if echo "$email" | grep -qP '\+.*@users\.noreply\.github\.com'; then + continue # GitHub noreply emails auto-resolve + fi + + if ! grep -qF "\"${email}\"" scripts/release.py 2>/dev/null; then + AUTHOR=$(git log --author="$email" --format='%an' -1) + MISSING="${MISSING}\n ${email} (${AUTHOR})" + fi + done <<< "$NEW_EMAILS" + + if [ -n "$MISSING" ]; then + echo "" + echo "⚠️ New contributor email(s) not in AUTHOR_MAP:" + echo -e "$MISSING" + echo "" + echo "Please add mappings to scripts/release.py AUTHOR_MAP:" + echo -e "$MISSING" | while read -r line; do + email=$(echo "$line" | sed 's/^ *//' | cut -d' ' -f1) + [ -z "$email" ] && continue + echo " \"${email}\": \"\"," + done + echo "" + echo "To find the GitHub username for an email:" + echo " gh api 'search/users?q=EMAIL+in:email' --jq '.items[0].login'" + exit 1 + else + echo "✅ All contributor emails are mapped in AUTHOR_MAP." + fi diff --git a/.github/workflows/deploy-site.yml b/.github/workflows/deploy-site.yml index c55a62908d1b..3e78bc61b18e 100644 --- a/.github/workflows/deploy-site.yml +++ b/.github/workflows/deploy-site.yml @@ -1,11 +1,12 @@ name: Deploy Site on: + release: + types: [published] push: branches: [main] paths: - 'website/**' - - 'landingpage/**' - 'skills/**' - 'optional-skills/**' - '.github/workflows/deploy-site.yml' @@ -20,28 +21,34 @@ concurrency: cancel-in-progress: false jobs: - build-and-deploy: - # Only run on the upstream repository, not on forks + deploy-vercel: + if: github.event_name == 'release' + runs-on: ubuntu-latest + steps: + - name: Trigger Vercel Deploy + run: curl -X POST "${{ secrets.VERCEL_DEPLOY_HOOK }}" + + deploy-docs: if: github.repository == 'NousResearch/hermes-agent' runs-on: ubuntu-latest environment: name: github-pages url: ${{ steps.deploy.outputs.page_url }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: node-version: 20 cache: npm cache-dependency-path: website/package-lock.json - - uses: actions/setup-python@v5 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 with: python-version: '3.11' - name: Install PyYAML for skill extraction - run: pip install pyyaml httpx + run: pip install pyyaml==6.0.2 httpx==0.28.1 - name: Extract skill metadata for dashboard run: python3 website/scripts/extract-skills.py @@ -65,18 +72,13 @@ jobs: - name: Stage deployment run: | mkdir -p _site/docs - # Landing page at root - cp -r landingpage/* _site/ - # Docusaurus at /docs/ cp -r website/build/* _site/docs/ - # CNAME so GitHub Pages keeps the custom domain between deploys - echo "hermes-agent.nousresearch.com" > _site/CNAME - name: Upload artifact - uses: actions/upload-pages-artifact@v3 + uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3 with: path: _site - name: Deploy to GitHub Pages id: deploy - uses: actions/deploy-pages@v4 + uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4 diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml index 6b360b8c641f..228ee3396464 100644 --- a/.github/workflows/docker-publish.yml +++ b/.github/workflows/docker-publish.yml @@ -3,8 +3,13 @@ name: Docker Build and Publish on: push: branches: [main] - pull_request: - branches: [main] + paths: + - '**/*.py' + - 'pyproject.toml' + - 'uv.lock' + - 'Dockerfile' + - 'docker/**' + - '.github/workflows/docker-publish.yml' release: types: [published] @@ -23,21 +28,21 @@ jobs: timeout-minutes: 60 steps: - name: Checkout code - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: submodules: recursive - name: Set up QEMU - uses: docker/setup-qemu-action@v3 + uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 # Build amd64 only so we can `load` the image for smoke testing. # `load: true` cannot export a multi-arch manifest to the local daemon. # The multi-arch build follows on push to main / release. - name: Build image (amd64, smoke test) - uses: docker/build-push-action@v6 + uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6 with: context: . file: Dockerfile @@ -49,6 +54,14 @@ jobs: - name: Test image starts run: | + # The image runs as the hermes user (UID 10000). GitHub Actions + # creates /tmp/hermes-test root-owned by default, which hermes + # can't write to — chown it to match the in-container UID before + # bind-mounting. Real users doing `docker run -v ~/.hermes:...` + # with their own UID hit the same issue and have their own + # remediations (HERMES_UID env var, or chown locally). + mkdir -p /tmp/hermes-test + sudo chown -R 10000:10000 /tmp/hermes-test docker run --rm \ -v /tmp/hermes-test:/opt/data \ --entrypoint /opt/hermes/docker/entrypoint.sh \ @@ -56,14 +69,14 @@ jobs: - name: Log in to Docker Hub if: github.event_name == 'push' && github.ref == 'refs/heads/main' || github.event_name == 'release' - uses: docker/login-action@v3 + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - name: Push multi-arch image (main branch) if: github.event_name == 'push' && github.ref == 'refs/heads/main' - uses: docker/build-push-action@v6 + uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6 with: context: . file: Dockerfile @@ -75,7 +88,7 @@ jobs: - name: Push multi-arch image (release) if: github.event_name == 'release' - uses: docker/build-push-action@v6 + uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6 with: context: . file: Dockerfile diff --git a/.github/workflows/docs-site-checks.yml b/.github/workflows/docs-site-checks.yml index ea05d280466f..2f985122cb5d 100644 --- a/.github/workflows/docs-site-checks.yml +++ b/.github/workflows/docs-site-checks.yml @@ -7,13 +7,16 @@ on: - '.github/workflows/docs-site-checks.yml' workflow_dispatch: +permissions: + contents: read + jobs: docs-site-checks: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: node-version: 20 cache: npm @@ -23,7 +26,7 @@ jobs: run: npm ci working-directory: website - - uses: actions/setup-python@v5 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 with: python-version: '3.11' diff --git a/.github/workflows/electron-desktop.yml b/.github/workflows/electron-desktop.yml index eef967b79a74..74198f83b42c 100644 --- a/.github/workflows/electron-desktop.yml +++ b/.github/workflows/electron-desktop.yml @@ -92,6 +92,7 @@ jobs: run: npm run build:all env: POSTHOG_API_KEY: ${{ secrets.POSTHOG_API_KEY }} + VITE_ATOMIC_BACKEND_URL: ${{ secrets.VITE_ATOMIC_BACKEND_URL }} - name: Determine publish mode id: publish-mode diff --git a/.github/workflows/nix-lockfile-check.yml b/.github/workflows/nix-lockfile-check.yml new file mode 100644 index 000000000000..9c9bc734a64e --- /dev/null +++ b/.github/workflows/nix-lockfile-check.yml @@ -0,0 +1,68 @@ +name: Nix Lockfile Check + +on: + pull_request: + workflow_dispatch: + +permissions: + contents: read + pull-requests: write + +concurrency: + group: nix-lockfile-check-${{ github.ref }} + cancel-in-progress: true + +jobs: + check: + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + + - uses: ./.github/actions/nix-setup + + - name: Resolve head SHA + id: sha + shell: bash + run: | + FULL="${{ github.event.pull_request.head.sha || github.sha }}" + echo "full=$FULL" >> "$GITHUB_OUTPUT" + echo "short=${FULL:0:7}" >> "$GITHUB_OUTPUT" + + - name: Check lockfile hashes + id: check + continue-on-error: true + env: + LINK_SHA: ${{ steps.sha.outputs.full }} + run: nix run .#fix-lockfiles -- --check + + - name: Post sticky PR comment (stale) + if: steps.check.outputs.stale == 'true' && github.event_name == 'pull_request' + uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2.9.1 + with: + header: nix-lockfile-check + message: | + ### ⚠️ npm lockfile hash out of date + + Checked against commit [`${{ steps.sha.outputs.short }}`](${{ github.server_url }}/${{ github.repository }}/commit/${{ steps.sha.outputs.full }}) (PR head at check time). + + The `hash = "sha256-..."` line in these nix files no longer matches the committed `package-lock.json`: + + ${{ steps.check.outputs.report }} + + #### Apply the fix + + - [ ] **Apply lockfile fix** — tick to push a commit with the correct hashes to this PR branch + - Or [run the Nix Lockfile Fix workflow](${{ github.server_url }}/${{ github.repository }}/actions/workflows/nix-lockfile-fix.yml) manually (pass PR `#${{ github.event.pull_request.number }}`) + - Or locally: `nix run .#fix-lockfiles -- --apply` and commit the diff + + - name: Clear sticky PR comment (resolved) + if: steps.check.outputs.stale == 'false' && github.event_name == 'pull_request' + uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2.9.1 + with: + header: nix-lockfile-check + delete: true + + - name: Fail if stale + if: steps.check.outputs.stale == 'true' + run: exit 1 diff --git a/.github/workflows/nix-lockfile-fix.yml b/.github/workflows/nix-lockfile-fix.yml new file mode 100644 index 000000000000..a1c7dd6e5c99 --- /dev/null +++ b/.github/workflows/nix-lockfile-fix.yml @@ -0,0 +1,149 @@ +name: Nix Lockfile Fix + +on: + workflow_dispatch: + inputs: + pr_number: + description: 'PR number to fix (leave empty to run on the selected branch)' + required: false + type: string + issue_comment: + types: [edited] + +permissions: + contents: write + pull-requests: write + +concurrency: + group: nix-lockfile-fix-${{ github.event.issue.number || github.event.inputs.pr_number || github.ref }} + cancel-in-progress: false + +jobs: + fix: + # Run on manual dispatch OR when a task-list checkbox in the sticky + # lockfile-check comment flips from `[ ]` to `[x]`. + if: | + github.event_name == 'workflow_dispatch' || + (github.event_name == 'issue_comment' + && github.event.issue.pull_request != null + && contains(github.event.comment.body, '[x] **Apply lockfile fix**') + && !contains(github.event.changes.body.from, '[x] **Apply lockfile fix**')) + runs-on: ubuntu-latest + timeout-minutes: 25 + steps: + - name: Authorize & resolve PR + id: resolve + uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1 + with: + script: | + // 1. Verify the actor has write access — applies to both checkbox + // clicks and manual dispatch. + const { data: perm } = + await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, + repo: context.repo.repo, + username: context.actor, + }); + if (!['admin', 'write', 'maintain'].includes(perm.permission)) { + core.setFailed( + `${context.actor} lacks write access (has: ${perm.permission})` + ); + return; + } + + // 2. Resolve which ref to check out. + let prNumber = ''; + if (context.eventName === 'issue_comment') { + prNumber = String(context.payload.issue.number); + } else if (context.eventName === 'workflow_dispatch') { + prNumber = context.payload.inputs.pr_number || ''; + } + + if (!prNumber) { + core.setOutput('ref', context.ref.replace(/^refs\/heads\//, '')); + core.setOutput('repo', context.repo.repo); + core.setOutput('owner', context.repo.owner); + core.setOutput('pr', ''); + return; + } + + const { data: pr } = await github.rest.pulls.get({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: Number(prNumber), + }); + core.setOutput('ref', pr.head.ref); + core.setOutput('repo', pr.head.repo.name); + core.setOutput('owner', pr.head.repo.owner.login); + core.setOutput('pr', String(pr.number)); + + # Wipe the sticky lockfile-check comment to a "running" state as soon + # as the job is authorized, so the user sees their click was picked up + # before the ~minute of nix build work. + - name: Mark sticky as running + if: steps.resolve.outputs.pr != '' + uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2.9.1 + with: + header: nix-lockfile-check + number: ${{ steps.resolve.outputs.pr }} + message: | + ### 🔄 Applying lockfile fix… + + Triggered by @${{ github.actor }} — [workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}). + + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + with: + repository: ${{ steps.resolve.outputs.owner }}/${{ steps.resolve.outputs.repo }} + ref: ${{ steps.resolve.outputs.ref }} + token: ${{ secrets.GITHUB_TOKEN }} + fetch-depth: 0 + + - uses: ./.github/actions/nix-setup + + - name: Apply lockfile hashes + id: apply + run: nix run .#fix-lockfiles -- --apply + + - name: Commit & push + if: steps.apply.outputs.changed == 'true' + shell: bash + run: | + set -euo pipefail + git config user.name 'github-actions[bot]' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add nix/tui.nix nix/web.nix + git commit -m "fix(nix): refresh npm lockfile hashes" + git push + + - name: Update sticky (applied) + if: steps.apply.outputs.changed == 'true' && steps.resolve.outputs.pr != '' + uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2.9.1 + with: + header: nix-lockfile-check + number: ${{ steps.resolve.outputs.pr }} + message: | + ### ✅ Lockfile fix applied + + Pushed a commit refreshing the npm lockfile hashes — [workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}). + + - name: Update sticky (already current) + if: steps.apply.outputs.changed == 'false' && steps.resolve.outputs.pr != '' + uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2.9.1 + with: + header: nix-lockfile-check + number: ${{ steps.resolve.outputs.pr }} + message: | + ### ✅ Lockfile hashes already current + + Nothing to commit — [workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}). + + - name: Update sticky (failed) + if: failure() && steps.resolve.outputs.pr != '' + uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2.9.1 + with: + header: nix-lockfile-check + number: ${{ steps.resolve.outputs.pr }} + message: | + ### ❌ Lockfile fix failed + + See the [workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for logs. diff --git a/.github/workflows/nix.yml b/.github/workflows/nix.yml index dba33bfffcdc..7cae6f8151c1 100644 --- a/.github/workflows/nix.yml +++ b/.github/workflows/nix.yml @@ -4,15 +4,9 @@ on: push: branches: [main] pull_request: - paths: - - 'flake.nix' - - 'flake.lock' - - 'nix/**' - - 'pyproject.toml' - - 'uv.lock' - - 'hermes_cli/**' - - 'run_agent.py' - - 'acp_adapter/**' + +permissions: + contents: read concurrency: group: nix-${{ github.ref }} @@ -26,9 +20,8 @@ jobs: runs-on: ${{ matrix.os }} timeout-minutes: 30 steps: - - uses: actions/checkout@v4 - - uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22 - - uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + - uses: ./.github/actions/nix-setup - name: Check flake if: runner.os == 'Linux' run: nix flake check --print-build-logs diff --git a/.github/workflows/skills-index.yml b/.github/workflows/skills-index.yml index 6c03e40746ca..8beda195c664 100644 --- a/.github/workflows/skills-index.yml +++ b/.github/workflows/skills-index.yml @@ -20,14 +20,14 @@ jobs: if: github.repository == 'NousResearch/hermes-agent' runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - - uses: actions/setup-python@v5 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 with: python-version: '3.11' - name: Install dependencies - run: pip install httpx pyyaml + run: pip install httpx==0.28.1 pyyaml==6.0.2 - name: Build skills index env: @@ -35,7 +35,7 @@ jobs: run: python scripts/build_skills_index.py - name: Upload index artifact - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: skills-index path: website/static/api/skills-index.json @@ -53,25 +53,25 @@ jobs: # Only deploy on schedule or manual trigger (not on every push to the script) if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: name: skills-index path: website/static/api/ - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: node-version: 20 cache: npm cache-dependency-path: website/package-lock.json - - uses: actions/setup-python@v5 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 with: python-version: '3.11' - name: Install PyYAML for skill extraction - run: pip install pyyaml + run: pip install pyyaml==6.0.2 - name: Extract skill metadata for dashboard run: python3 website/scripts/extract-skills.py @@ -92,10 +92,10 @@ jobs: echo "hermes-agent.nousresearch.com" > _site/CNAME - name: Upload artifact - uses: actions/upload-pages-artifact@v3 + uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3 with: path: _site - name: Deploy to GitHub Pages id: deploy - uses: actions/deploy-pages@v4 + uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4 diff --git a/.github/workflows/supply-chain-audit.yml b/.github/workflows/supply-chain-audit.yml index b94e1dda4333..417e7b21f843 100644 --- a/.github/workflows/supply-chain-audit.yml +++ b/.github/workflows/supply-chain-audit.yml @@ -3,22 +3,39 @@ name: Supply Chain Audit on: pull_request: types: [opened, synchronize, reopened] + paths: + - '**/*.py' + - '**/*.pth' + - '**/setup.py' + - '**/setup.cfg' + - '**/sitecustomize.py' + - '**/usercustomize.py' + - '**/__init__.pth' permissions: pull-requests: write contents: read +# Narrow, high-signal scanner. Only fires on critical indicators of supply +# chain attacks (e.g. the litellm-style payloads). Low-signal heuristics +# (plain base64, plain exec/eval, dependency/Dockerfile/workflow edits, +# Actions version unpinning, outbound POST/PUT) were intentionally +# removed — they fired on nearly every PR and trained reviewers to ignore +# the scanner. Keep this file's checks ruthlessly narrow: if you find +# yourself adding WARNING-tier patterns here again, make a separate +# advisory-only workflow instead. + jobs: scan: - name: Scan PR for supply chain risks + name: Scan PR for critical supply chain risks runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: fetch-depth: 0 - - name: Scan diff for suspicious patterns + - name: Scan diff for critical patterns id: scan env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} @@ -28,19 +45,19 @@ jobs: BASE="${{ github.event.pull_request.base.sha }}" HEAD="${{ github.event.pull_request.head.sha }}" - # Get the full diff (added lines only) + # Added lines only, excluding lockfiles. DIFF=$(git diff "$BASE".."$HEAD" -- . ':!uv.lock' ':!*.lock' ':!package-lock.json' ':!yarn.lock' || true) FINDINGS="" - CRITICAL=false # --- .pth files (auto-execute on Python startup) --- + # The exact mechanism used in the litellm supply chain attack: + # https://github.com/BerriAI/litellm/issues/24512 PTH_FILES=$(git diff --name-only "$BASE".."$HEAD" | grep '\.pth$' || true) if [ -n "$PTH_FILES" ]; then - CRITICAL=true FINDINGS="${FINDINGS} ### 🚨 CRITICAL: .pth file added or modified - Python \`.pth\` files in \`site-packages/\` execute automatically when the interpreter starts — no import required. This is the exact mechanism used in the [litellm supply chain attack](https://github.com/BerriAI/litellm/issues/24512). + Python \`.pth\` files in \`site-packages/\` execute automatically when the interpreter starts — no import required. **Files:** \`\`\` @@ -49,13 +66,12 @@ jobs: " fi - # --- base64 + exec/eval combo (the litellm attack pattern) --- + # --- base64 decode + exec/eval on the same line (the litellm attack pattern) --- B64_EXEC_HITS=$(echo "$DIFF" | grep -n '^\+' | grep -iE 'base64\.(b64decode|decodebytes|urlsafe_b64decode)' | grep -iE 'exec\(|eval\(' | head -10 || true) if [ -n "$B64_EXEC_HITS" ]; then - CRITICAL=true FINDINGS="${FINDINGS} ### 🚨 CRITICAL: base64 decode + exec/eval combo - This is the exact pattern used in the [litellm supply chain attack](https://github.com/BerriAI/litellm/issues/24512) — base64-decoded strings passed to exec/eval to hide credential-stealing payloads. + Base64-decoded strings passed directly to exec/eval — the signature of hidden credential-stealing payloads. **Matches:** \`\`\` @@ -64,41 +80,12 @@ jobs: " fi - # --- base64 decode/encode (alone — legitimate uses exist) --- - B64_HITS=$(echo "$DIFF" | grep -n '^\+' | grep -iE 'base64\.(b64decode|b64encode|decodebytes|encodebytes|urlsafe_b64decode)|atob\(|btoa\(|Buffer\.from\(.*base64' | head -20 || true) - if [ -n "$B64_HITS" ]; then - FINDINGS="${FINDINGS} - ### ⚠️ WARNING: base64 encoding/decoding detected - Base64 has legitimate uses (images, JWT, etc.) but is also commonly used to obfuscate malicious payloads. Verify the usage is appropriate. - - **Matches (first 20):** - \`\`\` - ${B64_HITS} - \`\`\` - " - fi - - # --- exec/eval with string arguments --- - EXEC_HITS=$(echo "$DIFF" | grep -n '^\+' | grep -E '(exec|eval)\s*\(' | grep -v '^\+\s*#' | grep -v 'test_\|mock\|assert\|# ' | head -20 || true) - if [ -n "$EXEC_HITS" ]; then - FINDINGS="${FINDINGS} - ### ⚠️ WARNING: exec() or eval() usage - Dynamic code execution can hide malicious behavior, especially when combined with base64 or network fetches. - - **Matches (first 20):** - \`\`\` - ${EXEC_HITS} - \`\`\` - " - fi - - # --- subprocess with encoded/obfuscated commands --- - PROC_HITS=$(echo "$DIFF" | grep -n '^\+' | grep -E 'subprocess\.(Popen|call|run)\s*\(' | grep -iE 'base64|decode|encode|\\x|chr\(' | head -10 || true) + # --- subprocess with encoded/obfuscated command argument --- + PROC_HITS=$(echo "$DIFF" | grep -n '^\+' | grep -E 'subprocess\.(Popen|call|run)\s*\(' | grep -iE 'base64|\\x[0-9a-f]{2}|chr\(' | head -10 || true) if [ -n "$PROC_HITS" ]; then - CRITICAL=true FINDINGS="${FINDINGS} ### 🚨 CRITICAL: subprocess with encoded/obfuscated command - Subprocess calls with encoded arguments are a strong indicator of payload execution. + Subprocess calls whose command strings are base64- or hex-encoded are a strong indicator of payload execution. **Matches:** \`\`\` @@ -107,25 +94,12 @@ jobs: " fi - # --- Network calls to non-standard domains --- - EXFIL_HITS=$(echo "$DIFF" | grep -n '^\+' | grep -iE 'requests\.(post|put)\(|httpx\.(post|put)\(|urllib\.request\.urlopen' | grep -v '^\+\s*#' | grep -v 'test_\|mock\|assert' | head -10 || true) - if [ -n "$EXFIL_HITS" ]; then - FINDINGS="${FINDINGS} - ### ⚠️ WARNING: Outbound network calls (POST/PUT) - Outbound POST/PUT requests in new code could be data exfiltration. Verify the destination URLs are legitimate. - - **Matches (first 10):** - \`\`\` - ${EXFIL_HITS} - \`\`\` - " - fi - - # --- setup.py / setup.cfg install hooks --- - SETUP_HITS=$(git diff --name-only "$BASE".."$HEAD" | grep -E '(setup\.py|setup\.cfg|__init__\.pth|sitecustomize\.py|usercustomize\.py)$' || true) + # --- Install-hook files (setup.py/sitecustomize/usercustomize/__init__.pth) --- + # These execute during pip install or interpreter startup. + SETUP_HITS=$(git diff --name-only "$BASE".."$HEAD" | grep -E '(^|/)(setup\.py|setup\.cfg|sitecustomize\.py|usercustomize\.py|__init__\.pth)$' || true) if [ -n "$SETUP_HITS" ]; then FINDINGS="${FINDINGS} - ### ⚠️ WARNING: Install hook files modified + ### 🚨 CRITICAL: Install-hook file added or modified These files can execute code during package installation or interpreter startup. **Files:** @@ -135,58 +109,31 @@ jobs: " fi - # --- Compile/marshal/pickle (code object injection) --- - MARSHAL_HITS=$(echo "$DIFF" | grep -n '^\+' | grep -iE 'marshal\.loads|pickle\.loads|compile\(' | grep -v '^\+\s*#' | grep -v 'test_\|re\.compile\|ast\.compile' | head -10 || true) - if [ -n "$MARSHAL_HITS" ]; then - FINDINGS="${FINDINGS} - ### ⚠️ WARNING: marshal/pickle/compile usage - These can deserialize or construct executable code objects. - - **Matches:** - \`\`\` - ${MARSHAL_HITS} - \`\`\` - " - fi - - # --- Output results --- if [ -n "$FINDINGS" ]; then echo "found=true" >> "$GITHUB_OUTPUT" - if [ "$CRITICAL" = true ]; then - echo "critical=true" >> "$GITHUB_OUTPUT" - else - echo "critical=false" >> "$GITHUB_OUTPUT" - fi - # Write findings to a file (multiline env vars are fragile) echo "$FINDINGS" > /tmp/findings.md else echo "found=false" >> "$GITHUB_OUTPUT" - echo "critical=false" >> "$GITHUB_OUTPUT" fi - - name: Post warning comment + - name: Post critical finding comment if: steps.scan.outputs.found == 'true' env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | - SEVERITY="⚠️ Supply Chain Risk Detected" - if [ "${{ steps.scan.outputs.critical }}" = "true" ]; then - SEVERITY="🚨 CRITICAL Supply Chain Risk Detected" - fi + BODY="## 🚨 CRITICAL Supply Chain Risk Detected - BODY="## ${SEVERITY} - - This PR contains patterns commonly associated with supply chain attacks. This does **not** mean the PR is malicious — but these patterns require careful human review before merging. + This PR contains a pattern that has been used in real supply chain attacks. A maintainer must review the flagged code carefully before merging. $(cat /tmp/findings.md) --- - *Automated scan triggered by [supply-chain-audit](/.github/workflows/supply-chain-audit.yml). If this is a false positive, a maintainer can approve after manual review.*" + *Scanner only fires on high-signal indicators: .pth files, base64+exec/eval combos, subprocess with encoded commands, or install-hook files. Low-signal warnings were removed intentionally — if you're seeing this comment, the finding is worth inspecting.*" - gh pr comment "${{ github.event.pull_request.number }}" --body "$BODY" + gh pr comment "${{ github.event.pull_request.number }}" --body "$BODY" || echo "::warning::Could not post PR comment (expected for fork PRs — GITHUB_TOKEN is read-only)" - name: Fail on critical findings - if: steps.scan.outputs.critical == 'true' + if: steps.scan.outputs.found == 'true' run: | echo "::error::CRITICAL supply chain risk patterns detected in this PR. See the PR comment for details." exit 1 diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 1e45193b8d07..a92afdfa40dd 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -3,8 +3,17 @@ name: Tests on: push: branches: [main] + paths-ignore: + - '**/*.md' + - 'docs/**' pull_request: branches: [main] + paths-ignore: + - '**/*.md' + - 'docs/**' + +permissions: + contents: read # Cancel in-progress runs for the same PR/branch concurrency: @@ -14,16 +23,16 @@ concurrency: jobs: test: runs-on: ubuntu-latest - timeout-minutes: 10 + timeout-minutes: 20 steps: - name: Checkout code - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - name: Install system dependencies run: sudo apt-get update && sudo apt-get install -y ripgrep - name: Install uv - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5 - name: Set up Python 3.11 run: uv python install 3.11 @@ -49,10 +58,10 @@ jobs: timeout-minutes: 10 steps: - name: Checkout code - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - name: Install uv - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5 - name: Set up Python 3.11 run: uv python install 3.11 diff --git a/.gitignore b/.gitignore index 228fae50b40d..f66e70d29e15 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ +.DS_Store /venv/ /_pycache/ *.pyc* @@ -54,12 +55,18 @@ environments/benchmarks/evals/ # Web UI build output hermes_cli/web_dist/ +# Web UI assets — synced from @nous-research/ui at build time via +# `npm run sync-assets` (see web/package.json). +web/public/fonts/ +web/public/ds-assets/ + # Release script temp files .release_notes.md mini-swe-agent/ # Nix .direnv/ +.nix-stamps/ result .history website/static/api/skills-index.json @@ -67,3 +74,4 @@ website/static/api/skills-index.json # Desktop build artifacts desktop/build/ desktop/release/ +.cursor/ \ No newline at end of file diff --git a/.mailmap b/.mailmap new file mode 100644 index 000000000000..3f093fb5abe7 --- /dev/null +++ b/.mailmap @@ -0,0 +1,108 @@ +# .mailmap — canonical author mapping for git shortlog / git log / GitHub +# Format: Canonical Name +# See: https://git-scm.com/docs/gitmailmap +# +# This maps commit emails to GitHub noreply addresses so that: +# 1. `git shortlog -sn` shows deduplicated contributor counts +# 2. GitHub's contributor graph can attribute commits correctly +# 3. Contributors with personal/work emails get proper credit +# +# When adding entries: use the contributor's GitHub noreply email as canonical +# so GitHub can link commits to their profile. + +# === Teknium (multiple emails) === +Teknium <127238744+teknium1@users.noreply.github.com> +Teknium <127238744+teknium1@users.noreply.github.com> + +# === Contributors — personal/work emails mapped to GitHub noreply === +# Format: Canonical Name + +# Verified via GH API email search +luyao618 <364939526@qq.com> <364939526@qq.com> +ethernet8023 +nicoloboschi +cherifya +BongSuCHOI +dsocolobsky +pefontana +Helmi +hata1234 + +# Verified via PR investigation / salvage PR bodies +DeployFaith +flobo3 +gaixianggeng +KUSH42 +konsisumer +WorldInnovationsDepartment +m0n5t3r +sprmn24 +fancydirty +fxfitz +limars874 +AaronWong1999 +dippwho +duerzy +geoffwellman +hcshen0111 +jamesarch +stephenschoettler +Tranquil-Flow +Dusk1e +Awsh1 +WAXLYY +donrhmexe +hqhq1025 <1506751656@qq.com> <1506751656@qq.com> +BlackishGreen33 +tomqiaozc +MagicRay1217 +aaronagent <1115117931@qq.com> <1115117931@qq.com> +YoungYang963 +LongOddCode +Cafexss +Cygra +DomGrieco + +# Duplicate email mapping (same person, multiple emails) +Sertug17 <104278804+Sertug17@users.noreply.github.com> +yyovil +DomGrieco +dsocolobsky +olafthiele + +# Verified via git display name matching GH contributor username +cokemine +dalianmao000 +emozilla +jjovalle99 +kagura-agent +spniyant +olafthiele +r266-tech +xingkongliang +win4r +zhouboli +yongtenglei + +# Nous Research team +benbarclay +jquesnelle + +# GH contributor list verified +spideystreet +dorukardahan +MustafaKara7 +Hmbown +kamil-gwozdz +kira-ariaki +knopki +Unayung +SeeYangZhi +Julientalbot +lesterli +JiayuuWang +tesseracttars-creator +xinbenlv +SaulJWu +angelos +MestreY0d4-Uninter <241404605+MestreY0d4-Uninter@users.noreply.github.com> diff --git a/AGENTS.md b/AGENTS.md index 8f227968e3ae..ae78e005a05a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,65 +5,61 @@ Instructions for AI coding assistants and developers working on the hermes-agent ## Development Environment ```bash -source venv/bin/activate # ALWAYS activate before running Python +# Prefer .venv; fall back to venv if that's what your checkout has. +source .venv/bin/activate # or: source venv/bin/activate ``` +`scripts/run_tests.sh` probes `.venv` first, then `venv`, then +`$HOME/.hermes/hermes-agent/venv` (for worktrees that share a venv with the +main checkout). + ## Project Structure +File counts shift constantly — don't treat the tree below as exhaustive. +The canonical source is the filesystem. The notes call out the load-bearing +entry points you'll actually edit. + ``` hermes-agent/ -├── run_agent.py # AIAgent class — core conversation loop -├── model_tools.py # Tool orchestration, _discover_tools(), handle_function_call() +├── run_agent.py # AIAgent class — core conversation loop (~12k LOC) +├── model_tools.py # Tool orchestration, discover_builtin_tools(), handle_function_call() ├── toolsets.py # Toolset definitions, _HERMES_CORE_TOOLS list -├── cli.py # HermesCLI class — interactive CLI orchestrator +├── cli.py # HermesCLI class — interactive CLI orchestrator (~11k LOC) ├── hermes_state.py # SessionDB — SQLite session store (FTS5 search) -├── agent/ # Agent internals -│ ├── prompt_builder.py # System prompt assembly -│ ├── context_compressor.py # Auto context compression -│ ├── prompt_caching.py # Anthropic prompt caching -│ ├── auxiliary_client.py # Auxiliary LLM client (vision, summarization) -│ ├── model_metadata.py # Model context lengths, token estimation -│ ├── models_dev.py # models.dev registry integration (provider-aware context) -│ ├── display.py # KawaiiSpinner, tool preview formatting -│ ├── skill_commands.py # Skill slash commands (shared CLI/gateway) -│ └── trajectory.py # Trajectory saving helpers -├── hermes_cli/ # CLI subcommands and setup -│ ├── main.py # Entry point — all `hermes` subcommands -│ ├── config.py # DEFAULT_CONFIG, OPTIONAL_ENV_VARS, migration -│ ├── commands.py # Slash command definitions + SlashCommandCompleter -│ ├── callbacks.py # Terminal callbacks (clarify, sudo, approval) -│ ├── setup.py # Interactive setup wizard -│ ├── skin_engine.py # Skin/theme engine — CLI visual customization -│ ├── skills_config.py # `hermes skills` — enable/disable skills per platform -│ ├── tools_config.py # `hermes tools` — enable/disable tools per platform -│ ├── skills_hub.py # `/skills` slash command (search, browse, install) -│ ├── models.py # Model catalog, provider model lists -│ ├── model_switch.py # Shared /model switch pipeline (CLI + gateway) -│ └── auth.py # Provider credential resolution -├── tools/ # Tool implementations (one file per tool) -│ ├── registry.py # Central tool registry (schemas, handlers, dispatch) -│ ├── approval.py # Dangerous command detection -│ ├── terminal_tool.py # Terminal orchestration -│ ├── process_registry.py # Background process management -│ ├── file_tools.py # File read/write/search/patch -│ ├── web_tools.py # Web search/extract (Parallel + Firecrawl) -│ ├── browser_tool.py # Browserbase browser automation -│ ├── code_execution_tool.py # execute_code sandbox -│ ├── delegate_tool.py # Subagent delegation -│ ├── mcp_tool.py # MCP client (~1050 lines) +├── hermes_constants.py # get_hermes_home(), display_hermes_home() — profile-aware paths +├── hermes_logging.py # setup_logging() — agent.log / errors.log / gateway.log (profile-aware) +├── batch_runner.py # Parallel batch processing +├── agent/ # Agent internals (provider adapters, memory, caching, compression, etc.) +├── hermes_cli/ # CLI subcommands, setup wizard, plugins loader, skin engine +├── tools/ # Tool implementations — auto-discovered via tools/registry.py │ └── environments/ # Terminal backends (local, docker, ssh, modal, daytona, singularity) -├── gateway/ # Messaging platform gateway -│ ├── run.py # Main loop, slash commands, message dispatch -│ ├── session.py # SessionStore — conversation persistence -│ └── platforms/ # Adapters: telegram, discord, slack, whatsapp, homeassistant, signal +├── gateway/ # Messaging gateway — run.py + session.py + platforms/ +│ ├── platforms/ # Adapter per platform (telegram, discord, slack, whatsapp, +│ │ # homeassistant, signal, matrix, mattermost, email, sms, +│ │ # dingtalk, wecom, weixin, feishu, qqbot, bluebubbles, +│ │ # webhook, api_server, ...). See ADDING_A_PLATFORM.md. +│ └── builtin_hooks/ # Always-registered gateway hooks (boot-md, ...) +├── plugins/ # Plugin system (see "Plugins" section below) +│ ├── memory/ # Memory-provider plugins (honcho, mem0, supermemory, ...) +│ ├── context_engine/ # Context-engine plugins +│ └── / # Dashboard, image-gen, disk-cleanup, examples, ... +├── optional-skills/ # Heavier/niche skills shipped but NOT active by default +├── skills/ # Built-in skills bundled with the repo +├── ui-tui/ # Ink (React) terminal UI — `hermes --tui` +│ └── src/ # entry.tsx, app.tsx, gatewayClient.ts + app/components/hooks/lib +├── tui_gateway/ # Python JSON-RPC backend for the TUI ├── acp_adapter/ # ACP server (VS Code / Zed / JetBrains integration) -├── cron/ # Scheduler (jobs.py, scheduler.py) +├── cron/ # Scheduler — jobs.py, scheduler.py ├── environments/ # RL training environments (Atropos) -├── tests/ # Pytest suite (~3000 tests) -└── batch_runner.py # Parallel batch processing +├── scripts/ # run_tests.sh, release.py, auxiliary scripts +├── website/ # Docusaurus docs site +└── tests/ # Pytest suite (~15k tests across ~700 files as of Apr 2026) ``` -**User config:** `~/.hermes/config.yaml` (settings), `~/.hermes/.env` (API keys) +**User config:** `~/.hermes/config.yaml` (settings), `~/.hermes/.env` (API keys only). +**Logs:** `~/.hermes/logs/` — `agent.log` (INFO+), `errors.log` (WARNING+), +`gateway.log` when running the gateway. Profile-aware via `get_hermes_home()`. +Browse with `hermes logs [--follow] [--level ...] [--session ...]`. ## File Dependency Chain @@ -81,20 +77,30 @@ run_agent.py, cli.py, batch_runner.py, environments/ ## AIAgent Class (run_agent.py) +The real `AIAgent.__init__` takes ~60 parameters (credentials, routing, callbacks, +session context, budget, credential pool, etc.). The signature below is the +minimum subset you'll usually touch — read `run_agent.py` for the full list. + ```python class AIAgent: def __init__(self, - model: str = "anthropic/claude-opus-4.6", - max_iterations: int = 90, + base_url: str = None, + api_key: str = None, + provider: str = None, + api_mode: str = None, # "chat_completions" | "codex_responses" | ... + model: str = "", # empty → resolved from config/provider later + max_iterations: int = 90, # tool-calling iterations (shared with subagents) enabled_toolsets: list = None, disabled_toolsets: list = None, quiet_mode: bool = False, save_trajectories: bool = False, - platform: str = None, # "cli", "telegram", etc. + platform: str = None, # "cli", "telegram", etc. session_id: str = None, skip_context_files: bool = False, skip_memory: bool = False, - # ... plus provider, api_mode, callbacks, routing params + credential_pool=None, + # ... plus callbacks, thread/user/chat IDs, iteration_budget, fallback_model, + # checkpoints config, prefill_messages, service_tier, reasoning_config, etc. ): ... def chat(self, message: str) -> str: @@ -107,10 +113,13 @@ class AIAgent: ### Agent Loop -The core loop is inside `run_conversation()` — entirely synchronous: +The core loop is inside `run_conversation()` — entirely synchronous, with +interrupt checks, budget tracking, and a one-turn grace call: ```python -while api_call_count < self.max_iterations and self.iteration_budget.remaining > 0: +while (api_call_count < self.max_iterations and self.iteration_budget.remaining > 0) \ + or self._budget_grace_call: + if self._interrupt_requested: break response = client.chat.completions.create(model=model, messages=messages, tools=tool_schemas) if response.tool_calls: for tool_call in response.tool_calls: @@ -121,7 +130,8 @@ while api_call_count < self.max_iterations and self.iteration_budget.remaining > return response.content ``` -Messages follow OpenAI format: `{"role": "system/user/assistant/tool", ...}`. Reasoning content is stored in `assistant_msg["reasoning"]`. +Messages follow OpenAI format: `{"role": "system/user/assistant/tool", ...}`. +Reasoning content is stored in `assistant_msg["reasoning"]`. --- @@ -179,9 +189,62 @@ if canonical == "mycommand": --- +## TUI Architecture (ui-tui + tui_gateway) + +The TUI is a full replacement for the classic (prompt_toolkit) CLI, activated via `hermes --tui` or `HERMES_TUI=1`. + +### Process Model + +``` +hermes --tui + └─ Node (Ink) ──stdio JSON-RPC── Python (tui_gateway) + │ └─ AIAgent + tools + sessions + └─ renders transcript, composer, prompts, activity +``` + +TypeScript owns the screen. Python owns sessions, tools, model calls, and slash command logic. + +### Transport + +Newline-delimited JSON-RPC over stdio. Requests from Ink, events from Python. See `tui_gateway/server.py` for the full method/event catalog. + +### Key Surfaces + +| Surface | Ink component | Gateway method | +|---------|---------------|----------------| +| Chat streaming | `app.tsx` + `messageLine.tsx` | `prompt.submit` → `message.delta/complete` | +| Tool activity | `thinking.tsx` | `tool.start/progress/complete` | +| Approvals | `prompts.tsx` | `approval.respond` ← `approval.request` | +| Clarify/sudo/secret | `prompts.tsx`, `maskedPrompt.tsx` | `clarify/sudo/secret.respond` | +| Session picker | `sessionPicker.tsx` | `session.list/resume` | +| Slash commands | Local handler + fallthrough | `slash.exec` → `_SlashWorker`, `command.dispatch` | +| Completions | `useCompletion` hook | `complete.slash`, `complete.path` | +| Theming | `theme.ts` + `branding.tsx` | `gateway.ready` with skin data | + +### Slash Command Flow + +1. Built-in client commands (`/help`, `/quit`, `/clear`, `/resume`, `/copy`, `/paste`, etc.) handled locally in `app.tsx` +2. Everything else → `slash.exec` (runs in persistent `_SlashWorker` subprocess) → `command.dispatch` fallback + +### Dev Commands + +```bash +cd ui-tui +npm install # first time +npm run dev # watch mode (rebuilds hermes-ink + tsx --watch) +npm start # production +npm run build # full build (hermes-ink + tsc) +npm run type-check # typecheck only (tsc --noEmit) +npm run lint # eslint +npm run fmt # prettier +npm test # vitest +``` + +--- + ## Adding New Tools -Requires changes in **3 files**: +Requires changes in **2 files**: **1. Create `tools/your_tool.py`:** ```python @@ -204,9 +267,9 @@ registry.register( ) ``` -**2. Add import** in `model_tools.py` `_discover_tools()` list. +**2. Add to `toolsets.py`** — either `_HERMES_CORE_TOOLS` (all platforms) or a new toolset. -**3. Add to `toolsets.py`** — either `_HERMES_CORE_TOOLS` (all platforms) or a new toolset. +Auto-discovery: any `tools/*.py` file with a top-level `registry.register()` call is imported automatically — no manual import list to maintain. The registry handles schema collection, dispatch, availability checking, and error wrapping. All handlers MUST return a JSON string. @@ -214,7 +277,7 @@ The registry handles schema collection, dispatch, availability checking, and err **State files**: If a tool stores persistent state (caches, logs, checkpoints), use `get_hermes_home()` for the base directory — never `Path.home() / ".hermes"`. This ensures each profile gets its own state. -**Agent-level tools** (todo, memory): intercepted by `run_agent.py` before `handle_function_call()`. See `todo_tool.py` for the pattern. +**Agent-level tools** (todo, memory): intercepted by `run_agent.py` before `handle_function_call()`. See `tools/todo_tool.py` for the pattern. --- @@ -222,9 +285,13 @@ The registry handles schema collection, dispatch, availability checking, and err ### config.yaml options: 1. Add to `DEFAULT_CONFIG` in `hermes_cli/config.py` -2. Bump `_config_version` (currently 5) to trigger migration for existing users +2. Bump `_config_version` (check the current value at the top of `DEFAULT_CONFIG`) + ONLY if you need to actively migrate/transform existing user config + (renaming keys, changing structure). Adding a new key to an existing + section is handled automatically by the deep-merge and does NOT require + a version bump. -### .env variables: +### .env variables (SECRETS ONLY — API keys, tokens, passwords): 1. Add to `OPTIONAL_ENV_VARS` in `hermes_cli/config.py` with metadata: ```python "NEW_API_KEY": { @@ -236,13 +303,29 @@ The registry handles schema collection, dispatch, availability checking, and err }, ``` -### Config loaders (two separate systems): +Non-secret settings (timeouts, thresholds, feature flags, paths, display +preferences) belong in `config.yaml`, not `.env`. If internal code needs an +env var mirror for backward compatibility, bridge it from `config.yaml` to +the env var in code (see `gateway_timeout`, `terminal.cwd` → `TERMINAL_CWD`). + +### Config loaders (three paths — know which one you're in): | Loader | Used by | Location | |--------|---------|----------| -| `load_cli_config()` | CLI mode | `cli.py` | -| `load_config()` | `hermes tools`, `hermes setup` | `hermes_cli/config.py` | -| Direct YAML load | Gateway | `gateway/run.py` | +| `load_cli_config()` | CLI mode | `cli.py` — merges CLI-specific defaults + user YAML | +| `load_config()` | `hermes tools`, `hermes setup`, most CLI subcommands | `hermes_cli/config.py` — merges `DEFAULT_CONFIG` + user YAML | +| Direct YAML load | Gateway runtime | `gateway/run.py` + `gateway/config.py` — reads user YAML raw | + +If you add a new key and the CLI sees it but the gateway doesn't (or vice +versa), you're on the wrong loader. Check `DEFAULT_CONFIG` coverage. + +### Working directory: +- **CLI** — uses the process's current directory (`os.getcwd()`). +- **Messaging** — uses `terminal.cwd` from `config.yaml`. The gateway bridges this + to the `TERMINAL_CWD` env var for child tools. **`MESSAGING_CWD` has been + removed** — the config loader prints a deprecation warning if it's set in + `.env`. Same for `TERMINAL_CWD` in `.env`; the canonical setting is + `terminal.cwd` in `config.yaml`. --- @@ -335,7 +418,95 @@ Activate with `/skin cyberpunk` or `display.skin: cyberpunk` in config.yaml. --- +## Plugins + +Hermes has two plugin surfaces. Both live under `plugins/` in the repo so +repo-shipped plugins can be discovered alongside user-installed ones in +`~/.hermes/plugins/` and pip-installed entry points. + +### General plugins (`hermes_cli/plugins.py` + `plugins//`) + +`PluginManager` discovers plugins from `~/.hermes/plugins/`, `./.hermes/plugins/`, +and pip entry points. Each plugin exposes a `register(ctx)` function that +can: + +- Register Python-callback lifecycle hooks: + `pre_tool_call`, `post_tool_call`, `pre_llm_call`, `post_llm_call`, + `on_session_start`, `on_session_end` +- Register new tools via `ctx.register_tool(...)` +- Register CLI subcommands via `ctx.register_cli_command(...)` — the + plugin's argparse tree is wired into `hermes` at startup so + `hermes ` works with no change to `main.py` + +Hooks are invoked from `model_tools.py` (pre/post tool) and `run_agent.py` +(lifecycle). **Discovery timing pitfall:** `discover_plugins()` only runs +as a side effect of importing `model_tools.py`. Code paths that read plugin +state without importing `model_tools.py` first must call `discover_plugins()` +explicitly (it's idempotent). + +### Memory-provider plugins (`plugins/memory//`) + +Separate discovery system for pluggable memory backends. Current built-in +providers include **honcho, mem0, supermemory, byterover, hindsight, +holographic, openviking, retaindb**. + +Each provider implements the `MemoryProvider` ABC (see `agent/memory_provider.py`) +and is orchestrated by `agent/memory_manager.py`. Lifecycle hooks include +`sync_turn(turn_messages)`, `prefetch(query)`, `shutdown()`, and optional +`post_setup(hermes_home, config)` for setup-wizard integration. + +**CLI commands via `plugins/memory//cli.py`:** if a memory plugin +defines `register_cli(subparser)`, `discover_plugin_cli_commands()` finds +it at argparse setup time and wires it into `hermes `. The +framework only exposes CLI commands for the **currently active** memory +provider (read from `memory.provider` in config.yaml), so disabled +providers don't clutter `hermes --help`. + +**Rule (Teknium, May 2026):** plugins MUST NOT modify core files +(`run_agent.py`, `cli.py`, `gateway/run.py`, `hermes_cli/main.py`, etc.). +If a plugin needs a capability the framework doesn't expose, expand the +generic plugin surface (new hook, new ctx method) — never hardcode +plugin-specific logic into core. PR #5295 removed 95 lines of hardcoded +honcho argparse from `main.py` for exactly this reason. + +### Dashboard / context-engine / image-gen plugin directories + +`plugins/context_engine/`, `plugins/image_gen/`, `plugins/example-dashboard/`, +etc. follow the same pattern (ABC + orchestrator + per-plugin directory). +Context engines plug into `agent/context_engine.py`; image-gen providers +into `agent/image_gen_provider.py`. + +--- + +## Skills + +Two parallel surfaces: + +- **`skills/`** — built-in skills shipped and loadable by default. + Organized by category directories (e.g. `skills/github/`, `skills/mlops/`). +- **`optional-skills/`** — heavier or niche skills shipped with the repo but + NOT active by default. Installed explicitly via + `hermes skills install official//`. Adapter lives in + `tools/skills_hub.py` (`OptionalSkillSource`). Categories include + `autonomous-ai-agents`, `blockchain`, `communication`, `creative`, + `devops`, `email`, `health`, `mcp`, `migration`, `mlops`, `productivity`, + `research`, `security`, `web-development`. + +When reviewing skill PRs, check which directory they target — heavy-dep or +niche skills belong in `optional-skills/`. + +### SKILL.md frontmatter + +Standard fields: `name`, `description`, `version`, `platforms` +(OS-gating list: `[macos]`, `[linux, macos]`, ...), +`metadata.hermes.tags`, `metadata.hermes.category`, +`metadata.hermes.config` (config.yaml settings the skill needs — stored +under `skills.config.`, prompted during setup, injected at load time). + +--- + ## Important Policies + ### Prompt Caching Must Not Break Hermes-Agent ensures caching remains valid throughout a conversation. **Do NOT implement changes that would:** @@ -345,9 +516,10 @@ Hermes-Agent ensures caching remains valid throughout a conversation. **Do NOT i Cache-breaking forces dramatically higher costs. The ONLY time we alter context is during context compression. -### Working Directory Behavior -- **CLI**: Uses current directory (`.` → `os.getcwd()`) -- **Messaging**: Uses `MESSAGING_CWD` env var (default: home directory) +Slash commands that mutate system-prompt state (skills, tools, memory, etc.) +must be **cache-aware**: default to deferred invalidation (change takes +effect next session), with an opt-in `--now` flag for immediate +invalidation. See `/skills install --now` for the canonical pattern. ### Background Process Notifications (Gateway) @@ -369,7 +541,7 @@ Hermes supports **profiles** — multiple fully isolated instances, each with it `HERMES_HOME` directory (config, API keys, memory, sessions, skills, gateway, etc.). The core mechanism: `_apply_profile_override()` in `hermes_cli/main.py` sets -`HERMES_HOME` before any module imports. All 119+ references to `get_hermes_home()` +`HERMES_HOME` before any module imports. All `get_hermes_home()` references automatically scope to the active profile. ### Rules for profile-safe code @@ -426,8 +598,12 @@ Use `get_hermes_home()` from `hermes_constants` for code paths. Use `display_her for user-facing print/log messages. Hardcoding `~/.hermes` breaks profiles — each profile has its own `HERMES_HOME` directory. This was the source of 5 bugs fixed in PR #3575. -### DO NOT use `simple_term_menu` for interactive menus -Rendering bugs in tmux/iTerm2 — ghosting on scroll. Use `curses` (stdlib) instead. See `hermes_cli/tools_config.py` for the pattern. +### DO NOT introduce new `simple_term_menu` usage +Existing call sites in `hermes_cli/main.py` remain for legacy fallback only; +the preferred UI is curses (stdlib) because `simple_term_menu` has +ghost-duplication rendering bugs in tmux/iTerm2 with arrow keys. New +interactive menus must use `hermes_cli/curses_ui.py` — see +`hermes_cli/tools_config.py` for the canonical pattern. ### DO NOT use `\033[K` (ANSI erase-to-EOL) in spinner/display code Leaks as literal `?[K` text under `prompt_toolkit`'s `patch_stdout`. Use space-padding: `f"\r{line}{' ' * pad}"`. @@ -438,6 +614,30 @@ Leaks as literal `?[K` text under `prompt_toolkit`'s `patch_stdout`. Use space-p ### DO NOT hardcode cross-tool references in schema descriptions Tool schema descriptions must not mention tools from other toolsets by name (e.g., `browser_navigate` saying "prefer web_search"). Those tools may be unavailable (missing API keys, disabled toolset), causing the model to hallucinate calls to non-existent tools. If a cross-reference is needed, add it dynamically in `get_tool_definitions()` in `model_tools.py` — see the `browser_navigate` / `execute_code` post-processing blocks for the pattern. +### The gateway has TWO message guards — both must bypass approval/control commands +When an agent is running, messages pass through two sequential guards: +(1) **base adapter** (`gateway/platforms/base.py`) queues messages in +`_pending_messages` when `session_key in self._active_sessions`, and +(2) **gateway runner** (`gateway/run.py`) intercepts `/stop`, `/new`, +`/queue`, `/status`, `/approve`, `/deny` before they reach +`running_agent.interrupt()`. Any new command that must reach the runner +while the agent is blocked (e.g. approval prompts) MUST bypass BOTH +guards and be dispatched inline, not via `_process_message_background()` +(which races session lifecycle). + +### Squash merges from stale branches silently revert recent fixes +Before squash-merging a PR, ensure the branch is up to date with `main` +(`git fetch origin main && git reset --hard origin/main` in the worktree, +then re-apply the PR's commits). A stale branch's version of an unrelated +file will silently overwrite recent fixes on main when squashed. Verify +with `git diff HEAD~1..HEAD` after merging — unexpected deletions are a +red flag. + +### Don't wire in dead code without E2E validation +Unused code that was never shipped was dead for a reason. Before wiring an +unused module into a live code path, E2E test the real resolution chain +with actual imports (not mocks) against a temp `HERMES_HOME`. + ### Tests must not write to `~/.hermes/` The `_isolate_hermes_home` autouse fixture in `tests/conftest.py` redirects `HERMES_HOME` to a temp dir. Never hardcode `~/.hermes/` paths in tests. @@ -458,13 +658,94 @@ def profile_env(tmp_path, monkeypatch): ## Testing +**ALWAYS use `scripts/run_tests.sh`** — do not call `pytest` directly. The script enforces +hermetic environment parity with CI (unset credential vars, TZ=UTC, LANG=C.UTF-8, +4 xdist workers matching GHA ubuntu-latest). Direct `pytest` on a 16+ core +developer machine with API keys set diverges from CI in ways that have caused +multiple "works locally, fails in CI" incidents (and the reverse). + ```bash -source venv/bin/activate -python -m pytest tests/ -q # Full suite (~3000 tests, ~3 min) -python -m pytest tests/test_model_tools.py -q # Toolset resolution -python -m pytest tests/test_cli_init.py -q # CLI config loading -python -m pytest tests/gateway/ -q # Gateway tests -python -m pytest tests/tools/ -q # Tool-level tests +scripts/run_tests.sh # full suite, CI-parity +scripts/run_tests.sh tests/gateway/ # one directory +scripts/run_tests.sh tests/agent/test_foo.py::test_x # one test +scripts/run_tests.sh -v --tb=long # pass-through pytest flags ``` +### Why the wrapper (and why the old "just call pytest" doesn't work) + +Five real sources of local-vs-CI drift the script closes: + +| | Without wrapper | With wrapper | +|---|---|---| +| Provider API keys | Whatever is in your env (auto-detects pool) | All `*_API_KEY`/`*_TOKEN`/etc. unset | +| HOME / `~/.hermes/` | Your real config+auth.json | Temp dir per test | +| Timezone | Local TZ (PDT etc.) | UTC | +| Locale | Whatever is set | C.UTF-8 | +| xdist workers | `-n auto` = all cores (20+ on a workstation) | `-n 4` matching CI | + +`tests/conftest.py` also enforces points 1-4 as an autouse fixture so ANY pytest +invocation (including IDE integrations) gets hermetic behavior — but the wrapper +is belt-and-suspenders. + +### Running without the wrapper (only if you must) + +If you can't use the wrapper (e.g. on Windows or inside an IDE that shells +pytest directly), at minimum activate the venv and pass `-n 4`: + +```bash +source .venv/bin/activate # or: source venv/bin/activate +python -m pytest tests/ -q -n 4 +``` + +Worker count above 4 will surface test-ordering flakes that CI never sees. + Always run the full suite before pushing changes. + +### Don't write change-detector tests + +A test is a **change-detector** if it fails whenever data that is **expected +to change** gets updated — model catalogs, config version numbers, +enumeration counts, hardcoded lists of provider models. These tests add no +behavioral coverage; they just guarantee that routine source updates break +CI and cost engineering time to "fix." + +**Do not write:** + +```python +# catalog snapshot — breaks every model release +assert "gemini-2.5-pro" in _PROVIDER_MODELS["gemini"] +assert "MiniMax-M2.7" in models + +# config version literal — breaks every schema bump +assert DEFAULT_CONFIG["_config_version"] == 21 + +# enumeration count — breaks every time a skill/provider is added +assert len(_PROVIDER_MODELS["huggingface"]) == 8 +``` + +**Do write:** + +```python +# behavior: does the catalog plumbing work at all? +assert "gemini" in _PROVIDER_MODELS +assert len(_PROVIDER_MODELS["gemini"]) >= 1 + +# behavior: does migration bump the user's version to current latest? +assert raw["_config_version"] == DEFAULT_CONFIG["_config_version"] + +# invariant: no plan-only model leaks into the legacy list +assert not (set(moonshot_models) & coding_plan_only_models) + +# invariant: every model in the catalog has a context-length entry +for m in _PROVIDER_MODELS["huggingface"]: + assert m.lower() in DEFAULT_CONTEXT_LENGTHS_LOWER +``` + +The rule: if the test reads like a snapshot of current data, delete it. If +it reads like a contract about how two pieces of data must relate, keep it. +When a PR adds a new provider/model and you want a test, make the test +assert the relationship (e.g. "catalog entries all have context lengths"), +not the specific names. + +Reviewers should reject new change-detector tests; authors should convert +them into invariants before re-requesting review. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4577454e441c..146cb1161bde 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,7 +9,7 @@ Thank you for contributing to Hermes Agent! This guide covers everything you nee We value contributions in this order: 1. **Bug fixes** — crashes, incorrect behavior, data loss. Always top priority. -2. **Cross-platform compatibility** — Windows, macOS, different Linux distros, different terminal emulators. We want Hermes to work everywhere. +2. **Cross-platform compatibility** — macOS, different Linux distros, and WSL2 on Windows. We want Hermes to work everywhere. 3. **Security hardening** — shell injection, prompt injection, path traversal, privilege escalation. See [Security](#security-considerations). 4. **Performance and robustness** — retry logic, error handling, graceful degradation. 5. **New skills** — but only broadly useful ones. See [Should it be a Skill or a Tool?](#should-it-be-a-skill-or-a-tool) @@ -55,10 +55,10 @@ If your skill is specialized, community-contributed, or niche, it's better suite | Requirement | Notes | |-------------|-------| -| **Git** | With `--recurse-submodules` support | +| **Git** | With `--recurse-submodules` support, and the `git-lfs` extension installed | | **Python 3.11+** | uv will install it if missing | | **uv** | Fast Python package manager ([install](https://docs.astral.sh/uv/)) | -| **Node.js 18+** | Optional — needed for browser tools and WhatsApp bridge | +| **Node.js 20+** | Optional — needed for browser tools and WhatsApp bridge (matches root `package.json` engines) | ### Clone and install @@ -88,7 +88,7 @@ cp cli-config.yaml.example ~/.hermes/config.yaml touch ~/.hermes/.env # Add at minimum an LLM provider key: -echo 'OPENROUTER_API_KEY=sk-or-v1-your-key' >> ~/.hermes/.env +echo "OPENROUTER_API_KEY=***" >> ~/.hermes/.env ``` ### Run @@ -515,7 +515,7 @@ See `hermes_cli/skin_engine.py` for the full schema and existing skins as exampl ## Cross-Platform Compatibility -Hermes runs on Linux, macOS, and Windows. When writing code that touches the OS: +Hermes runs on Linux, macOS, and WSL2 on Windows. When writing code that touches the OS: ### Critical rules @@ -597,7 +597,7 @@ refactor/description # Code restructuring 1. **Run tests**: `pytest tests/ -v` 2. **Test manually**: Run `hermes` and exercise the code path you changed -3. **Check cross-platform impact**: If you touch file I/O, process management, or terminal handling, consider Windows and macOS +3. **Check cross-platform impact**: If you touch file I/O, process management, or terminal handling, consider macOS, Linux, and WSL2 4. **Keep PRs focused**: One logical change per PR. Don't mix a bug fix with a refactor with a new feature. ### PR description diff --git a/Dockerfile b/Dockerfile index 4935d222ae99..8904c4c74def 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,7 +12,7 @@ ENV PLAYWRIGHT_BROWSERS_PATH=/opt/hermes/.playwright # Install system dependencies in one layer, clear APT cache RUN apt-get update && \ apt-get install -y --no-install-recommends \ - build-essential nodejs npm python3 ripgrep ffmpeg gcc python3-dev libffi-dev procps && \ + build-essential nodejs npm python3 ripgrep ffmpeg gcc python3-dev libffi-dev procps git openssh-client docker-cli && \ rm -rf /var/lib/apt/lists/* # Non-root user for runtime; UID can be overridden via HERMES_UID at runtime @@ -21,26 +21,35 @@ RUN useradd -u 10000 -m -d /opt/data hermes COPY --chmod=0755 --from=gosu_source /gosu /usr/local/bin/ COPY --chmod=0755 --from=uv_source /usr/local/bin/uv /usr/local/bin/uvx /usr/local/bin/ -COPY . /opt/hermes WORKDIR /opt/hermes -# Install Node dependencies and Playwright as root (--with-deps needs apt) +# ---------- Layer-cached dependency install ---------- +# Copy only package manifests first so npm install + Playwright are cached +# unless the lockfiles themselves change. +COPY package.json package-lock.json ./ +COPY web/package.json web/package-lock.json web/ + RUN npm install --prefer-offline --no-audit && \ npx playwright install --with-deps chromium --only-shell && \ - cd /opt/hermes/scripts/whatsapp-bridge && \ - npm install --prefer-offline --no-audit && \ + (cd web && npm install --prefer-offline --no-audit) && \ npm cache clean --force -# Hand ownership to hermes user, then install Python deps in a virtualenv -RUN chown -R hermes:hermes /opt/hermes -USER hermes +# ---------- Source code ---------- +# .dockerignore excludes node_modules, so the installs above survive. +COPY --chown=hermes:hermes . . + +# Build web dashboard (Vite outputs to hermes_cli/web_dist/) +RUN cd web && npm run build +# ---------- Python virtualenv ---------- +RUN chown hermes:hermes /opt/hermes +USER hermes RUN uv venv && \ uv pip install --no-cache-dir -e ".[all]" -USER root -RUN chmod +x /opt/hermes/docker/entrypoint.sh - +# ---------- Runtime ---------- +ENV HERMES_WEB_DIST=/opt/hermes/hermes_cli/web_dist ENV HERMES_HOME=/opt/data +ENV PATH="/opt/data/.local/bin:${PATH}" VOLUME [ "/opt/data" ] ENTRYPOINT [ "/opt/hermes/docker/entrypoint.sh" ] diff --git a/RELEASE_v0.10.0.md b/RELEASE_v0.10.0.md new file mode 100644 index 000000000000..1bfb1015685c --- /dev/null +++ b/RELEASE_v0.10.0.md @@ -0,0 +1,27 @@ +# Hermes Agent v0.10.0 (v2026.4.16) + +**Release Date:** April 16, 2026 + +> The Tool Gateway release — paid Nous Portal subscribers can now use web search, image generation, text-to-speech, and browser automation through their existing subscription with zero additional API keys. + +--- + +## ✨ Highlights + +- **Nous Tool Gateway** — Paid [Nous Portal](https://portal.nousresearch.com) subscribers now get automatic access to **web search** (Firecrawl), **image generation** (FAL / FLUX 2 Pro), **text-to-speech** (OpenAI TTS), and **browser automation** (Browser Use) through their existing subscription. No separate API keys needed — just run `hermes model`, select Nous Portal, and pick which tools to enable. Per-tool opt-in via `use_gateway` config, full integration with `hermes tools` and `hermes status`, and the runtime correctly prefers the gateway even when direct API keys exist. Replaces the old hidden `HERMES_ENABLE_NOUS_MANAGED_TOOLS` env var with clean subscription-based detection. ([#11206](https://github.com/NousResearch/hermes-agent/pull/11206), based on work by @jquesnelle; docs: [#11208](https://github.com/NousResearch/hermes-agent/pull/11208)) + +--- + +## 🐛 Bug Fixes & Improvements + +This release includes 180+ commits with numerous bug fixes, platform improvements, and reliability enhancements across the agent core, gateway, CLI, and tool system. Full details will be published in the v0.11.0 changelog. + +--- + +## 👥 Contributors + +- **@jquesnelle** (emozilla) — Original Tool Gateway implementation ([#10799](https://github.com/NousResearch/hermes-agent/pull/10799)), salvaged and shipped in this release + +--- + +**Full Changelog**: [v2026.4.13...v2026.4.16](https://github.com/NousResearch/hermes-agent/compare/v2026.4.13...v2026.4.16) diff --git a/RELEASE_v0.11.0.md b/RELEASE_v0.11.0.md new file mode 100644 index 000000000000..ed25f5a14dc5 --- /dev/null +++ b/RELEASE_v0.11.0.md @@ -0,0 +1,453 @@ +# Hermes Agent v0.11.0 (v2026.4.23) + +**Release Date:** April 23, 2026 +**Since v0.9.0:** 1,556 commits · 761 merged PRs · 1,314 files changed · 224,174 insertions · 29 community contributors (290 including co-authors) + +> The Interface release — a full React/Ink rewrite of the interactive CLI, a pluggable transport architecture underneath every provider, native AWS Bedrock support, five new inference paths, a 17th messaging platform (QQBot), a dramatically expanded plugin surface, and GPT-5.5 via Codex OAuth. + +This release also folds in all the highlights deferred from v0.10.0 (which shipped only the Nous Tool Gateway) — so it covers roughly two weeks of work across the whole stack. + +--- + +## ✨ Highlights + +- **New Ink-based TUI** — `hermes --tui` is now a full React/Ink rewrite of the interactive CLI, with a Python JSON-RPC backend (`tui_gateway`). Sticky composer, live streaming with OSC-52 clipboard support, stable picker keys, status bar with per-turn stopwatch and git branch, `/clear` confirm, light-theme preset, and a subagent spawn observability overlay. ~310 commits to `ui-tui/` + `tui_gateway/`. (@OutThisLife + Teknium) + +- **Transport ABC + Native AWS Bedrock** — Format conversion and HTTP transport were extracted from `run_agent.py` into a pluggable `agent/transports/` layer. `AnthropicTransport`, `ChatCompletionsTransport`, `ResponsesApiTransport`, and `BedrockTransport` each own their own format conversion and API shape. Native AWS Bedrock support via the Converse API ships on top of the new abstraction. ([#10549](https://github.com/NousResearch/hermes-agent/pull/10549), [#13347](https://github.com/NousResearch/hermes-agent/pull/13347), [#13366](https://github.com/NousResearch/hermes-agent/pull/13366), [#13430](https://github.com/NousResearch/hermes-agent/pull/13430), [#13805](https://github.com/NousResearch/hermes-agent/pull/13805), [#13814](https://github.com/NousResearch/hermes-agent/pull/13814) — @kshitijk4poor + Teknium) + +- **Five new inference paths** — Native NVIDIA NIM ([#11774](https://github.com/NousResearch/hermes-agent/pull/11774)), Arcee AI ([#9276](https://github.com/NousResearch/hermes-agent/pull/9276)), Step Plan ([#13893](https://github.com/NousResearch/hermes-agent/pull/13893)), Google Gemini CLI OAuth ([#11270](https://github.com/NousResearch/hermes-agent/pull/11270)), and Vercel ai-gateway with pricing + dynamic discovery ([#13223](https://github.com/NousResearch/hermes-agent/pull/13223) — @jerilynzheng). Plus Gemini routed through the native AI Studio API for better performance ([#12674](https://github.com/NousResearch/hermes-agent/pull/12674)). + +- **GPT-5.5 over Codex OAuth** — OpenAI's new GPT-5.5 reasoning model is now available through your ChatGPT Codex OAuth, with live model discovery wired into the model picker so new OpenAI releases show up without catalog updates. ([#14720](https://github.com/NousResearch/hermes-agent/pull/14720)) + +- **QQBot — 17th supported platform** — Native QQBot adapter via QQ Official API v2, with QR scan-to-configure setup wizard, streaming cursor, emoji reactions, and DM/group policy gating that matches WeCom/Weixin parity. ([#9364](https://github.com/NousResearch/hermes-agent/pull/9364), [#11831](https://github.com/NousResearch/hermes-agent/pull/11831)) + +- **Plugin surface expanded** — Plugins can now register slash commands (`register_command`), dispatch tools directly (`dispatch_tool`), block tool execution from hooks (`pre_tool_call` can veto), rewrite tool results (`transform_tool_result`), transform terminal output (`transform_terminal_output`), ship image_gen backends, and add custom dashboard tabs. The bundled disk-cleanup plugin is opt-in by default as a reference implementation. ([#9377](https://github.com/NousResearch/hermes-agent/pull/9377), [#10626](https://github.com/NousResearch/hermes-agent/pull/10626), [#10763](https://github.com/NousResearch/hermes-agent/pull/10763), [#10951](https://github.com/NousResearch/hermes-agent/pull/10951), [#12929](https://github.com/NousResearch/hermes-agent/pull/12929), [#12944](https://github.com/NousResearch/hermes-agent/pull/12944), [#12972](https://github.com/NousResearch/hermes-agent/pull/12972), [#13799](https://github.com/NousResearch/hermes-agent/pull/13799), [#14175](https://github.com/NousResearch/hermes-agent/pull/14175)) + +- **`/steer` — mid-run agent nudges** — `/steer ` injects a note that the running agent sees after its next tool call, without interrupting the turn or breaking prompt cache. For when you want to course-correct an agent in-flight. ([#12116](https://github.com/NousResearch/hermes-agent/pull/12116)) + +- **Shell hooks** — Wire any shell script as a Hermes lifecycle hook (pre_tool_call, post_tool_call, on_session_start, etc.) without writing a Python plugin. ([#13296](https://github.com/NousResearch/hermes-agent/pull/13296)) + +- **Webhook direct-delivery mode** — Webhook subscriptions can now forward payloads straight to a platform chat without going through the agent — zero-LLM push notifications for alerting, uptime checks, and event streams. ([#12473](https://github.com/NousResearch/hermes-agent/pull/12473)) + +- **Smarter delegation** — Subagents now have an explicit `orchestrator` role that can spawn their own workers, with configurable `max_spawn_depth` (default flat). Concurrent sibling subagents share filesystem state through a file-coordination layer so they don't clobber each other's edits. ([#13691](https://github.com/NousResearch/hermes-agent/pull/13691), [#13718](https://github.com/NousResearch/hermes-agent/pull/13718)) + +- **Auxiliary models — configurable UI + main-model-first** — `hermes model` has a dedicated "Configure auxiliary models" screen for per-task overrides (compression, vision, session_search, title_generation). `auto` routing now defaults to the main model for side tasks across all users (previously aggregator users were silently routed to a cheap provider-side default). ([#11891](https://github.com/NousResearch/hermes-agent/pull/11891), [#11900](https://github.com/NousResearch/hermes-agent/pull/11900)) + +- **Dashboard plugin system + live theme switching** — The web dashboard is now extensible. Third-party plugins can add custom tabs, widgets, and views without forking. Paired with a live-switching theme system — themes now control colors, fonts, layout, and density — so users can hot-swap the dashboard look without a reload. Same theming discipline the CLI has, now on the web. ([#10951](https://github.com/NousResearch/hermes-agent/pull/10951), [#10687](https://github.com/NousResearch/hermes-agent/pull/10687), [#14725](https://github.com/NousResearch/hermes-agent/pull/14725)) + +- **Dashboard polish** — i18n (English + Chinese), react-router sidebar layout, mobile-responsive, Vercel deployment, real per-session API call tracking, and one-click update + gateway restart buttons. ([#9228](https://github.com/NousResearch/hermes-agent/pull/9228), [#9370](https://github.com/NousResearch/hermes-agent/pull/9370), [#9453](https://github.com/NousResearch/hermes-agent/pull/9453), [#10686](https://github.com/NousResearch/hermes-agent/pull/10686), [#13526](https://github.com/NousResearch/hermes-agent/pull/13526), [#14004](https://github.com/NousResearch/hermes-agent/pull/14004) — @austinpickett + @DeployFaith + Teknium) + +--- + +## 🏗️ Core Agent & Architecture + +### Transport Layer (NEW) +- **Transport ABC** abstracts format conversion and HTTP transport from `run_agent.py` into `agent/transports/` ([#13347](https://github.com/NousResearch/hermes-agent/pull/13347)) +- **AnthropicTransport** — Anthropic Messages API path ([#13366](https://github.com/NousResearch/hermes-agent/pull/13366), @kshitijk4poor) +- **ChatCompletionsTransport** — default path for OpenAI-compatible providers ([#13805](https://github.com/NousResearch/hermes-agent/pull/13805)) +- **ResponsesApiTransport** — OpenAI Responses API + Codex build_kwargs wiring ([#13430](https://github.com/NousResearch/hermes-agent/pull/13430), @kshitijk4poor) +- **BedrockTransport** — AWS Bedrock Converse API transport ([#13814](https://github.com/NousResearch/hermes-agent/pull/13814)) + +### Provider & Model Support +- **Native AWS Bedrock provider** via Converse API ([#10549](https://github.com/NousResearch/hermes-agent/pull/10549)) +- **NVIDIA NIM native provider** (salvage of #11703) ([#11774](https://github.com/NousResearch/hermes-agent/pull/11774)) +- **Arcee AI direct provider** ([#9276](https://github.com/NousResearch/hermes-agent/pull/9276)) +- **Step Plan provider** (salvage #6005) ([#13893](https://github.com/NousResearch/hermes-agent/pull/13893), @kshitijk4poor) +- **Google Gemini CLI OAuth** inference provider ([#11270](https://github.com/NousResearch/hermes-agent/pull/11270)) +- **Vercel ai-gateway** with pricing, attribution, and dynamic discovery ([#13223](https://github.com/NousResearch/hermes-agent/pull/13223), @jerilynzheng) +- **GPT-5.5 over Codex OAuth** with live model discovery in the picker ([#14720](https://github.com/NousResearch/hermes-agent/pull/14720)) +- **Gemini routed through native AI Studio API** ([#12674](https://github.com/NousResearch/hermes-agent/pull/12674)) +- **xAI Grok upgraded to Responses API** ([#10783](https://github.com/NousResearch/hermes-agent/pull/10783)) +- **Ollama improvements** — Cloud provider support, GLM continuation, `think=false` control, surrogate sanitization, `/v1` hint ([#10782](https://github.com/NousResearch/hermes-agent/pull/10782)) +- **Kimi K2.6** across OpenRouter, Nous Portal, native Kimi, and HuggingFace ([#13148](https://github.com/NousResearch/hermes-agent/pull/13148), [#13152](https://github.com/NousResearch/hermes-agent/pull/13152), [#13169](https://github.com/NousResearch/hermes-agent/pull/13169)) +- **Kimi K2.5** promoted to first position in all model suggestion lists ([#11745](https://github.com/NousResearch/hermes-agent/pull/11745), @kshitijk4poor) +- **Xiaomi MiMo v2.5-pro + v2.5** on OpenRouter, Nous Portal, and native ([#14184](https://github.com/NousResearch/hermes-agent/pull/14184), [#14635](https://github.com/NousResearch/hermes-agent/pull/14635), @kshitijk4poor) +- **GLM-5V-Turbo** for coding plan ([#9907](https://github.com/NousResearch/hermes-agent/pull/9907)) +- **Claude Opus 4.7** in Nous Portal catalog ([#11398](https://github.com/NousResearch/hermes-agent/pull/11398)) +- **OpenRouter elephant-alpha** in curated lists ([#9378](https://github.com/NousResearch/hermes-agent/pull/9378)) +- **OpenCode-Go** — Kimi K2.6 and Qwen3.5/3.6 Plus in curated catalog ([#13429](https://github.com/NousResearch/hermes-agent/pull/13429)) +- **minimax/minimax-m2.5:free** in OpenRouter catalog ([#13836](https://github.com/NousResearch/hermes-agent/pull/13836)) +- **`/model` merges models.dev entries** for lesser-loved providers ([#14221](https://github.com/NousResearch/hermes-agent/pull/14221)) +- **Per-provider + per-model `request_timeout_seconds`** config ([#12652](https://github.com/NousResearch/hermes-agent/pull/12652)) +- **Configurable API retry count** via `agent.api_max_retries` ([#14730](https://github.com/NousResearch/hermes-agent/pull/14730)) +- **ctx_size context length key** for Lemonade server (salvage #8536) ([#14215](https://github.com/NousResearch/hermes-agent/pull/14215)) +- **Custom provider display name prompt** ([#9420](https://github.com/NousResearch/hermes-agent/pull/9420)) +- **Recommendation badges** on tool provider selection ([#9929](https://github.com/NousResearch/hermes-agent/pull/9929)) +- Fix: correct GPT-5 family context lengths in fallback defaults ([#9309](https://github.com/NousResearch/hermes-agent/pull/9309)) +- Fix: clamp `minimal` reasoning effort to `low` on Responses API ([#9429](https://github.com/NousResearch/hermes-agent/pull/9429)) +- Fix: strip reasoning item IDs from Responses API input when `store=False` ([#10217](https://github.com/NousResearch/hermes-agent/pull/10217)) +- Fix: OpenViking correct account default + commit session on `/new` and compress ([#10463](https://github.com/NousResearch/hermes-agent/pull/10463)) +- Fix: Kimi `/coding` thinking block survival + empty reasoning_content + block ordering (multiple PRs) +- Fix: don't send Anthropic thinking to api.kimi.com/coding ([#13826](https://github.com/NousResearch/hermes-agent/pull/13826)) +- Fix: send `max_tokens`, `reasoning_effort`, and `thinking` for Kimi/Moonshot +- Fix: stream reasoning content through OpenAI-compatible providers that emit it + +### Agent Loop & Conversation +- **`/steer `** — mid-run agent nudges after next tool call ([#12116](https://github.com/NousResearch/hermes-agent/pull/12116)) +- **Orchestrator role + configurable spawn depth** for `delegate_task` (default flat) ([#13691](https://github.com/NousResearch/hermes-agent/pull/13691)) +- **Cross-agent file state coordination** for concurrent subagents ([#13718](https://github.com/NousResearch/hermes-agent/pull/13718)) +- **Compressor smart collapse, dedup, anti-thrashing**, template upgrade, hardening ([#10088](https://github.com/NousResearch/hermes-agent/pull/10088)) +- **Compression summaries respect the conversation's language** ([#12556](https://github.com/NousResearch/hermes-agent/pull/12556)) +- **Compression model falls back to main model** on permanent 503/404 ([#10093](https://github.com/NousResearch/hermes-agent/pull/10093)) +- **Auto-continue interrupted agent work** after gateway restart ([#9934](https://github.com/NousResearch/hermes-agent/pull/9934)) +- **Activity heartbeats** prevent false gateway inactivity timeouts ([#10501](https://github.com/NousResearch/hermes-agent/pull/10501)) +- **Auxiliary models UI** — dedicated screen for per-task overrides ([#11891](https://github.com/NousResearch/hermes-agent/pull/11891)) +- **Auxiliary auto routing defaults to main model** for all users ([#11900](https://github.com/NousResearch/hermes-agent/pull/11900)) +- **PLATFORM_HINTS for Matrix, Mattermost, Feishu** ([#14428](https://github.com/NousResearch/hermes-agent/pull/14428), @alt-glitch) +- Fix: reset retry counters after compression; stop poisoning conversation history ([#10055](https://github.com/NousResearch/hermes-agent/pull/10055)) +- Fix: break compression-exhaustion infinite loop and auto-reset session ([#10063](https://github.com/NousResearch/hermes-agent/pull/10063)) +- Fix: stale agent timeout, uv venv detection, empty response after tools ([#10065](https://github.com/NousResearch/hermes-agent/pull/10065)) +- Fix: prevent premature loop exit when weak models return empty after substantive tool calls ([#10472](https://github.com/NousResearch/hermes-agent/pull/10472)) +- Fix: preserve pre-start terminal interrupts ([#10504](https://github.com/NousResearch/hermes-agent/pull/10504)) +- Fix: improve interrupt responsiveness during concurrent tool execution ([#10935](https://github.com/NousResearch/hermes-agent/pull/10935)) +- Fix: word-wrap spinner, interruptable agent join, and delegate_task interrupt ([#10940](https://github.com/NousResearch/hermes-agent/pull/10940)) +- Fix: `/stop` no longer resets the session ([#9224](https://github.com/NousResearch/hermes-agent/pull/9224)) +- Fix: honor interrupts during MCP tool waits ([#9382](https://github.com/NousResearch/hermes-agent/pull/9382), @helix4u) +- Fix: break stuck session resume loops after repeated restarts ([#9941](https://github.com/NousResearch/hermes-agent/pull/9941)) +- Fix: empty response nudge crash + placeholder leak to cron targets ([#11021](https://github.com/NousResearch/hermes-agent/pull/11021)) +- Fix: streaming cursor sanitization to prevent message truncation (multiple PRs) +- Fix: resolve `context_length` for plugin context engines ([#9238](https://github.com/NousResearch/hermes-agent/pull/9238)) + +### Session & Memory +- **Auto-prune old sessions + VACUUM state.db** at startup ([#13861](https://github.com/NousResearch/hermes-agent/pull/13861)) +- **Honcho overhaul** — context injection, 5-tool surface, cost safety, session isolation ([#10619](https://github.com/NousResearch/hermes-agent/pull/10619)) +- **Hindsight richer session-scoped retain metadata** (salvage of #6290) ([#13987](https://github.com/NousResearch/hermes-agent/pull/13987)) +- Fix: deduplicate memory provider tools to prevent 400 on strict providers ([#10511](https://github.com/NousResearch/hermes-agent/pull/10511)) +- Fix: discover user-installed memory providers from `$HERMES_HOME/plugins/` ([#10529](https://github.com/NousResearch/hermes-agent/pull/10529)) +- Fix: add `on_memory_write` bridge to sequential tool execution path ([#10507](https://github.com/NousResearch/hermes-agent/pull/10507)) +- Fix: preserve `session_id` across `previous_response_id` chains in `/v1/responses` ([#10059](https://github.com/NousResearch/hermes-agent/pull/10059)) + +--- + +## 🖥️ New Ink-based TUI + +A full React/Ink rewrite of the interactive CLI — invoked via `hermes --tui` or `HERMES_TUI=1`. Shipped across ~310 commits to `ui-tui/` and `tui_gateway/`. + +### TUI Foundations +- New TUI based on Ink + Python JSON-RPC backend +- Prettier + ESLint + vitest tooling for `ui-tui/` +- Entry split between `src/entry.tsx` (TTY gate) and `src/app.tsx` (state machine) +- Persistent `_SlashWorker` subprocess for slash command dispatch + +### UX & Features +- **Stable picker keys, /clear confirm, light-theme preset** ([#12312](https://github.com/NousResearch/hermes-agent/pull/12312), @OutThisLife) +- **Git branch in status bar** cwd label ([#12305](https://github.com/NousResearch/hermes-agent/pull/12305), @OutThisLife) +- **Per-turn elapsed stopwatch in FaceTicker + done-in sys line** ([#13105](https://github.com/NousResearch/hermes-agent/pull/13105), @OutThisLife) +- **Subagent spawn observability overlay** ([#14045](https://github.com/NousResearch/hermes-agent/pull/14045), @OutThisLife) +- **Per-prompt elapsed stopwatch in status bar** ([#12948](https://github.com/NousResearch/hermes-agent/pull/12948)) +- Sticky composer that freezes during scroll +- OSC-52 clipboard support for copy across SSH sessions +- Virtualized history rendering for performance +- Slash command autocomplete via `complete.slash` RPC +- Path autocomplete via `complete.path` RPC +- Dozens of resize/ghosting/sticky-prompt fixes landed through the week + +### Structural Refactors +- Decomposed `app.tsx` into `app/event-handler`, `app/slash-handler`, `app/stores`, `app/hooks` ([#14640](https://github.com/NousResearch/hermes-agent/pull/14640) and surrounding) +- Component split: `branding.tsx`, `markdown.tsx`, `prompts.tsx`, `sessionPicker.tsx`, `messageLine.tsx`, `thinking.tsx`, `maskedPrompt.tsx` +- Hook split: `useCompletion`, `useInputHistory`, `useQueue`, `useVirtualHistory` + +--- + +## 📱 Messaging Platforms (Gateway) + +### New Platforms +- **QQBot (17th platform)** — QQ Official API v2 adapter with QR setup, streaming, package split ([#9364](https://github.com/NousResearch/hermes-agent/pull/9364), [#11831](https://github.com/NousResearch/hermes-agent/pull/11831)) + +### Telegram +- **Dedicated `TELEGRAM_PROXY` env var + config.yaml proxy support** (closes #9414, #6530, #9074, #7786) ([#10681](https://github.com/NousResearch/hermes-agent/pull/10681)) +- **`ignored_threads` config** for Telegram groups ([#9530](https://github.com/NousResearch/hermes-agent/pull/9530)) +- **Config option to disable link previews** (closes #8728) ([#10610](https://github.com/NousResearch/hermes-agent/pull/10610)) +- **Auto-wrap markdown tables** in code blocks ([#11794](https://github.com/NousResearch/hermes-agent/pull/11794)) +- Fix: prevent duplicate replies when stream task is cancelled ([#9319](https://github.com/NousResearch/hermes-agent/pull/9319)) +- Fix: prevent streaming cursor (▉) from appearing as standalone messages ([#9538](https://github.com/NousResearch/hermes-agent/pull/9538)) +- Fix: retry transient tool sends + cold-boot budget ([#10947](https://github.com/NousResearch/hermes-agent/pull/10947)) +- Fix: Markdown special char escaping in `send_exec_approval` +- Fix: parentheses in URLs during MarkdownV2 link conversion +- Fix: Unicode dash normalization in model switch (closes iOS smart-punctuation issue) +- Many platform hint / streaming / session-key fixes + +### Discord +- **Forum channel support** (salvage of #10145 + media + polish) ([#11920](https://github.com/NousResearch/hermes-agent/pull/11920)) +- **`DISCORD_ALLOWED_ROLES`** for role-based access control ([#11608](https://github.com/NousResearch/hermes-agent/pull/11608)) +- **Config option to disable slash commands** (salvage #13130) ([#14315](https://github.com/NousResearch/hermes-agent/pull/14315)) +- **Native `send_animation`** for inline GIF playback ([#10283](https://github.com/NousResearch/hermes-agent/pull/10283)) +- **`send_message` Discord media attachments** ([#10246](https://github.com/NousResearch/hermes-agent/pull/10246)) +- **`/skill` command group** with category subcommands ([#9909](https://github.com/NousResearch/hermes-agent/pull/9909)) +- **Extract reply text from message references** ([#9781](https://github.com/NousResearch/hermes-agent/pull/9781)) + +### Feishu +- **Intelligent reply on document comments** with 3-tier access control ([#11898](https://github.com/NousResearch/hermes-agent/pull/11898)) +- **Show processing state via reactions** on user messages ([#12927](https://github.com/NousResearch/hermes-agent/pull/12927)) +- **Preserve @mention context for agent consumption** (salvage #13874) ([#14167](https://github.com/NousResearch/hermes-agent/pull/14167)) + +### DingTalk +- **`require_mention` + `allowed_users` gating** (parity with Slack/Telegram/Discord) ([#11564](https://github.com/NousResearch/hermes-agent/pull/11564)) +- **QR-code device-flow authorization** for setup wizard ([#11574](https://github.com/NousResearch/hermes-agent/pull/11574)) +- **AI Cards streaming, emoji reactions, and media handling** (salvage of #10985) ([#11910](https://github.com/NousResearch/hermes-agent/pull/11910)) + +### WhatsApp +- **`send_voice`** — native audio message delivery ([#13002](https://github.com/NousResearch/hermes-agent/pull/13002)) +- **`dm_policy` and `group_policy`** parity with WeCom/Weixin/QQ adapters ([#13151](https://github.com/NousResearch/hermes-agent/pull/13151)) + +### WeCom / Weixin +- **WeCom QR-scan bot creation + interactive setup wizard** (salvage #13923) ([#13961](https://github.com/NousResearch/hermes-agent/pull/13961)) + +### Signal +- **Media delivery support** via `send_message` ([#13178](https://github.com/NousResearch/hermes-agent/pull/13178)) + +### Slack +- **Per-thread sessions for DMs by default** ([#10987](https://github.com/NousResearch/hermes-agent/pull/10987)) + +### BlueBubbles (iMessage) +- Group chat session separation, webhook registration & auth fixes ([#9806](https://github.com/NousResearch/hermes-agent/pull/9806)) + +### Gateway Core +- **Gateway proxy mode** — forward messages to a remote API server ([#9787](https://github.com/NousResearch/hermes-agent/pull/9787)) +- **Per-channel ephemeral prompts** (Discord, Telegram, Slack, Mattermost) ([#10564](https://github.com/NousResearch/hermes-agent/pull/10564)) +- **Surface plugin slash commands** natively on all platforms + decision-capable command hook ([#14175](https://github.com/NousResearch/hermes-agent/pull/14175)) +- **Support document/archive extensions in MEDIA: tag extraction** (salvage #8255) ([#14307](https://github.com/NousResearch/hermes-agent/pull/14307)) +- **Recognize `.pdf` in MEDIA: tag extraction** ([#13683](https://github.com/NousResearch/hermes-agent/pull/13683)) +- **`--all` flag for `gateway start` and `restart`** ([#10043](https://github.com/NousResearch/hermes-agent/pull/10043)) +- **Notify active sessions on gateway shutdown** + update health check ([#9850](https://github.com/NousResearch/hermes-agent/pull/9850)) +- **Block agent from self-destructing the gateway** via terminal (closes #6666) ([#9895](https://github.com/NousResearch/hermes-agent/pull/9895)) +- Fix: suppress duplicate replies on interrupt and streaming flood control ([#10235](https://github.com/NousResearch/hermes-agent/pull/10235)) +- Fix: close temporary agents after one-off tasks ([#11028](https://github.com/NousResearch/hermes-agent/pull/11028), @kshitijk4poor) +- Fix: busy-session ack when user messages during active agent run ([#10068](https://github.com/NousResearch/hermes-agent/pull/10068)) +- Fix: route watch-pattern notifications to the originating session ([#10460](https://github.com/NousResearch/hermes-agent/pull/10460)) +- Fix: preserve notify context in executor threads ([#10921](https://github.com/NousResearch/hermes-agent/pull/10921), @kshitijk4poor) +- Fix: avoid duplicate replies after interrupted long tasks ([#11018](https://github.com/NousResearch/hermes-agent/pull/11018)) +- Fix: unlink stale PID + lock files on cleanup +- Fix: force-unlink stale PID file after `--replace` takeover + +--- + +## 🔧 Tool System + +### Plugin Surface (major expansion) +- **`register_command()`** — plugins can now add slash commands ([#10626](https://github.com/NousResearch/hermes-agent/pull/10626)) +- **`dispatch_tool()`** — plugins can invoke tools from their code ([#10763](https://github.com/NousResearch/hermes-agent/pull/10763)) +- **`pre_tool_call` blocking** — plugins can veto tool execution ([#9377](https://github.com/NousResearch/hermes-agent/pull/9377)) +- **`transform_tool_result`** — plugins rewrite tool results generically ([#12972](https://github.com/NousResearch/hermes-agent/pull/12972)) +- **`transform_terminal_output`** — plugins rewrite terminal tool output ([#12929](https://github.com/NousResearch/hermes-agent/pull/12929)) +- **Namespaced skill registration** for plugin skill bundles ([#9786](https://github.com/NousResearch/hermes-agent/pull/9786)) +- **Opt-in-by-default + bundled disk-cleanup plugin** (salvage #12212) ([#12944](https://github.com/NousResearch/hermes-agent/pull/12944)) +- **Pluggable `image_gen` backends + OpenAI provider** ([#13799](https://github.com/NousResearch/hermes-agent/pull/13799)) +- **`openai-codex` image_gen plugin** (gpt-image-2 via Codex OAuth) ([#14317](https://github.com/NousResearch/hermes-agent/pull/14317)) +- **Shell hooks** — wire shell scripts as hook callbacks ([#13296](https://github.com/NousResearch/hermes-agent/pull/13296)) + +### Browser +- **`browser_cdp` raw DevTools Protocol passthrough** ([#12369](https://github.com/NousResearch/hermes-agent/pull/12369)) +- Camofox hardening + connection stability across the window + +### Execute Code +- **Project/strict execution modes** (default: project) ([#11971](https://github.com/NousResearch/hermes-agent/pull/11971)) + +### Image Generation +- **Multi-model FAL support** with picker in `hermes tools` ([#11265](https://github.com/NousResearch/hermes-agent/pull/11265)) +- **Recraft V3 → V4 Pro, Nano Banana → Pro upgrades** ([#11406](https://github.com/NousResearch/hermes-agent/pull/11406)) +- **GPT Image 2** in FAL catalog ([#13677](https://github.com/NousResearch/hermes-agent/pull/13677)) +- **xAI image generation provider** (grok-imagine-image) ([#14765](https://github.com/NousResearch/hermes-agent/pull/14765)) + +### TTS / STT / Voice +- **Google Gemini TTS provider** ([#11229](https://github.com/NousResearch/hermes-agent/pull/11229)) +- **xAI Grok STT provider** ([#14473](https://github.com/NousResearch/hermes-agent/pull/14473)) +- **xAI TTS** (shipped with Responses API upgrade) ([#10783](https://github.com/NousResearch/hermes-agent/pull/10783)) +- **KittenTTS local provider** (salvage of #2109) ([#13395](https://github.com/NousResearch/hermes-agent/pull/13395)) +- **CLI record beep toggle** ([#13247](https://github.com/NousResearch/hermes-agent/pull/13247), @helix4u) + +### Webhook / Cron +- **Webhook direct-delivery mode** — zero-LLM push notifications ([#12473](https://github.com/NousResearch/hermes-agent/pull/12473)) +- **Cron `wakeAgent` gate** — scripts can skip the agent entirely ([#12373](https://github.com/NousResearch/hermes-agent/pull/12373)) +- **Cron per-job `enabled_toolsets`** — cap token overhead + cost per job ([#14767](https://github.com/NousResearch/hermes-agent/pull/14767)) + +### Delegate +- **Orchestrator role** + configurable spawn depth (default flat) ([#13691](https://github.com/NousResearch/hermes-agent/pull/13691)) +- **Cross-agent file state coordination** ([#13718](https://github.com/NousResearch/hermes-agent/pull/13718)) + +### File / Patch +- **`patch` — "did you mean?" feedback** when patch fails to match ([#13435](https://github.com/NousResearch/hermes-agent/pull/13435)) + +### API Server +- **Stream `/v1/responses` SSE tool events** (salvage #9779) ([#10049](https://github.com/NousResearch/hermes-agent/pull/10049)) +- **Inline image inputs** on `/v1/chat/completions` and `/v1/responses` ([#12969](https://github.com/NousResearch/hermes-agent/pull/12969)) + +### Docker / Podman +- **Entry-level Podman support** — `find_docker()` + rootless entrypoint ([#10066](https://github.com/NousResearch/hermes-agent/pull/10066)) +- **Add docker-cli to Docker image** (salvage #10096) ([#14232](https://github.com/NousResearch/hermes-agent/pull/14232)) +- **File-sync back to host on teardown** (salvage of #8189 + hardening) ([#11291](https://github.com/NousResearch/hermes-agent/pull/11291)) + +### MCP +- 12 MCP improvements across the window (status, timeout handling, tool-call forwarding, etc.) + +--- + +## 🧩 Skills Ecosystem + +### Skill System +- **Namespaced skill registration** for plugin bundles ([#9786](https://github.com/NousResearch/hermes-agent/pull/9786)) +- **`hermes skills reset`** to un-stick bundled skills ([#11468](https://github.com/NousResearch/hermes-agent/pull/11468)) +- **Skills guard opt-in** — `config.skills.guard_agent_created` (default off) ([#14557](https://github.com/NousResearch/hermes-agent/pull/14557)) +- **Bundled skill scripts runnable out of the box** ([#13384](https://github.com/NousResearch/hermes-agent/pull/13384)) +- **`xitter` replaced with `xurl`** — the official X API CLI ([#12303](https://github.com/NousResearch/hermes-agent/pull/12303)) +- **MiniMax-AI/cli as default skill tap** (salvage #7501) ([#14493](https://github.com/NousResearch/hermes-agent/pull/14493)) +- **Fuzzy `@` file completions + mtime sorting** ([#9467](https://github.com/NousResearch/hermes-agent/pull/9467)) + +### New Skills +- **concept-diagrams** (salvage of #11045, @v1k22) ([#11363](https://github.com/NousResearch/hermes-agent/pull/11363)) +- **architecture-diagram** (Cocoon AI port) ([#9906](https://github.com/NousResearch/hermes-agent/pull/9906)) +- **pixel-art** with hardware palettes and video animation ([#12663](https://github.com/NousResearch/hermes-agent/pull/12663), [#12725](https://github.com/NousResearch/hermes-agent/pull/12725)) +- **baoyu-comic** ([#13257](https://github.com/NousResearch/hermes-agent/pull/13257), @JimLiu) +- **baoyu-infographic** — 21 layouts × 21 styles (salvage #9901) ([#12254](https://github.com/NousResearch/hermes-agent/pull/12254)) +- **page-agent** — embed Alibaba's in-page GUI agent in your webapp ([#13976](https://github.com/NousResearch/hermes-agent/pull/13976)) +- **fitness-nutrition** optional skill + optional env var support ([#9355](https://github.com/NousResearch/hermes-agent/pull/9355)) +- **drug-discovery** — ChEMBL, PubChem, OpenFDA, ADMET ([#9443](https://github.com/NousResearch/hermes-agent/pull/9443)) +- **touchdesigner-mcp** (salvage of #10081) ([#12298](https://github.com/NousResearch/hermes-agent/pull/12298)) +- **adversarial-ux-test** optional skill (salvage of #2494, @omnissiah-comelse) ([#13425](https://github.com/NousResearch/hermes-agent/pull/13425)) +- **maps** — added `guest_house`, `camp_site`, and dual-key bakery lookup ([#13398](https://github.com/NousResearch/hermes-agent/pull/13398)) +- **llm-wiki** — port provenance markers, source hashing, and quality signals ([#13700](https://github.com/NousResearch/hermes-agent/pull/13700)) + +--- + +## 📊 Web Dashboard + +- **i18n (English + Chinese) language switcher** ([#9453](https://github.com/NousResearch/hermes-agent/pull/9453)) +- **Live-switching theme system** ([#10687](https://github.com/NousResearch/hermes-agent/pull/10687)) +- **Dashboard plugin system** — extend the web UI with custom tabs ([#10951](https://github.com/NousResearch/hermes-agent/pull/10951)) +- **react-router, sidebar layout, sticky header, dropdown component** ([#9370](https://github.com/NousResearch/hermes-agent/pull/9370), @austinpickett) +- **Responsive for mobile** ([#9228](https://github.com/NousResearch/hermes-agent/pull/9228), @DeployFaith) +- **Vercel deployment** ([#10686](https://github.com/NousResearch/hermes-agent/pull/10686), [#11061](https://github.com/NousResearch/hermes-agent/pull/11061), @austinpickett) +- **Context window config support** ([#9357](https://github.com/NousResearch/hermes-agent/pull/9357)) +- **HTTP health probe for cross-container gateway detection** ([#9894](https://github.com/NousResearch/hermes-agent/pull/9894)) +- **Update + restart gateway buttons** ([#13526](https://github.com/NousResearch/hermes-agent/pull/13526), @austinpickett) +- **Real API call count per session** (salvages #10140) ([#14004](https://github.com/NousResearch/hermes-agent/pull/14004)) + +--- + +## 🖱️ CLI & User Experience + +- **Dynamic shell completion for bash, zsh, and fish** ([#9785](https://github.com/NousResearch/hermes-agent/pull/9785)) +- **Light-mode skins + skin-aware completion menus** ([#9461](https://github.com/NousResearch/hermes-agent/pull/9461)) +- **Numbered keyboard shortcuts** on approval and clarify prompts ([#13416](https://github.com/NousResearch/hermes-agent/pull/13416)) +- **Markdown stripping, compact multiline previews, external editor** ([#12934](https://github.com/NousResearch/hermes-agent/pull/12934)) +- **`--ignore-user-config` and `--ignore-rules` flags** (port codex#18646) ([#14277](https://github.com/NousResearch/hermes-agent/pull/14277)) +- **Account limits section in `/usage`** ([#13428](https://github.com/NousResearch/hermes-agent/pull/13428)) +- **Doctor: Command Installation check** for `hermes` bin symlink ([#10112](https://github.com/NousResearch/hermes-agent/pull/10112)) +- **ESC cancels secret/sudo prompts**, clearer skip messaging ([#9902](https://github.com/NousResearch/hermes-agent/pull/9902)) +- Fix: agent-facing text uses `display_hermes_home()` instead of hardcoded `~/.hermes` ([#10285](https://github.com/NousResearch/hermes-agent/pull/10285)) +- Fix: enforce `config.yaml` as sole CWD source + deprecate `.env` CWD vars + add `hermes memory reset` ([#11029](https://github.com/NousResearch/hermes-agent/pull/11029)) + +--- + +## 🔒 Security & Reliability + +- **Global toggle to allow private/internal URL resolution** ([#14166](https://github.com/NousResearch/hermes-agent/pull/14166)) +- **Block agent from self-destructing the gateway** via terminal (closes #6666) ([#9895](https://github.com/NousResearch/hermes-agent/pull/9895)) +- **Telegram callback authorization** on update prompts ([#10536](https://github.com/NousResearch/hermes-agent/pull/10536)) +- **SECURITY.md** added ([#10532](https://github.com/NousResearch/hermes-agent/pull/10532), @I3eg1nner) +- **Warn about legacy hermes.service units** during `hermes update` ([#11918](https://github.com/NousResearch/hermes-agent/pull/11918)) +- **Complete ASCII-locale UnicodeEncodeError recovery** for `api_messages`/`reasoning_content` (closes #6843) ([#10537](https://github.com/NousResearch/hermes-agent/pull/10537)) +- **Prevent stale `os.environ` leak** after `clear_session_vars` ([#10527](https://github.com/NousResearch/hermes-agent/pull/10527)) +- **Prevent agent hang when backgrounding processes** via terminal tool ([#10584](https://github.com/NousResearch/hermes-agent/pull/10584)) +- Many smaller session-resume, interrupt, streaming, and memory-race fixes throughout the window + +--- + +## 🐛 Notable Bug Fixes + +The `fix:` category in this window covers 482 PRs. Highlights: + +- Streaming cursor artifacts filtered from Matrix, Telegram, WhatsApp, Discord (multiple PRs) +- `` and `` blocks filtered from gateway stream consumers ([#9408](https://github.com/NousResearch/hermes-agent/pull/9408)) +- Gateway display.streaming root-config override regression ([#9799](https://github.com/NousResearch/hermes-agent/pull/9799)) +- Context `session_search` coerces limit to int (prevents TypeError) ([#10522](https://github.com/NousResearch/hermes-agent/pull/10522)) +- Memory tool stays available when `fcntl` is unavailable (Windows) ([#9783](https://github.com/NousResearch/hermes-agent/pull/9783)) +- Trajectory compressor credentials load from `HERMES_HOME/.env` ([#9632](https://github.com/NousResearch/hermes-agent/pull/9632), @Dusk1e) +- `@_context_completions` no longer crashes on `@` mention ([#9683](https://github.com/NousResearch/hermes-agent/pull/9683), @kshitijk4poor) +- Group session `user_id` no longer treated as `thread_id` in shutdown notifications ([#10546](https://github.com/NousResearch/hermes-agent/pull/10546)) +- Telegram `platform_hint` — markdown is supported (closes #8261) ([#10612](https://github.com/NousResearch/hermes-agent/pull/10612)) +- Doctor checks for Kimi China credentials fixed +- Streaming: don't suppress final response when commentary message is sent ([#10540](https://github.com/NousResearch/hermes-agent/pull/10540)) +- Rapid Telegram follow-ups no longer get cut off + +--- + +## 🧪 Testing & CI + +- **Contributor attribution CI check** on PRs ([#9376](https://github.com/NousResearch/hermes-agent/pull/9376)) +- Hermetic test parity (`scripts/run_tests.sh`) held across this window +- Test count stabilized post-Transport refactor; CI matrix held green through the transport rollout + +--- + +## 📚 Documentation + +- Atropos + wandb links in user guide +- ACP / VS Code / Zed / JetBrains integration docs refresh +- Webhook subscription docs updated for direct-delivery mode +- Plugin author guide expanded for new hooks (`register_command`, `dispatch_tool`, `transform_tool_result`) +- Transport layer developer guide added +- Website removed Discussions link from README + +--- + +## 👥 Contributors + +### Core +- **@teknium1** (Teknium) + +### Top Community Contributors (by merged PR count) +- **@kshitijk4poor** — 49 PRs · Transport refactor (AnthropicTransport, ResponsesApiTransport), Step Plan provider, Xiaomi MiMo v2.5 support, numerous gateway fixes, promoted Kimi K2.5, @ mention crash fix +- **@OutThisLife** (Brooklyn) — 31 PRs · TUI polish, git branch in status bar, per-turn stopwatch, stable picker keys, `/clear` confirm, light-theme preset, subagent spawn observability overlay +- **@helix4u** — 11 PRs · Voice CLI record beep, MCP tool interrupt handling, assorted stability fixes +- **@austinpickett** — 8 PRs · Dashboard react-router + sidebar + sticky header + dropdown, Vercel deployment, update + restart buttons +- **@alt-glitch** — 8 PRs · PLATFORM_HINTS for Matrix/Mattermost/Feishu, Matrix fixes +- **@ethernet8023** — 3 PRs +- **@benbarclay** — 3 PRs +- **@Aslaaen** — 2 PRs + +### Also contributing +@jerilynzheng (ai-gateway pricing), @JimLiu (baoyu-comic skill), @Dusk1e (trajectory compressor credentials), @DeployFaith (mobile-responsive dashboard), @LeonSGP43, @v1k22 (concept-diagrams), @omnissiah-comelse (adversarial-ux-test), @coekfung (Telegram MarkdownV2 expandable blockquotes), @liftaris (TUI provider resolution), @arihantsethia (skill analytics dashboard), @topcheer + @xing8star (QQBot foundation), @kovyrin, @I3eg1nner (SECURITY.md), @PeterBerthelsen, @lengxii, @priveperfumes, @sjz-ks, @cuyua9, @Disaster-Terminator, @leozeli, @LehaoLin, @trevthefoolish, @loongfay, @MrNiceRicee, @WideLee, @bluefishs, @malaiwah, @bobashopcashier, @dsocolobsky, @iamagenius00, @IAvecilla, @aniruddhaadak80, @Es1la, @asheriif, @walli, @jquesnelle (original Tool Gateway work). + +### All Contributors (alphabetical) + +@0xyg3n, @10ishq, @A-afflatus, @Abnertheforeman, @admin28980, @adybag14-cyber, @akhater, @alexzhu0, +@AllardQuek, @alt-glitch, @aniruddhaadak80, @anna-oake, @anniesurla, @anthhub, @areu01or00, @arihantsethia, +@arthurbr11, @asheriif, @Aslaaen, @Asunfly, @austinpickett, @AviArora02-commits, @AxDSan, @azhengbot, @Bartok9, +@benbarclay, @bennytimz, @bernylinville, @bingo906, @binhnt92, @bkadish, @bluefishs, @bobashopcashier, +@brantzh6, @BrennerSpear, @brianclemens, @briandevans, @brooklynnicholson, @bugkill3r, @buray, @burtenshaw, +@cdanis, @cgarwood82, @ChimingLiu, @chongweiliu, @christopherwoodall, @coekfung, @cola-runner, @corazzione, +@counterposition, @cresslank, @cuyua9, @cypres0099, @danieldoderlein, @davetist, @davidvv, @DeployFaith, +@Dev-Mriganka, @devorun, @dieutx, @Disaster-Terminator, @dodo-reach, @draix, @DrStrangerUJN, @dsocolobsky, +@Dusk1e, @dyxushuai, @elkimek, @elmatadorgh, @emozilla, @entropidelic, @Erosika, @erosika, @Es1la, @etcircle, +@etherman-os, @ethernet8023, @fancydirty, @farion1231, @fatinghenji, @Fatty911, @fengtianyu88, @Feranmi10, +@flobo3, @francip, @fuleinist, @g-guthrie, @GenKoKo, @gianfrancopiana, @gnanam1990, @GuyCui, @haileymarshall, +@haimu0x, @handsdiff, @hansnow, @hedgeho9X, @helix4u, @hengm3467, @HenkDz, @heykb, @hharry11, @HiddenPuppy, +@honghua, @houko, @houziershi, @hsy5571616, @huangke19, @hxp-plus, @Hypn0sis, @I3eg1nner, @iacker, +@iamagenius00, @IAvecilla, @iborazzi, @Ifkellx, @ifrederico, @imink, @isaachuangGMICLOUD, @ismell0992-afk, +@j0sephz, @Jaaneek, @jackjin1997, @JackTheGit, @jaffarkeikei, @jerilynzheng, @JiaDe-Wu, @Jiawen-lee, @JimLiu, +@jinzheng8115, @jneeee, @jplew, @jquesnelle, @Julientalbot, @Junass1, @jvcl, @kagura-agent, @keifergu, +@kevinskysunny, @keyuyuan, @konsisumer, @kovyrin, @kshitijk4poor, @leeyang1990, @LehaoLin, @lengxii, +@LeonSGP43, @leozeli, @li0near, @liftaris, @Lind3ey, @Linux2010, @liujinkun2025, @LLQWQ, @Llugaes, @lmoncany, +@longsizhuo, @lrawnsley, @Lubrsy706, @lumenradley, @luyao618, @lvnilesh, @LVT382009, @m0n5t3r, @Magaav, +@MagicRay1217, @malaiwah, @manuelschipper, @Marvae, @MassiveMassimo, @mavrickdeveloper, @maxchernin, @memosr, +@meng93, @mengjian-github, @MestreY0d4-Uninter, @Mibayy, @MikeFac, @mikewaters, @milkoor, @minorgod, +@MrNiceRicee, @ms-alan, @mvanhorn, @n-WN, @N0nb0at, @Nan93, @NIDNASSER-Abdelmajid, @nish3451, @niyoh120, +@nocoo, @nosleepcassette, @NousResearch, @ogzerber, @omnissiah-comelse, @Only-Code-A, @opriz, @OwenYWT, @pedh, +@pefontana, @PeterBerthelsen, @phpoh, @pinion05, @plgonzalezrx8, @pradeep7127, @priveperfumes, +@projectadmin-dev, @PStarH, @rnijhara, @Roy-oss1, @roytian1217, @RucchiZ, @Ruzzgar, @RyanLee-Dev, @Salt-555, +@Sanjays2402, @sgaofen, @sharziki, @shenuu, @shin4, @SHL0MS, @shushuzn, @sicnuyudidi, @simon-gtcl, +@simon-marcus, @sirEven, @Sisyphus, @sjz-ks, @snreynolds, @Societus, @Somme4096, @sontianye, @sprmn24, +@StefanIsMe, @stephenschoettler, @Swift42, @taeng0204, @taeuk178, @tannerfokkens-maker, @TaroballzChen, +@ten-ltw, @teyrebaz33, @Tianworld, @topcheer, @Tranquil-Flow, @trevthefoolish, @TroyMitchell911, @UNLINEARITY, +@v1k22, @vivganes, @vominh1919, @vrinek, @VTRiot, @WadydX, @walli, @wenhao7, @WhiteWorld, @WideLee, @wujhsu, +@WuTianyi123, @Wysie, @xandersbell, @xiaoqiang243, @xiayh0107, @xinpengdr, @Xowiek, @ycbai, @yeyitech, @ygd58, +@youngDoo, @yudaiyan, @Yukipukii1, @yule975, @yyq4193, @yzx9, @ZaynJarvis, @zhang9w0v5, @zhanggttry, +@zhangxicen, @zhongyueming1121, @zhouxiaoya12, @zons-zhaozhy + +Also: @maelrx, @Marco Rutsch, @MaxsolcuCrypto, @Mind-Dragon, @Paul Bergeron, @say8hi, @whitehatjr1001. + + +--- + +**Full Changelog**: [v2026.4.13...v2026.4.23](https://github.com/NousResearch/hermes-agent/compare/v2026.4.13...v2026.4.23) diff --git a/RELEASE_v0.9.0.md b/RELEASE_v0.9.0.md index e895d818bc4e..15d5b84b4023 100644 --- a/RELEASE_v0.9.0.md +++ b/RELEASE_v0.9.0.md @@ -318,6 +318,7 @@ - **@JiayuuWang** — CLI uninstall import fix - **@HiddenPuppy** — Docker procps installation - **@dsocolobsky** — Test suite fixes +- **@bobashopcashier** (1 PR) — Graceful gateway drain before restart (salvaged into #7503 from #7290) - **@benbarclay** — Docker image tag simplification - **@sosyz** — Shallow git clone for faster install - **@devorun** — Nix setupSecrets optional diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 000000000000..3cede2885e61 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,84 @@ +# Hermes Agent Security Policy + +This document outlines the security protocols, trust model, and deployment hardening guidelines for the **Hermes Agent** project. + +## 1. Vulnerability Reporting + +Hermes Agent does **not** operate a bug bounty program. Security issues should be reported via [GitHub Security Advisories (GHSA)](https://github.com/NousResearch/hermes-agent/security/advisories/new) or by emailing **security@nousresearch.com**. Do not open public issues for security vulnerabilities. + +### Required Submission Details +- **Title & Severity:** Concise description and CVSS score/rating. +- **Affected Component:** Exact file path and line range (e.g., `tools/approval.py:120-145`). +- **Environment:** Output of `hermes version`, commit SHA, OS, and Python version. +- **Reproduction:** Step-by-step Proof-of-Concept (PoC) against `main` or the latest release. +- **Impact:** Explanation of what trust boundary was crossed. + +--- + +## 2. Trust Model + +The core assumption is that Hermes is a **personal agent** with one trusted operator. + +### Operator & Session Trust +- **Single Tenant:** The system protects the operator from LLM actions, not from malicious co-tenants. Multi-user isolation must happen at the OS/host level. +- **Gateway Security:** Authorized callers (Telegram, Discord, Slack, etc.) receive equal trust. Session keys are used for routing, not as authorization boundaries. +- **Execution:** Defaults to `terminal.backend: local` (direct host execution). Container isolation (Docker, Modal, Daytona) is opt-in for sandboxing. + +### Dangerous Command Approval +The approval system (`tools/approval.py`) is a core security boundary. Terminal commands, file operations, and other potentially destructive actions are gated behind explicit user confirmation before execution. The approval mode is configurable via `approvals.mode` in `config.yaml`: +- `"on"` (default) — prompts the user to approve dangerous commands. +- `"auto"` — auto-approves after a configurable delay. +- `"off"` — disables the gate entirely (break-glass; see Section 3). + +### Output Redaction +`agent/redact.py` strips secret-like patterns (API keys, tokens, credentials) from all display output before it reaches the terminal or gateway platform. This prevents accidental credential leakage in chat logs, tool previews, and response text. Redaction operates on the display layer only — underlying values remain intact for internal agent operations. + +### Skills vs. MCP Servers +- **Installed Skills:** High trust. Equivalent to local host code; skills can read environment variables and run arbitrary commands. +- **MCP Servers:** Lower trust. MCP subprocesses receive a filtered environment (`_build_safe_env()` in `tools/mcp_tool.py`) — only safe baseline variables (`PATH`, `HOME`, `XDG_*`) plus variables explicitly declared in the server's `env` config block are passed through. Host credentials are stripped by default. Additionally, packages invoked via `npx`/`uvx` are checked against the OSV malware database before spawning. + +### Code Execution Sandbox +The `execute_code` tool (`tools/code_execution_tool.py`) runs LLM-generated Python scripts in a child process with API keys and tokens stripped from the environment to prevent credential exfiltration. Only environment variables explicitly declared by loaded skills (via `env_passthrough`) or by the user in `config.yaml` (`terminal.env_passthrough`) are passed through. The child accesses Hermes tools via RPC, not direct API calls. + +### Subagents +- **No recursive delegation:** The `delegate_task` tool is disabled for child agents. +- **Depth limit:** `MAX_DEPTH = 2` — parent (depth 0) can spawn a child (depth 1); grandchildren are rejected. +- **Memory isolation:** Subagents run with `skip_memory=True` and do not have access to the parent's persistent memory provider. The parent receives only the task prompt and final response as an observation. + +--- + +## 3. Out of Scope (Non-Vulnerabilities) + +The following scenarios are **not** considered security breaches: +- **Prompt Injection:** Unless it results in a concrete bypass of the approval system, toolset restrictions, or container sandbox. +- **Public Exposure:** Deploying the gateway to the public internet without external authentication or network protection. +- **Trusted State Access:** Reports that require pre-existing write access to `~/.hermes/`, `.env`, or `config.yaml` (these are operator-owned files). +- **Default Behavior:** Host-level command execution when `terminal.backend` is set to `local` — this is the documented default, not a vulnerability. +- **Configuration Trade-offs:** Intentional break-glass settings such as `approvals.mode: "off"` or `terminal.backend: local` in production. +- **Tool-level read/access restrictions:** The agent has unrestricted shell access via the `terminal` tool by design. Reports that a specific tool (e.g., `read_file`) can access a resource are not vulnerabilities if the same access is available through `terminal`. Tool-level deny lists only constitute a meaningful security boundary when paired with equivalent restrictions on the terminal side (as with write operations, where `WRITE_DENIED_PATHS` is paired with the dangerous command approval system). + +--- + +## 4. Deployment Hardening & Best Practices + +### Filesystem & Network +- **Production sandboxing:** Use container backends (`docker`, `modal`, `daytona`) instead of `local` for untrusted workloads. +- **File permissions:** Run as non-root (the Docker image uses UID 10000); protect credentials with `chmod 600 ~/.hermes/.env` on local installs. +- **Network exposure:** Do not expose the gateway or API server to the public internet without VPN, Tailscale, or firewall protection. SSRF protection is enabled by default across all gateway platform adapters (Telegram, Discord, Slack, Matrix, Mattermost, etc.) with redirect validation. Note: the local terminal backend does not apply SSRF filtering, as it operates within the trusted operator's environment. + +### Skills & Supply Chain +- **Skill installation:** Review Skills Guard reports (`tools/skills_guard.py`) before installing third-party skills. The audit log at `~/.hermes/skills/.hub/audit.log` tracks every install and removal. +- **MCP safety:** OSV malware checking runs automatically for `npx`/`uvx` packages before MCP server processes are spawned. +- **CI/CD:** GitHub Actions are pinned to full commit SHAs. The `supply-chain-audit.yml` workflow blocks PRs containing `.pth` files or suspicious `base64`+`exec` patterns. + +### Credential Storage +- API keys and tokens belong exclusively in `~/.hermes/.env` — never in `config.yaml` or checked into version control. +- The credential pool system (`agent/credential_pool.py`) handles key rotation and fallback. Credentials are resolved from environment variables, not stored in plaintext databases. + +--- + +## 5. Disclosure Process + +- **Coordinated Disclosure:** 90-day window or until a fix is released, whichever comes first. +- **Communication:** All updates occur via the GHSA thread or email correspondence with security@nousresearch.com. +- **Credits:** Reporters are credited in release notes unless anonymity is requested. diff --git a/acp_adapter/entry.py b/acp_adapter/entry.py index 7db5747a4df5..3089f78c27e0 100644 --- a/acp_adapter/entry.py +++ b/acp_adapter/entry.py @@ -20,6 +20,46 @@ from hermes_constants import get_hermes_home +# Methods clients send as periodic liveness probes. They are not part of the +# ACP schema, so the acp router correctly returns JSON-RPC -32601 to the +# caller — but the supervisor task that dispatches the request then surfaces +# the raised RequestError via ``logging.exception("Background task failed")``, +# which dumps a traceback to stderr every probe interval. Clients like +# acp-bridge already treat the -32601 response as "agent alive", so the +# traceback is pure noise. We keep the protocol response intact and only +# silence the stderr noise for this specific benign case. +_BENIGN_PROBE_METHODS = frozenset({"ping", "health", "healthcheck"}) + + +class _BenignProbeMethodFilter(logging.Filter): + """Suppress acp 'Background task failed' tracebacks caused by unknown + liveness-probe methods (e.g. ``ping``) while leaving every other + background-task error — including method_not_found for any non-probe + method — visible in stderr. + """ + + def filter(self, record: logging.LogRecord) -> bool: + if record.getMessage() != "Background task failed": + return True + exc_info = record.exc_info + if not exc_info: + return True + exc = exc_info[1] + # Imported lazily so this module stays importable when the optional + # ``agent-client-protocol`` dependency is not installed. + try: + from acp.exceptions import RequestError + except ImportError: + return True + if not isinstance(exc, RequestError): + return True + if getattr(exc, "code", None) != -32601: + return True + data = getattr(exc, "data", None) + method = data.get("method") if isinstance(data, dict) else None + return method not in _BENIGN_PROBE_METHODS + + def _setup_logging() -> None: """Route all logging to stderr so stdout stays clean for ACP stdio.""" handler = logging.StreamHandler(sys.stderr) @@ -29,6 +69,7 @@ def _setup_logging() -> None: datefmt="%Y-%m-%d %H:%M:%S", ) ) + handler.addFilter(_BenignProbeMethodFilter()) root = logging.getLogger() root.handlers.clear() root.addHandler(handler) diff --git a/acp_adapter/events.py b/acp_adapter/events.py index 08da40a685e8..1257f902ebbb 100644 --- a/acp_adapter/events.py +++ b/acp_adapter/events.py @@ -49,6 +49,7 @@ def make_tool_progress_cb( session_id: str, loop: asyncio.AbstractEventLoop, tool_call_ids: Dict[str, Deque[str]], + tool_call_meta: Dict[str, Dict[str, Any]], ) -> Callable: """Create a ``tool_progress_callback`` for AIAgent. @@ -84,6 +85,16 @@ def _tool_progress(event_type: str, name: str = None, preview: str = None, args: tool_call_ids[name] = queue queue.append(tc_id) + snapshot = None + if name in {"write_file", "patch", "skill_manage"}: + try: + from agent.display import capture_local_edit_snapshot + + snapshot = capture_local_edit_snapshot(name, args) + except Exception: + logger.debug("Failed to capture ACP edit snapshot for %s", name, exc_info=True) + tool_call_meta[tc_id] = {"args": args, "snapshot": snapshot} + update = build_tool_start(tc_id, name, args) _send_update(conn, session_id, loop, update) @@ -119,6 +130,7 @@ def make_step_cb( session_id: str, loop: asyncio.AbstractEventLoop, tool_call_ids: Dict[str, Deque[str]], + tool_call_meta: Dict[str, Dict[str, Any]], ) -> Callable: """Create a ``step_callback`` for AIAgent. @@ -132,10 +144,12 @@ def _step(api_call_count: int, prev_tools: Any = None) -> None: for tool_info in prev_tools: tool_name = None result = None + function_args = None if isinstance(tool_info, dict): tool_name = tool_info.get("name") or tool_info.get("function_name") result = tool_info.get("result") or tool_info.get("output") + function_args = tool_info.get("arguments") or tool_info.get("args") elif isinstance(tool_info, str): tool_name = tool_info @@ -145,8 +159,13 @@ def _step(api_call_count: int, prev_tools: Any = None) -> None: tool_call_ids[tool_name] = queue if tool_name and queue: tc_id = queue.popleft() + meta = tool_call_meta.pop(tc_id, {}) update = build_tool_complete( - tc_id, tool_name, result=str(result) if result is not None else None + tc_id, + tool_name, + result=str(result) if result is not None else None, + function_args=function_args or meta.get("args"), + snapshot=meta.get("snapshot"), ) _send_update(conn, session_id, loop, update) if not queue: diff --git a/acp_adapter/permissions.py b/acp_adapter/permissions.py index 68f61e340ab8..c2e1a598269b 100644 --- a/acp_adapter/permissions.py +++ b/acp_adapter/permissions.py @@ -63,6 +63,9 @@ def _callback(command: str, description: str) -> str: logger.warning("Permission request timed out or failed: %s", exc) return "deny" + if response is None: + return "deny" + outcome = response.outcome if isinstance(outcome, AllowedOutcome): option_id = outcome.option_id diff --git a/acp_adapter/server.py b/acp_adapter/server.py index 29f9a10e8b25..d73c71157ae1 100644 --- a/acp_adapter/server.py +++ b/acp_adapter/server.py @@ -4,6 +4,7 @@ import asyncio import logging +import os from collections import defaultdict, deque from concurrent.futures import ThreadPoolExecutor from typing import Any, Deque, Optional @@ -26,6 +27,7 @@ McpServerHttp, McpServerSse, McpServerStdio, + ModelInfo, NewSessionResponse, PromptResponse, ResumeSessionResponse, @@ -36,6 +38,7 @@ SessionCapabilities, SessionForkCapabilities, SessionListCapabilities, + SessionModelState, SessionResumeCapabilities, SessionInfo, TextContentBlock, @@ -49,7 +52,7 @@ except ImportError: from acp.schema import AuthMethod as AuthMethodAgent # type: ignore[attr-defined] -from acp_adapter.auth import detect_provider, has_provider +from acp_adapter.auth import detect_provider from acp_adapter.events import ( make_message_cb, make_step_cb, @@ -69,6 +72,11 @@ # Thread pool for running AIAgent (synchronous) in parallel. _executor = ThreadPoolExecutor(max_workers=4, thread_name_prefix="acp-agent") +# Server-side page size for list_sessions. The ACP ListSessionsRequest schema +# does not expose a client-side limit, so this is a fixed cap that clients +# paginate against using `cursor` / `next_cursor`. +_LIST_SESSIONS_PAGE_SIZE = 50 + def _extract_text( prompt: list[ @@ -147,6 +155,98 @@ def on_connect(self, conn: acp.Client) -> None: self._conn = conn logger.info("ACP client connected") + @staticmethod + def _encode_model_choice(provider: str | None, model: str | None) -> str: + """Encode a model selection so ACP clients can keep provider context.""" + raw_model = str(model or "").strip() + if not raw_model: + return "" + raw_provider = str(provider or "").strip().lower() + if not raw_provider: + return raw_model + return f"{raw_provider}:{raw_model}" + + def _build_model_state(self, state: SessionState) -> SessionModelState | None: + """Return the ACP model selector payload for editors like Zed.""" + model = str(state.model or getattr(state.agent, "model", "") or "").strip() + provider = getattr(state.agent, "provider", None) or detect_provider() or "openrouter" + + try: + from hermes_cli.models import curated_models_for_provider, normalize_provider, provider_label + + normalized_provider = normalize_provider(provider) + provider_name = provider_label(normalized_provider) + available_models: list[ModelInfo] = [] + seen_ids: set[str] = set() + + for model_id, description in curated_models_for_provider(normalized_provider): + rendered_model = str(model_id or "").strip() + if not rendered_model: + continue + choice_id = self._encode_model_choice(normalized_provider, rendered_model) + if choice_id in seen_ids: + continue + desc_parts = [f"Provider: {provider_name}"] + if description: + desc_parts.append(str(description).strip()) + if rendered_model == model: + desc_parts.append("current") + available_models.append( + ModelInfo( + model_id=choice_id, + name=rendered_model, + description=" • ".join(part for part in desc_parts if part), + ) + ) + seen_ids.add(choice_id) + + current_model_id = self._encode_model_choice(normalized_provider, model) + if current_model_id and current_model_id not in seen_ids: + available_models.insert( + 0, + ModelInfo( + model_id=current_model_id, + name=model, + description=f"Provider: {provider_name} • current", + ), + ) + + if available_models: + return SessionModelState( + available_models=available_models, + current_model_id=current_model_id or available_models[0].model_id, + ) + except Exception: + logger.debug("Could not build ACP model state", exc_info=True) + + if not model: + return None + + fallback_choice = self._encode_model_choice(provider, model) + return SessionModelState( + available_models=[ModelInfo(model_id=fallback_choice, name=model)], + current_model_id=fallback_choice, + ) + + @staticmethod + def _resolve_model_selection(raw_model: str, current_provider: str) -> tuple[str, str]: + """Resolve ``provider:model`` input into the provider and normalized model id.""" + target_provider = current_provider + new_model = raw_model.strip() + + try: + from hermes_cli.models import detect_provider_for_model, parse_model_input + + target_provider, new_model = parse_model_input(new_model, current_provider) + if target_provider == current_provider: + detected = detect_provider_for_model(new_model, current_provider) + if detected: + target_provider, new_model = detected + except Exception: + logger.debug("Provider detection failed, using model as-is", exc_info=True) + + return target_provider, new_model + async def _register_session_mcp_servers( self, state: SessionState, @@ -257,9 +357,18 @@ async def initialize( ) async def authenticate(self, method_id: str, **kwargs: Any) -> AuthenticateResponse | None: - if has_provider(): - return AuthenticateResponse() - return None + # Only accept authenticate() calls whose method_id matches the + # provider we advertised in initialize(). Without this check, + # authenticate() would acknowledge any method_id as long as the + # server has provider credentials configured — harmless under + # Hermes' threat model (ACP is stdio-only, local-trust), but poor + # API hygiene and confusing if ACP ever grows multi-method auth. + provider = detect_provider() + if not provider: + return None + if not isinstance(method_id, str) or method_id.strip().lower() != provider: + return None + return AuthenticateResponse() # ---- Session management ------------------------------------------------- @@ -273,7 +382,10 @@ async def new_session( await self._register_session_mcp_servers(state, mcp_servers) logger.info("New session %s (cwd=%s)", state.session_id, cwd) self._schedule_available_commands_update(state.session_id) - return NewSessionResponse(session_id=state.session_id) + return NewSessionResponse( + session_id=state.session_id, + models=self._build_model_state(state), + ) async def load_session( self, @@ -289,7 +401,7 @@ async def load_session( await self._register_session_mcp_servers(state, mcp_servers) logger.info("Loaded session %s", session_id) self._schedule_available_commands_update(session_id) - return LoadSessionResponse() + return LoadSessionResponse(models=self._build_model_state(state)) async def resume_session( self, @@ -305,7 +417,7 @@ async def resume_session( await self._register_session_mcp_servers(state, mcp_servers) logger.info("Resumed session %s", state.session_id) self._schedule_available_commands_update(state.session_id) - return ResumeSessionResponse() + return ResumeSessionResponse(models=self._build_model_state(state)) async def cancel(self, session_id: str, **kwargs: Any) -> None: state = self.session_manager.get_session(session_id) @@ -340,12 +452,44 @@ async def list_sessions( cwd: str | None = None, **kwargs: Any, ) -> ListSessionsResponse: - infos = self.session_manager.list_sessions() - sessions = [ - SessionInfo(session_id=s["session_id"], cwd=s["cwd"]) - for s in infos - ] - return ListSessionsResponse(sessions=sessions) + """List ACP sessions with optional ``cwd`` filtering and cursor pagination. + + ``cwd`` is passed through to ``SessionManager.list_sessions`` which already + normalizes and filters by working directory. ``cursor`` is a ``session_id`` + previously returned as ``next_cursor``; results resume after that entry. + Server-side page size is capped at ``_LIST_SESSIONS_PAGE_SIZE``; when more + results remain, ``next_cursor`` is set to the last returned ``session_id``. + """ + infos = self.session_manager.list_sessions(cwd=cwd) + + if cursor: + for idx, s in enumerate(infos): + if s["session_id"] == cursor: + infos = infos[idx + 1:] + break + else: + # Unknown cursor -> empty page (do not fall back to full list). + infos = [] + + has_more = len(infos) > _LIST_SESSIONS_PAGE_SIZE + infos = infos[:_LIST_SESSIONS_PAGE_SIZE] + + sessions = [] + for s in infos: + updated_at = s.get("updated_at") + if updated_at is not None and not isinstance(updated_at, str): + updated_at = str(updated_at) + sessions.append( + SessionInfo( + session_id=s["session_id"], + cwd=s["cwd"], + title=s.get("title"), + updated_at=updated_at, + ) + ) + + next_cursor = sessions[-1].session_id if has_more and sessions else None + return ListSessionsResponse(sessions=sessions, next_cursor=next_cursor) # ---- Prompt (core) ------------------------------------------------------ @@ -389,12 +533,13 @@ async def prompt( state.cancel_event.clear() tool_call_ids: dict[str, Deque[str]] = defaultdict(deque) + tool_call_meta: dict[str, dict[str, Any]] = {} previous_approval_cb = None if conn: - tool_progress_cb = make_tool_progress_cb(conn, session_id, loop, tool_call_ids) + tool_progress_cb = make_tool_progress_cb(conn, session_id, loop, tool_call_ids, tool_call_meta) thinking_cb = make_thinking_cb(conn, session_id, loop) - step_cb = make_step_cb(conn, session_id, loop, tool_call_ids) + step_cb = make_step_cb(conn, session_id, loop, tool_call_ids, tool_call_meta) message_cb = make_message_cb(conn, session_id, loop) approval_cb = make_approval_callback(conn.request_permission, loop, session_id) else: @@ -410,15 +555,32 @@ async def prompt( agent.step_callback = step_cb agent.message_callback = message_cb - if approval_cb: - try: - from tools import terminal_tool as _terminal_tool - previous_approval_cb = getattr(_terminal_tool, "_approval_callback", None) - _terminal_tool.set_approval_callback(approval_cb) - except Exception: - logger.debug("Could not set ACP approval callback", exc_info=True) + # Approval callback is per-thread (thread-local, GHSA-qg5c-hvr5-hjgr). + # Set it INSIDE _run_agent so the TLS write happens in the executor + # thread — setting it here would write to the event-loop thread's TLS, + # not the executor's. Also set HERMES_INTERACTIVE so approval.py + # takes the CLI-interactive path (which calls the registered + # callback via prompt_dangerous_approval) instead of the + # non-interactive auto-approve branch (GHSA-96vc-wcxf-jjff). + # ACP's conn.request_permission maps cleanly to the interactive + # callback shape — not the gateway-queue HERMES_EXEC_ASK path, + # which requires a notify_cb registered in _gateway_notify_cbs. + previous_approval_cb = None + previous_interactive = None def _run_agent() -> dict: + nonlocal previous_approval_cb, previous_interactive + if approval_cb: + try: + from tools import terminal_tool as _terminal_tool + previous_approval_cb = _terminal_tool._get_approval_callback() + _terminal_tool.set_approval_callback(approval_cb) + except Exception: + logger.debug("Could not set ACP approval callback", exc_info=True) + # Signal to tools.approval that we have an interactive callback + # and the non-interactive auto-approve path must not fire. + previous_interactive = os.environ.get("HERMES_INTERACTIVE") + os.environ["HERMES_INTERACTIVE"] = "1" try: result = agent.run_conversation( user_message=user_text, @@ -430,6 +592,11 @@ def _run_agent() -> dict: logger.exception("Agent error in session %s", session_id) return {"final_response": f"Error: {e}", "messages": state.history} finally: + # Restore HERMES_INTERACTIVE. + if previous_interactive is None: + os.environ.pop("HERMES_INTERACTIVE", None) + else: + os.environ["HERMES_INTERACTIVE"] = previous_interactive if approval_cb: try: from tools import terminal_tool as _terminal_tool @@ -449,6 +616,19 @@ def _run_agent() -> dict: self.session_manager.save_session(session_id) final_response = result.get("final_response", "") + if final_response: + try: + from agent.title_generator import maybe_auto_title + + maybe_auto_title( + self.session_manager._get_db(), + session_id, + user_text, + final_response, + state.history, + ) + except Exception: + logger.debug("Failed to auto-title ACP session %s", session_id, exc_info=True) if final_response and conn: update = acp.update_agent_message_text(final_response) await conn.session_update(session_id, update) @@ -493,8 +673,8 @@ async def _send_available_commands_update(self, session_id: str) -> None: await self._conn.session_update( session_id=session_id, update=AvailableCommandsUpdate( - sessionUpdate="available_commands_update", - availableCommands=self._available_commands(), + session_update="available_commands_update", + available_commands=self._available_commands(), ), ) except Exception: @@ -556,27 +736,15 @@ def _cmd_model(self, args: str, state: SessionState) -> str: provider = getattr(state.agent, "provider", None) or "auto" return f"Current model: {model}\nProvider: {provider}" - new_model = args.strip() - target_provider = None current_provider = getattr(state.agent, "provider", None) or "openrouter" - - # Auto-detect provider for the requested model - try: - from hermes_cli.models import parse_model_input, detect_provider_for_model - target_provider, new_model = parse_model_input(new_model, current_provider) - if target_provider == current_provider: - detected = detect_provider_for_model(new_model, current_provider) - if detected: - target_provider, new_model = detected - except Exception: - logger.debug("Provider detection failed, using model as-is", exc_info=True) + target_provider, new_model = self._resolve_model_selection(args, current_provider) state.model = new_model state.agent = self.session_manager._make_agent( session_id=state.session_id, cwd=state.cwd, model=new_model, - requested_provider=target_provider or current_provider, + requested_provider=target_provider, ) self.session_manager.save_session(state.session_id) provider_label = getattr(state.agent, "provider", None) or target_provider or current_provider @@ -678,20 +846,30 @@ async def set_session_model( """Switch the model for a session (called by ACP protocol).""" state = self.session_manager.get_session(session_id) if state: - state.model = model_id current_provider = getattr(state.agent, "provider", None) - current_base_url = getattr(state.agent, "base_url", None) - current_api_mode = getattr(state.agent, "api_mode", None) + requested_provider, resolved_model = self._resolve_model_selection( + model_id, + current_provider or "openrouter", + ) + state.model = resolved_model + provider_changed = bool(current_provider and requested_provider != current_provider) + current_base_url = None if provider_changed else getattr(state.agent, "base_url", None) + current_api_mode = None if provider_changed else getattr(state.agent, "api_mode", None) state.agent = self.session_manager._make_agent( session_id=session_id, cwd=state.cwd, - model=model_id, - requested_provider=current_provider, + model=resolved_model, + requested_provider=requested_provider, base_url=current_base_url, api_mode=current_api_mode, ) self.session_manager.save_session(session_id) - logger.info("Session %s: model switched to %s", session_id, model_id) + logger.info( + "Session %s: model switched to %s via provider %s", + session_id, + resolved_model, + requested_provider, + ) return SetSessionModelResponse() logger.warning("Session %s: model switch requested for missing session", session_id) return None diff --git a/acp_adapter/session.py b/acp_adapter/session.py index 4bb823987e57..3f5f78f9a1d8 100644 --- a/acp_adapter/session.py +++ b/acp_adapter/session.py @@ -13,8 +13,12 @@ import copy import json import logging +import os +import re import sys +import time import uuid +from datetime import datetime, timezone from dataclasses import dataclass, field from threading import Lock from typing import Any, Dict, List, Optional @@ -22,6 +26,64 @@ logger = logging.getLogger(__name__) +def _normalize_cwd_for_compare(cwd: str | None) -> str: + raw = str(cwd or ".").strip() + if not raw: + raw = "." + expanded = os.path.expanduser(raw) + + # Normalize Windows drive paths into the equivalent WSL mount form so + # ACP history filters match the same workspace across Windows and WSL. + match = re.match(r"^([A-Za-z]):[\\/](.*)$", expanded) + if match: + drive = match.group(1).lower() + tail = match.group(2).replace("\\", "/") + expanded = f"/mnt/{drive}/{tail}" + elif re.match(r"^/mnt/[A-Za-z]/", expanded): + expanded = f"/mnt/{expanded[5].lower()}/{expanded[7:]}" + + return os.path.normpath(expanded) + + +def _build_session_title(title: Any, preview: Any, cwd: str | None) -> str: + explicit = str(title or "").strip() + if explicit: + return explicit + preview_text = str(preview or "").strip() + if preview_text: + return preview_text + leaf = os.path.basename(str(cwd or "").rstrip("/\\")) + return leaf or "New thread" + + +def _format_updated_at(value: Any) -> str | None: + if value is None: + return None + if isinstance(value, str) and value.strip(): + return value + try: + return datetime.fromtimestamp(float(value), tz=timezone.utc).isoformat() + except Exception: + return None + + +def _updated_at_sort_key(value: Any) -> float: + if value is None: + return float("-inf") + if isinstance(value, (int, float)): + return float(value) + raw = str(value).strip() + if not raw: + return float("-inf") + try: + return datetime.fromisoformat(raw.replace("Z", "+00:00")).timestamp() + except Exception: + try: + return float(raw) + except Exception: + return float("-inf") + + def _acp_stderr_print(*args, **kwargs) -> None: """Best-effort human-readable output sink for ACP stdio sessions. @@ -162,47 +224,78 @@ def fork_session(self, session_id: str, cwd: str = ".") -> Optional[SessionState logger.info("Forked ACP session %s -> %s", session_id, new_id) return state - def list_sessions(self) -> List[Dict[str, Any]]: + def list_sessions(self, cwd: str | None = None) -> List[Dict[str, Any]]: """Return lightweight info dicts for all sessions (memory + database).""" - # Collect in-memory sessions first. - with self._lock: - seen_ids = set(self._sessions.keys()) - results = [ - { - "session_id": s.session_id, - "cwd": s.cwd, - "model": s.model, - "history_len": len(s.history), - } - for s in self._sessions.values() - ] - - # Merge any persisted sessions not currently in memory. + normalized_cwd = _normalize_cwd_for_compare(cwd) if cwd else None db = self._get_db() + persisted_rows: dict[str, dict[str, Any]] = {} + if db is not None: try: - rows = db.search_sessions(source="acp", limit=1000) - for row in rows: - sid = row["id"] - if sid in seen_ids: - continue - # Extract cwd from model_config JSON. - cwd = "." - mc = row.get("model_config") - if mc: - try: - cwd = json.loads(mc).get("cwd", ".") - except (json.JSONDecodeError, TypeError): - pass - results.append({ - "session_id": sid, - "cwd": cwd, - "model": row.get("model") or "", - "history_len": row.get("message_count") or 0, - }) + for row in db.list_sessions_rich(source="acp", limit=1000): + persisted_rows[str(row["id"])] = dict(row) except Exception: - logger.debug("Failed to list ACP sessions from DB", exc_info=True) + logger.debug("Failed to load ACP sessions from DB", exc_info=True) + + # Collect in-memory sessions first. + with self._lock: + seen_ids = set(self._sessions.keys()) + results = [] + for s in self._sessions.values(): + history_len = len(s.history) + if history_len <= 0: + continue + if normalized_cwd and _normalize_cwd_for_compare(s.cwd) != normalized_cwd: + continue + persisted = persisted_rows.get(s.session_id, {}) + preview = next( + ( + str(msg.get("content") or "").strip() + for msg in s.history + if msg.get("role") == "user" and str(msg.get("content") or "").strip() + ), + persisted.get("preview") or "", + ) + results.append( + { + "session_id": s.session_id, + "cwd": s.cwd, + "model": s.model, + "history_len": history_len, + "title": _build_session_title(persisted.get("title"), preview, s.cwd), + "updated_at": _format_updated_at( + persisted.get("last_active") or persisted.get("started_at") or time.time() + ), + } + ) + # Merge any persisted sessions not currently in memory. + for sid, row in persisted_rows.items(): + if sid in seen_ids: + continue + message_count = int(row.get("message_count") or 0) + if message_count <= 0: + continue + # Extract cwd from model_config JSON. + session_cwd = "." + mc = row.get("model_config") + if mc: + try: + session_cwd = json.loads(mc).get("cwd", ".") + except (json.JSONDecodeError, TypeError): + pass + if normalized_cwd and _normalize_cwd_for_compare(session_cwd) != normalized_cwd: + continue + results.append({ + "session_id": sid, + "cwd": session_cwd, + "model": row.get("model") or "", + "history_len": message_count, + "title": _build_session_title(row.get("title"), row.get("preview"), session_cwd), + "updated_at": _format_updated_at(row.get("last_active") or row.get("started_at")), + }) + + results.sort(key=lambda item: _updated_at_sort_key(item.get("updated_at")), reverse=True) return results def update_cwd(self, session_id: str, cwd: str) -> Optional[SessionState]: diff --git a/acp_adapter/tools.py b/acp_adapter/tools.py index 52313220b7f6..067652106e16 100644 --- a/acp_adapter/tools.py +++ b/acp_adapter/tools.py @@ -2,6 +2,7 @@ from __future__ import annotations +import json import uuid from typing import Any, Dict, List, Optional @@ -96,6 +97,170 @@ def build_tool_title(tool_name: str, args: Dict[str, Any]) -> str: return tool_name +def _build_patch_mode_content(patch_text: str) -> List[Any]: + """Parse V4A patch mode input into ACP diff blocks when possible.""" + if not patch_text: + return [acp.tool_content(acp.text_block(""))] + + try: + from tools.patch_parser import OperationType, parse_v4a_patch + + operations, error = parse_v4a_patch(patch_text) + if error or not operations: + return [acp.tool_content(acp.text_block(patch_text))] + + content: List[Any] = [] + for op in operations: + if op.operation == OperationType.UPDATE: + old_chunks: list[str] = [] + new_chunks: list[str] = [] + for hunk in op.hunks: + old_lines = [line.content for line in hunk.lines if line.prefix in (" ", "-")] + new_lines = [line.content for line in hunk.lines if line.prefix in (" ", "+")] + if old_lines or new_lines: + old_chunks.append("\n".join(old_lines)) + new_chunks.append("\n".join(new_lines)) + + old_text = "\n...\n".join(chunk for chunk in old_chunks if chunk) + new_text = "\n...\n".join(chunk for chunk in new_chunks if chunk) + if old_text or new_text: + content.append( + acp.tool_diff_content( + path=op.file_path, + old_text=old_text or None, + new_text=new_text or "", + ) + ) + continue + + if op.operation == OperationType.ADD: + added_lines = [line.content for hunk in op.hunks for line in hunk.lines if line.prefix == "+"] + content.append( + acp.tool_diff_content( + path=op.file_path, + new_text="\n".join(added_lines), + ) + ) + continue + + if op.operation == OperationType.DELETE: + content.append( + acp.tool_diff_content( + path=op.file_path, + old_text=f"Delete file: {op.file_path}", + new_text="", + ) + ) + continue + + if op.operation == OperationType.MOVE: + content.append( + acp.tool_content(acp.text_block(f"Move file: {op.file_path} -> {op.new_path}")) + ) + + return content or [acp.tool_content(acp.text_block(patch_text))] + except Exception: + return [acp.tool_content(acp.text_block(patch_text))] + + +def _strip_diff_prefix(path: str) -> str: + raw = str(path or "").strip() + if raw.startswith(("a/", "b/")): + return raw[2:] + return raw + + +def _parse_unified_diff_content(diff_text: str) -> List[Any]: + """Convert unified diff text into ACP diff content blocks.""" + if not diff_text: + return [] + + content: List[Any] = [] + current_old_path: Optional[str] = None + current_new_path: Optional[str] = None + old_lines: list[str] = [] + new_lines: list[str] = [] + + def _flush() -> None: + nonlocal current_old_path, current_new_path, old_lines, new_lines + if current_old_path is None and current_new_path is None: + return + path = current_new_path if current_new_path and current_new_path != "/dev/null" else current_old_path + if not path or path == "/dev/null": + current_old_path = None + current_new_path = None + old_lines = [] + new_lines = [] + return + content.append( + acp.tool_diff_content( + path=_strip_diff_prefix(path), + old_text="\n".join(old_lines) if old_lines else None, + new_text="\n".join(new_lines), + ) + ) + current_old_path = None + current_new_path = None + old_lines = [] + new_lines = [] + + for line in diff_text.splitlines(): + if line.startswith("--- "): + _flush() + current_old_path = line[4:].strip() + continue + if line.startswith("+++ "): + current_new_path = line[4:].strip() + continue + if line.startswith("@@"): + continue + if current_old_path is None and current_new_path is None: + continue + if line.startswith("+"): + new_lines.append(line[1:]) + elif line.startswith("-"): + old_lines.append(line[1:]) + elif line.startswith(" "): + shared = line[1:] + old_lines.append(shared) + new_lines.append(shared) + + _flush() + return content + + +def _build_tool_complete_content( + tool_name: str, + result: Optional[str], + *, + function_args: Optional[Dict[str, Any]] = None, + snapshot: Any = None, +) -> List[Any]: + """Build structured ACP completion content, falling back to plain text.""" + display_result = result or "" + if len(display_result) > 5000: + display_result = display_result[:4900] + f"\n... ({len(result)} chars total, truncated)" + + if tool_name in {"write_file", "patch", "skill_manage"}: + try: + from agent.display import extract_edit_diff + + diff_text = extract_edit_diff( + tool_name, + result, + function_args=function_args, + snapshot=snapshot, + ) + if isinstance(diff_text, str) and diff_text.strip(): + diff_content = _parse_unified_diff_content(diff_text) + if diff_content: + return diff_content + except Exception: + pass + + return [acp.tool_content(acp.text_block(display_result))] + + # --------------------------------------------------------------------------- # Build ACP content objects for tool-call events # --------------------------------------------------------------------------- @@ -119,9 +284,8 @@ def build_tool_start( new = arguments.get("new_string", "") content = [acp.tool_diff_content(path=path, new_text=new, old_text=old)] else: - # Patch mode — show the patch content as text patch_text = arguments.get("patch", "") - content = [acp.tool_content(acp.text_block(patch_text))] + content = _build_patch_mode_content(patch_text) return acp.start_tool_call( tool_call_id, title, kind=kind, content=content, locations=locations, raw_input=arguments, @@ -178,16 +342,17 @@ def build_tool_complete( tool_call_id: str, tool_name: str, result: Optional[str] = None, + function_args: Optional[Dict[str, Any]] = None, + snapshot: Any = None, ) -> ToolCallProgress: """Create a ToolCallUpdate (progress) event for a completed tool call.""" kind = get_tool_kind(tool_name) - - # Truncate very large results for the UI - display_result = result or "" - if len(display_result) > 5000: - display_result = display_result[:4900] + f"\n... ({len(result)} chars total, truncated)" - - content = [acp.tool_content(acp.text_block(display_result))] + content = _build_tool_complete_content( + tool_name, + result, + function_args=function_args, + snapshot=snapshot, + ) return acp.update_tool_call( tool_call_id, kind=kind, diff --git a/agent/account_usage.py b/agent/account_usage.py new file mode 100644 index 000000000000..0e9562dcc9e7 --- /dev/null +++ b/agent/account_usage.py @@ -0,0 +1,326 @@ +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timezone +from typing import Any, Optional + +import httpx + +from agent.anthropic_adapter import _is_oauth_token, resolve_anthropic_token +from hermes_cli.auth import _read_codex_tokens, resolve_codex_runtime_credentials +from hermes_cli.runtime_provider import resolve_runtime_provider + + +def _utc_now() -> datetime: + return datetime.now(timezone.utc) + + +@dataclass(frozen=True) +class AccountUsageWindow: + label: str + used_percent: Optional[float] = None + reset_at: Optional[datetime] = None + detail: Optional[str] = None + + +@dataclass(frozen=True) +class AccountUsageSnapshot: + provider: str + source: str + fetched_at: datetime + title: str = "Account limits" + plan: Optional[str] = None + windows: tuple[AccountUsageWindow, ...] = () + details: tuple[str, ...] = () + unavailable_reason: Optional[str] = None + + @property + def available(self) -> bool: + return bool(self.windows or self.details) and not self.unavailable_reason + + +def _title_case_slug(value: Optional[str]) -> Optional[str]: + cleaned = str(value or "").strip() + if not cleaned: + return None + return cleaned.replace("_", " ").replace("-", " ").title() + + +def _parse_dt(value: Any) -> Optional[datetime]: + if value in (None, ""): + return None + if isinstance(value, (int, float)): + return datetime.fromtimestamp(float(value), tz=timezone.utc) + if isinstance(value, str): + text = value.strip() + if not text: + return None + if text.endswith("Z"): + text = text[:-1] + "+00:00" + try: + dt = datetime.fromisoformat(text) + return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc) + except ValueError: + return None + return None + + +def _format_reset(dt: Optional[datetime]) -> str: + if not dt: + return "unknown" + local_dt = dt.astimezone() + delta = dt - _utc_now() + total_seconds = int(delta.total_seconds()) + if total_seconds <= 0: + return f"now ({local_dt.strftime('%Y-%m-%d %H:%M %Z')})" + hours, rem = divmod(total_seconds, 3600) + minutes = rem // 60 + if hours >= 24: + days, hours = divmod(hours, 24) + rel = f"in {days}d {hours}h" + elif hours > 0: + rel = f"in {hours}h {minutes}m" + else: + rel = f"in {minutes}m" + return f"{rel} ({local_dt.strftime('%Y-%m-%d %H:%M %Z')})" + + +def render_account_usage_lines(snapshot: Optional[AccountUsageSnapshot], *, markdown: bool = False) -> list[str]: + if not snapshot: + return [] + header = f"📈 {'**' if markdown else ''}{snapshot.title}{'**' if markdown else ''}" + lines = [header] + if snapshot.plan: + lines.append(f"Provider: {snapshot.provider} ({snapshot.plan})") + else: + lines.append(f"Provider: {snapshot.provider}") + for window in snapshot.windows: + if window.used_percent is None: + base = f"{window.label}: unavailable" + else: + remaining = max(0, round(100 - float(window.used_percent))) + used = max(0, round(float(window.used_percent))) + base = f"{window.label}: {remaining}% remaining ({used}% used)" + if window.reset_at: + base += f" • resets {_format_reset(window.reset_at)}" + elif window.detail: + base += f" • {window.detail}" + lines.append(base) + for detail in snapshot.details: + lines.append(detail) + if snapshot.unavailable_reason: + lines.append(f"Unavailable: {snapshot.unavailable_reason}") + return lines + + +def _resolve_codex_usage_url(base_url: str) -> str: + normalized = (base_url or "").strip().rstrip("/") + if not normalized: + normalized = "https://chatgpt.com/backend-api/codex" + if normalized.endswith("/codex"): + normalized = normalized[: -len("/codex")] + if "/backend-api" in normalized: + return normalized + "/wham/usage" + return normalized + "/api/codex/usage" + + +def _fetch_codex_account_usage() -> Optional[AccountUsageSnapshot]: + creds = resolve_codex_runtime_credentials(refresh_if_expiring=True) + token_data = _read_codex_tokens() + tokens = token_data.get("tokens") or {} + account_id = str(tokens.get("account_id", "") or "").strip() or None + headers = { + "Authorization": f"Bearer {creds['api_key']}", + "Accept": "application/json", + "User-Agent": "codex-cli", + } + if account_id: + headers["ChatGPT-Account-Id"] = account_id + with httpx.Client(timeout=15.0) as client: + response = client.get(_resolve_codex_usage_url(creds.get("base_url", "")), headers=headers) + response.raise_for_status() + payload = response.json() or {} + rate_limit = payload.get("rate_limit") or {} + windows: list[AccountUsageWindow] = [] + for key, label in (("primary_window", "Session"), ("secondary_window", "Weekly")): + window = rate_limit.get(key) or {} + used = window.get("used_percent") + if used is None: + continue + windows.append( + AccountUsageWindow( + label=label, + used_percent=float(used), + reset_at=_parse_dt(window.get("reset_at")), + ) + ) + details: list[str] = [] + credits = payload.get("credits") or {} + if credits.get("has_credits"): + balance = credits.get("balance") + if isinstance(balance, (int, float)): + details.append(f"Credits balance: ${float(balance):.2f}") + elif credits.get("unlimited"): + details.append("Credits balance: unlimited") + return AccountUsageSnapshot( + provider="openai-codex", + source="usage_api", + fetched_at=_utc_now(), + plan=_title_case_slug(payload.get("plan_type")), + windows=tuple(windows), + details=tuple(details), + ) + + +def _fetch_anthropic_account_usage() -> Optional[AccountUsageSnapshot]: + token = (resolve_anthropic_token() or "").strip() + if not token: + return None + if not _is_oauth_token(token): + return AccountUsageSnapshot( + provider="anthropic", + source="oauth_usage_api", + fetched_at=_utc_now(), + unavailable_reason="Anthropic account limits are only available for OAuth-backed Claude accounts.", + ) + headers = { + "Authorization": f"Bearer {token}", + "Accept": "application/json", + "Content-Type": "application/json", + "anthropic-beta": "oauth-2025-04-20", + "User-Agent": "claude-code/2.1.0", + } + with httpx.Client(timeout=15.0) as client: + response = client.get("https://api.anthropic.com/api/oauth/usage", headers=headers) + response.raise_for_status() + payload = response.json() or {} + windows: list[AccountUsageWindow] = [] + mapping = ( + ("five_hour", "Current session"), + ("seven_day", "Current week"), + ("seven_day_opus", "Opus week"), + ("seven_day_sonnet", "Sonnet week"), + ) + for key, label in mapping: + window = payload.get(key) or {} + util = window.get("utilization") + if util is None: + continue + used = float(util) * 100 if float(util) <= 1 else float(util) + windows.append( + AccountUsageWindow( + label=label, + used_percent=used, + reset_at=_parse_dt(window.get("resets_at")), + ) + ) + details: list[str] = [] + extra = payload.get("extra_usage") or {} + if extra.get("is_enabled"): + used_credits = extra.get("used_credits") + monthly_limit = extra.get("monthly_limit") + currency = extra.get("currency") or "USD" + if isinstance(used_credits, (int, float)) and isinstance(monthly_limit, (int, float)): + details.append( + f"Extra usage: {used_credits:.2f} / {monthly_limit:.2f} {currency}" + ) + return AccountUsageSnapshot( + provider="anthropic", + source="oauth_usage_api", + fetched_at=_utc_now(), + windows=tuple(windows), + details=tuple(details), + ) + + +def _fetch_openrouter_account_usage(base_url: Optional[str], api_key: Optional[str]) -> Optional[AccountUsageSnapshot]: + runtime = resolve_runtime_provider( + requested="openrouter", + explicit_base_url=base_url, + explicit_api_key=api_key, + ) + token = str(runtime.get("api_key", "") or "").strip() + if not token: + return None + normalized = str(runtime.get("base_url", "") or "").rstrip("/") + credits_url = f"{normalized}/credits" + key_url = f"{normalized}/key" + headers = { + "Authorization": f"Bearer {token}", + "Accept": "application/json", + } + with httpx.Client(timeout=10.0) as client: + credits_resp = client.get(credits_url, headers=headers) + credits_resp.raise_for_status() + credits = (credits_resp.json() or {}).get("data") or {} + try: + key_resp = client.get(key_url, headers=headers) + key_resp.raise_for_status() + key_data = (key_resp.json() or {}).get("data") or {} + except Exception: + key_data = {} + total_credits = float(credits.get("total_credits") or 0.0) + total_usage = float(credits.get("total_usage") or 0.0) + details = [f"Credits balance: ${max(0.0, total_credits - total_usage):.2f}"] + windows: list[AccountUsageWindow] = [] + limit = key_data.get("limit") + limit_remaining = key_data.get("limit_remaining") + limit_reset = str(key_data.get("limit_reset") or "").strip() + usage = key_data.get("usage") + if ( + isinstance(limit, (int, float)) + and float(limit) > 0 + and isinstance(limit_remaining, (int, float)) + and 0 <= float(limit_remaining) <= float(limit) + ): + limit_value = float(limit) + remaining_value = float(limit_remaining) + used_percent = ((limit_value - remaining_value) / limit_value) * 100 + detail_parts = [f"${remaining_value:.2f} of ${limit_value:.2f} remaining"] + if limit_reset: + detail_parts.append(f"resets {limit_reset}") + windows.append( + AccountUsageWindow( + label="API key quota", + used_percent=used_percent, + detail=" • ".join(detail_parts), + ) + ) + if isinstance(usage, (int, float)): + usage_parts = [f"API key usage: ${float(usage):.2f} total"] + for value, label in ( + (key_data.get("usage_daily"), "today"), + (key_data.get("usage_weekly"), "this week"), + (key_data.get("usage_monthly"), "this month"), + ): + if isinstance(value, (int, float)) and float(value) > 0: + usage_parts.append(f"${float(value):.2f} {label}") + details.append(" • ".join(usage_parts)) + return AccountUsageSnapshot( + provider="openrouter", + source="credits_api", + fetched_at=_utc_now(), + windows=tuple(windows), + details=tuple(details), + ) + + +def fetch_account_usage( + provider: Optional[str], + *, + base_url: Optional[str] = None, + api_key: Optional[str] = None, +) -> Optional[AccountUsageSnapshot]: + normalized = str(provider or "").strip().lower() + if normalized in {"", "auto", "custom"}: + return None + try: + if normalized == "openai-codex": + return _fetch_codex_account_usage() + if normalized == "anthropic": + return _fetch_anthropic_account_usage() + if normalized == "openrouter": + return _fetch_openrouter_account_usage(base_url, api_key) + except Exception: + return None + return None diff --git a/agent/anthropic_adapter.py b/agent/anthropic_adapter.py index 830c0f4de707..ea09c11ea4ff 100644 --- a/agent/anthropic_adapter.py +++ b/agent/anthropic_adapter.py @@ -17,8 +17,8 @@ from pathlib import Path from hermes_constants import get_hermes_home -from types import SimpleNamespace from typing import Any, Dict, List, Optional, Tuple +from utils import normalize_proxy_env_vars try: import anthropic as _anthropic_sdk @@ -28,19 +28,45 @@ logger = logging.getLogger(__name__) THINKING_BUDGET = {"xhigh": 32000, "high": 16000, "medium": 8000, "low": 4000} +# Hermes effort → Anthropic adaptive-thinking effort (output_config.effort). +# Anthropic exposes 5 levels on 4.7+: low, medium, high, xhigh, max. +# Opus/Sonnet 4.6 only expose 4 levels: low, medium, high, max — no xhigh. +# We preserve xhigh as xhigh on 4.7+ (the recommended default for coding/ +# agentic work) and downgrade it to max on pre-4.7 adaptive models (which +# is the strongest level they accept). "minimal" is a legacy alias that +# maps to low on every model. See: +# https://platform.claude.com/docs/en/about-claude/models/migration-guide ADAPTIVE_EFFORT_MAP = { - "xhigh": "max", - "high": "high", - "medium": "medium", - "low": "low", + "max": "max", + "xhigh": "xhigh", + "high": "high", + "medium": "medium", + "low": "low", "minimal": "low", } +# Models that accept the "xhigh" output_config.effort level. Opus 4.7 added +# xhigh as a distinct level between high and max; older adaptive-thinking +# models (4.6) reject it with a 400. Keep this substring list in sync with +# the Anthropic migration guide as new model families ship. +_XHIGH_EFFORT_SUBSTRINGS = ("4-7", "4.7") + +# Models where extended thinking is deprecated/removed (4.6+ behavior: adaptive +# is the only supported mode; 4.7 additionally forbids manual thinking entirely +# and drops temperature/top_p/top_k). +_ADAPTIVE_THINKING_SUBSTRINGS = ("4-6", "4.6", "4-7", "4.7") + +# Models where temperature/top_p/top_k return 400 if set to non-default values. +# This is the Opus 4.7 contract; future 4.x+ models are expected to follow it. +_NO_SAMPLING_PARAMS_SUBSTRINGS = ("4-7", "4.7") + # ── Max output token limits per Anthropic model ─────────────────────── # Source: Anthropic docs + Cline model catalog. Anthropic's API requires # max_tokens as a mandatory field. Previously we hardcoded 16384, which # starves thinking-enabled models (thinking tokens count toward the limit). _ANTHROPIC_OUTPUT_LIMITS = { + # Claude 4.7 + "claude-opus-4-7": 128_000, # Claude 4.6 "claude-opus-4-6": 128_000, "claude-sonnet-4-6": 64_000, @@ -90,12 +116,95 @@ def _get_anthropic_max_output(model: str) -> int: return best_val +def _resolve_positive_anthropic_max_tokens(value) -> Optional[int]: + """Return ``value`` floored to a positive int, or ``None`` if it is not a + finite positive number. Ported from openclaw/openclaw#66664. + + Anthropic's Messages API rejects ``max_tokens`` values that are 0, + negative, non-integer, or non-finite with HTTP 400. Python's ``or`` + idiom (``max_tokens or fallback``) correctly catches ``0`` but lets + negative ints and fractional floats (``-1``, ``0.5``) through to the + API, producing a user-visible failure instead of a local error. + """ + # Booleans are a subclass of int — exclude explicitly so ``True`` doesn't + # silently become 1 and ``False`` doesn't become 0. + if isinstance(value, bool): + return None + if not isinstance(value, (int, float)): + return None + try: + import math + if not math.isfinite(value): + return None + except Exception: + return None + floored = int(value) # truncates toward zero for floats + return floored if floored > 0 else None + + +def _resolve_anthropic_messages_max_tokens( + requested, + model: str, + context_length: Optional[int] = None, +) -> int: + """Resolve the ``max_tokens`` budget for an Anthropic Messages call. + + Prefers ``requested`` when it is a positive finite number; otherwise + falls back to the model's output ceiling. Raises ``ValueError`` if no + positive budget can be resolved (should not happen with current model + table defaults, but guards against a future regression where + ``_get_anthropic_max_output`` could return ``0``). + + Separately, callers apply a context-window clamp — this resolver does + not, to keep the positive-value contract independent of endpoint + specifics. + + Ported from openclaw/openclaw#66664 (resolveAnthropicMessagesMaxTokens). + """ + resolved = _resolve_positive_anthropic_max_tokens(requested) + if resolved is not None: + return resolved + fallback = _get_anthropic_max_output(model) + if fallback > 0: + return fallback + raise ValueError( + f"Anthropic Messages adapter requires a positive max_tokens value for " + f"model {model!r}; got {requested!r} and no model default resolved." + ) + + def _supports_adaptive_thinking(model: str) -> bool: - """Return True for Claude 4.6 models that support adaptive thinking.""" - return any(v in model for v in ("4-6", "4.6")) + """Return True for Claude 4.6+ models that support adaptive thinking.""" + return any(v in model for v in _ADAPTIVE_THINKING_SUBSTRINGS) + + +def _supports_xhigh_effort(model: str) -> bool: + """Return True for models that accept the 'xhigh' adaptive effort level. + Opus 4.7 introduced xhigh as a distinct level between high and max. + Pre-4.7 adaptive models (Opus/Sonnet 4.6) only accept low/medium/high/max + and reject xhigh with an HTTP 400. Callers should downgrade xhigh→max + when this returns False. + """ + return any(v in model for v in _XHIGH_EFFORT_SUBSTRINGS) + + +def _forbids_sampling_params(model: str) -> bool: + """Return True for models that 400 on any non-default temperature/top_p/top_k. + + Opus 4.7 explicitly rejects sampling parameters; later Claude releases are + expected to follow suit. Callers should omit these fields entirely rather + than passing zero/default values (the API rejects anything non-null). + """ + return any(v in model for v in _NO_SAMPLING_PARAMS_SUBSTRINGS) -# Beta headers for enhanced features (sent with ALL auth types) + +# Beta headers for enhanced features (sent with ALL auth types). +# As of Opus 4.7 (2026-04-16), both of these are GA on Claude 4.6+ — the +# beta headers are still accepted (harmless no-op) but not required. Kept +# here so older Claude (4.5, 4.1) + third-party Anthropic-compat endpoints +# that still gate on the headers continue to get the enhanced features. +# Migration guide: remove these if you no longer support ≤4.5 models. _COMMON_BETAS = [ "interleaved-thinking-2025-05-14", "fine-grained-tool-streaming-2025-05-14", @@ -213,6 +322,14 @@ def _is_third_party_anthropic_endpoint(base_url: str | None) -> bool: return True # Any other endpoint is a third-party proxy +def _is_kimi_coding_endpoint(base_url: str | None) -> bool: + """Return True for Kimi's /coding endpoint that requires claude-code UA.""" + normalized = _normalize_base_url_text(base_url) + if not normalized: + return False + return normalized.rstrip("/").lower().startswith("https://api.kimi.com/coding") + + def _requires_bearer_auth(base_url: str | None) -> bool: """Return True for Anthropic-compatible providers that require Bearer auth. @@ -240,9 +357,15 @@ def _common_betas_for_base_url(base_url: str | None) -> list[str]: return _COMMON_BETAS -def build_anthropic_client(api_key: str, base_url: str = None): +def build_anthropic_client(api_key: str, base_url: str = None, timeout: float = None): """Create an Anthropic client, auto-detecting setup-tokens vs API keys. + If *timeout* is provided it overrides the default 900s read timeout. The + connect timeout stays at 10s. Callers pass this from the per-provider / + per-model ``request_timeout_seconds`` config so Anthropic-native and + Anthropic-compatible providers respect the same knob as OpenAI-wire + providers. + Returns an anthropic.Anthropic instance. """ if _anthropic_sdk is None: @@ -250,19 +373,32 @@ def build_anthropic_client(api_key: str, base_url: str = None): "The 'anthropic' package is required for the Anthropic provider. " "Install it with: pip install 'anthropic>=0.39.0'" ) + + normalize_proxy_env_vars() + from httpx import Timeout normalized_base_url = _normalize_base_url_text(base_url) + _read_timeout = timeout if (isinstance(timeout, (int, float)) and timeout > 0) else 900.0 kwargs = { - "timeout": Timeout(timeout=900.0, connect=10.0), + "timeout": Timeout(timeout=float(_read_timeout), connect=10.0), } if normalized_base_url: kwargs["base_url"] = normalized_base_url common_betas = _common_betas_for_base_url(normalized_base_url) - if _requires_bearer_auth(normalized_base_url): + if _is_kimi_coding_endpoint(base_url): + # Kimi's /coding endpoint requires User-Agent: claude-code/0.1.0 + # to be recognized as a valid Coding Agent. Without it, returns 403. + # Check this BEFORE _requires_bearer_auth since both match api.kimi.com/coding. + kwargs["api_key"] = api_key + kwargs["default_headers"] = { + "User-Agent": "claude-code/0.1.0", + **( {"anthropic-beta": ",".join(common_betas)} if common_betas else {} ) + } + elif _requires_bearer_auth(normalized_base_url): # Some Anthropic-compatible providers (e.g. MiniMax) expect the API key in - # Authorization: Bearer even for regular API keys. Route those endpoints + # Authorization: Bearer *** for regular API keys. Route those endpoints # through auth_token so the SDK sends Bearer auth instead of x-api-key. # Check this before OAuth token shape detection because MiniMax secrets do # not use Anthropic's sk-ant-api prefix and would otherwise be misread as @@ -298,6 +434,33 @@ def build_anthropic_client(api_key: str, base_url: str = None): return _anthropic_sdk.Anthropic(**kwargs) +def build_anthropic_bedrock_client(region: str): + """Create an AnthropicBedrock client for Bedrock Claude models. + + Uses the Anthropic SDK's native Bedrock adapter, which provides full + Claude feature parity: prompt caching, thinking budgets, adaptive + thinking, fast mode — features not available via the Converse API. + + Auth uses the boto3 default credential chain (IAM roles, SSO, env vars). + """ + if _anthropic_sdk is None: + raise ImportError( + "The 'anthropic' package is required for the Bedrock provider. " + "Install it with: pip install 'anthropic>=0.39.0'" + ) + if not hasattr(_anthropic_sdk, "AnthropicBedrock"): + raise ImportError( + "anthropic.AnthropicBedrock not available. " + "Upgrade with: pip install 'anthropic>=0.39.0'" + ) + from httpx import Timeout + + return _anthropic_sdk.AnthropicBedrock( + aws_region=region, + timeout=Timeout(timeout=900.0, connect=10.0), + ) + + def read_claude_code_credentials() -> Optional[Dict[str, Any]]: """Read refreshable Claude Code OAuth credentials from ~/.claude/.credentials.json. @@ -976,6 +1139,31 @@ def convert_messages_to_anthropic( "name": fn.get("name", ""), "input": parsed_args, }) + # Kimi's /coding endpoint (Anthropic protocol) requires assistant + # tool-call messages to carry reasoning_content when thinking is + # enabled server-side. Preserve it as a thinking block so Kimi + # can validate the message history. See hermes-agent#13848. + # + # Accept empty string "" — _copy_reasoning_content_for_api() + # injects "" as a tier-3 fallback for Kimi tool-call messages + # that had no reasoning. Kimi requires the field to exist, even + # if empty. + # + # Prepend (not append): Anthropic protocol requires thinking + # blocks before text and tool_use blocks. + # + # Guard: only add when reasoning_details didn't already contribute + # thinking blocks. On native Anthropic, reasoning_details produces + # signed thinking blocks — adding another unsigned one from + # reasoning_content would create a duplicate (same text) that gets + # downgraded to a spurious text block on the last assistant message. + reasoning_content = m.get("reasoning_content") + _already_has_thinking = any( + isinstance(b, dict) and b.get("type") in ("thinking", "redacted_thinking") + for b in blocks + ) + if isinstance(reasoning_content, str) and not _already_has_thinking: + blocks.insert(0, {"type": "thinking", "thinking": reasoning_content}) # Anthropic rejects empty assistant content effective = blocks or content if not effective or effective == "": @@ -1131,6 +1319,7 @@ def convert_messages_to_anthropic( # cache markers can interfere with signature validation. _THINKING_TYPES = frozenset(("thinking", "redacted_thinking")) _is_third_party = _is_third_party_anthropic_endpoint(base_url) + _is_kimi = _is_kimi_coding_endpoint(base_url) last_assistant_idx = None for i in range(len(result) - 1, -1, -1): @@ -1142,7 +1331,25 @@ def convert_messages_to_anthropic( if m.get("role") != "assistant" or not isinstance(m.get("content"), list): continue - if _is_third_party or idx != last_assistant_idx: + if _is_kimi: + # Kimi's /coding endpoint enables thinking server-side and + # requires unsigned thinking blocks on replayed assistant + # tool-call messages. Strip signed Anthropic blocks (Kimi + # can't validate signatures) but preserve the unsigned ones + # we synthesised from reasoning_content above. + new_content = [] + for b in m["content"]: + if not isinstance(b, dict) or b.get("type") not in _THINKING_TYPES: + new_content.append(b) + continue + if b.get("signature") or b.get("data"): + # Anthropic-signed block — Kimi can't validate, strip + continue + # Unsigned thinking (synthesised from reasoning_content) — + # keep it: Kimi needs it for message-history validation. + new_content.append(b) + m["content"] = new_content or [{"type": "text", "text": "(empty)"}] + elif _is_third_party or idx != last_assistant_idx: # Third-party endpoint: strip ALL thinking blocks from every # assistant message — signatures are Anthropic-proprietary. # Direct Anthropic: strip from non-latest assistant messages only. @@ -1230,16 +1437,22 @@ def build_anthropic_kwargs( When *base_url* points to a third-party Anthropic-compatible endpoint, thinking block signatures are stripped (they are Anthropic-proprietary). - When *fast_mode* is True, adds ``speed: "fast"`` and the fast-mode beta - header for ~2.5x faster output throughput on Opus 4.6. Currently only - supported on native Anthropic endpoints (not third-party compatible ones). + When *fast_mode* is True, adds ``extra_body["speed"] = "fast"`` and the + fast-mode beta header for ~2.5x faster output throughput on Opus 4.6. + Currently only supported on native Anthropic endpoints (not third-party + compatible ones). """ system, anthropic_messages = convert_messages_to_anthropic(messages, base_url=base_url) anthropic_tools = convert_tools_to_anthropic(tools) if tools else [] model = normalize_model_name(model, preserve_dots=preserve_dots) # effective_max_tokens = output cap for this call (≠ total context window) - effective_max_tokens = max_tokens or _get_anthropic_max_output(model) + # Use the resolver helper so non-positive values (negative ints, + # fractional floats, NaN, non-numeric) fail locally with a clear error + # rather than 400-ing at the Anthropic API. See openclaw/openclaw#66664. + effective_max_tokens = _resolve_anthropic_messages_max_tokens( + max_tokens, model, context_length=context_length + ) # Clamp output cap to fit inside the total context window. # Only matters for small custom endpoints where context_length < native @@ -1313,18 +1526,45 @@ def build_anthropic_kwargs( kwargs["tool_choice"] = {"type": "tool", "name": tool_choice} # Map reasoning_config to Anthropic's thinking parameter. - # Claude 4.6 models use adaptive thinking + output_config.effort. + # Claude 4.6+ models use adaptive thinking + output_config.effort. # Older models use manual thinking with budget_tokens. # MiniMax Anthropic-compat endpoints support thinking (manual mode only, # not adaptive). Haiku does NOT support extended thinking — skip entirely. - if reasoning_config and isinstance(reasoning_config, dict): + # + # Kimi's /coding endpoint speaks the Anthropic Messages protocol but has + # its own thinking semantics: when ``thinking.enabled`` is sent, Kimi + # validates the message history and requires every prior assistant + # tool-call message to carry OpenAI-style ``reasoning_content``. The + # Anthropic path never populates that field, and + # ``convert_messages_to_anthropic`` strips all Anthropic thinking blocks + # on third-party endpoints — so the request fails with HTTP 400 + # "thinking is enabled but reasoning_content is missing in assistant + # tool call message at index N". Kimi's reasoning is driven server-side + # on the /coding route, so skip Anthropic's thinking parameter entirely + # for that host. (Kimi on chat_completions enables thinking via + # extra_body in the ChatCompletionsTransport — see #13503.) + # + # On 4.7+ the `thinking.display` field defaults to "omitted", which + # silently hides reasoning text that Hermes surfaces in its CLI. We + # request "summarized" so the reasoning blocks stay populated — matching + # 4.6 behavior and preserving the activity-feed UX during long tool runs. + _is_kimi_coding = _is_kimi_coding_endpoint(base_url) + if reasoning_config and isinstance(reasoning_config, dict) and not _is_kimi_coding: if reasoning_config.get("enabled") is not False and "haiku" not in model.lower(): effort = str(reasoning_config.get("effort", "medium")).lower() budget = THINKING_BUDGET.get(effort, 8000) if _supports_adaptive_thinking(model): - kwargs["thinking"] = {"type": "adaptive"} + kwargs["thinking"] = { + "type": "adaptive", + "display": "summarized", + } + adaptive_effort = ADAPTIVE_EFFORT_MAP.get(effort, "medium") + # Downgrade xhigh→max on models that don't list xhigh as a + # supported level (Opus/Sonnet 4.6). Opus 4.7+ keeps xhigh. + if adaptive_effort == "xhigh" and not _supports_xhigh_effort(model): + adaptive_effort = "max" kwargs["output_config"] = { - "effort": ADAPTIVE_EFFORT_MAP.get(effort, "medium") + "effort": adaptive_effort, } else: kwargs["thinking"] = {"type": "enabled", "budget_tokens": budget} @@ -1332,12 +1572,21 @@ def build_anthropic_kwargs( kwargs["temperature"] = 1 kwargs["max_tokens"] = max(effective_max_tokens, budget + 4096) + # ── Strip sampling params on 4.7+ ───────────────────────────────── + # Opus 4.7 rejects any non-default temperature/top_p/top_k with a 400. + # Callers (auxiliary_client, flush_memories, etc.) may set these for + # older models; drop them here as a safety net so upstream 4.6 → 4.7 + # migrations don't require coordinated edits everywhere. + if _forbids_sampling_params(model): + for _sampling_key in ("temperature", "top_p", "top_k"): + kwargs.pop(_sampling_key, None) + # ── Fast mode (Opus 4.6 only) ──────────────────────────────────── - # Adds speed:"fast" + the fast-mode beta header for ~2.5x output speed. - # Only for native Anthropic endpoints — third-party providers would - # reject the unknown beta header and speed parameter. + # Adds extra_body.speed="fast" + the fast-mode beta header for ~2.5x + # output speed. Only for native Anthropic endpoints — third-party + # providers would reject the unknown beta header and speed parameter. if fast_mode and not _is_third_party_anthropic_endpoint(base_url): - kwargs["speed"] = "fast" + kwargs.setdefault("extra_body", {})["speed"] = "fast" # Build extra_headers with ALL applicable betas (the per-request # extra_headers override the client-level anthropic-beta header). betas = list(_common_betas_for_base_url(base_url)) @@ -1349,62 +1598,4 @@ def build_anthropic_kwargs( return kwargs -def normalize_anthropic_response( - response, - strip_tool_prefix: bool = False, -) -> Tuple[SimpleNamespace, str]: - """Normalize Anthropic response to match the shape expected by AIAgent. - - Returns (assistant_message, finish_reason) where assistant_message has - .content, .tool_calls, and .reasoning attributes. - When *strip_tool_prefix* is True, removes the ``mcp_`` prefix that was - added to tool names for OAuth Claude Code compatibility. - """ - text_parts = [] - reasoning_parts = [] - reasoning_details = [] - tool_calls = [] - - for block in response.content: - if block.type == "text": - text_parts.append(block.text) - elif block.type == "thinking": - reasoning_parts.append(block.thinking) - block_dict = _to_plain_data(block) - if isinstance(block_dict, dict): - reasoning_details.append(block_dict) - elif block.type == "tool_use": - name = block.name - if strip_tool_prefix and name.startswith(_MCP_TOOL_PREFIX): - name = name[len(_MCP_TOOL_PREFIX):] - tool_calls.append( - SimpleNamespace( - id=block.id, - type="function", - function=SimpleNamespace( - name=name, - arguments=json.dumps(block.input), - ), - ) - ) - - # Map Anthropic stop_reason to OpenAI finish_reason - stop_reason_map = { - "end_turn": "stop", - "tool_use": "tool_calls", - "max_tokens": "length", - "stop_sequence": "stop", - } - finish_reason = stop_reason_map.get(response.stop_reason, "stop") - - return ( - SimpleNamespace( - content="\n".join(text_parts) if text_parts else None, - tool_calls=tool_calls or None, - reasoning="\n\n".join(reasoning_parts) if reasoning_parts else None, - reasoning_content=None, - reasoning_details=reasoning_details or None, - ), - finish_reason, - ) diff --git a/agent/auxiliary_client.py b/agent/auxiliary_client.py index 5016662d5815..e812a337f5d9 100644 --- a/agent/auxiliary_client.py +++ b/agent/auxiliary_client.py @@ -48,6 +48,7 @@ from agent.credential_pool import load_pool from hermes_cli.config import get_hermes_home from hermes_constants import OPENROUTER_BASE_URL +from utils import base_url_host_matches, base_url_hostname, normalize_proxy_env_vars logger = logging.getLogger(__name__) @@ -58,6 +59,9 @@ "google": "gemini", "google-gemini": "gemini", "google-ai-studio": "gemini", + "x-ai": "xai", + "x.ai": "xai", + "grok": "xai", "glm": "zai", "z-ai": "zai", "z.ai": "zai", @@ -91,11 +95,46 @@ def _normalize_aux_provider(provider: Optional[str]) -> str: return "custom" return _PROVIDER_ALIASES.get(normalized, normalized) + +# Sentinel: when returned by _fixed_temperature_for_model(), callers must +# strip the ``temperature`` key from API kwargs entirely so the provider's +# server-side default applies. Kimi/Moonshot models manage temperature +# internally — sending *any* value (even the "correct" one) can conflict +# with gateway-side mode selection (thinking → 1.0, non-thinking → 0.6). +OMIT_TEMPERATURE: object = object() + + +def _is_kimi_model(model: Optional[str]) -> bool: + """True for any Kimi / Moonshot model that manages temperature server-side.""" + bare = (model or "").strip().lower().rsplit("/", 1)[-1] + return bare.startswith("kimi-") or bare == "kimi" + + +def _fixed_temperature_for_model( + model: Optional[str], + base_url: Optional[str] = None, +) -> "Optional[float] | object": + """Return a temperature directive for models with strict contracts. + + Returns: + ``OMIT_TEMPERATURE`` — caller must remove the ``temperature`` key so the + provider chooses its own default. Used for all Kimi / Moonshot + models whose gateway selects temperature server-side. + ``float`` — a specific value the caller must use (reserved for future + models with fixed-temperature contracts). + ``None`` — no override; caller should use its own default. + """ + if _is_kimi_model(model): + logger.debug("Omitting temperature for Kimi model %r (server-managed)", model) + return OMIT_TEMPERATURE + return None + # Default auxiliary models for direct API-key providers (cheap/fast for side tasks) _API_KEY_PROVIDER_AUX_MODELS: Dict[str, str] = { "gemini": "gemini-3-flash-preview", "zai": "glm-4.5-flash", "kimi-coding": "kimi-k2-turbo-preview", + "stepfun": "step-3.5-flash", "kimi-coding-cn": "kimi-k2-turbo-preview", "minimax": "MiniMax-M2.7", "minimax-cn": "MiniMax-M2.7", @@ -104,6 +143,7 @@ def _normalize_aux_provider(provider: Optional[str]) -> str: "opencode-zen": "gemini-3-flash", "opencode-go": "glm-5", "kilocode": "google/gemini-3-flash-preview", + "ollama-cloud": "nemotron-3-nano:30b", } # Vision-specific model overrides for direct providers. @@ -111,7 +151,8 @@ def _normalize_aux_provider(provider: Optional[str]) -> str: # differs from their main chat model, map it here. The vision auto-detect # "exotic provider" branch checks this before falling back to the main model. _PROVIDER_VISION_MODELS: Dict[str, str] = { - "xiaomi": "mimo-v2-omni", + "xiaomi": "mimo-v2.5", + "zai": "glm-5v-turbo", } # OpenRouter app attribution headers @@ -121,6 +162,16 @@ def _normalize_aux_provider(provider: Optional[str]) -> str: "X-OpenRouter-Categories": "productivity,cli-agent", } +# Vercel AI Gateway app attribution headers. HTTP-Referer maps to +# referrerUrl and X-Title maps to appName in the gateway's analytics. +from hermes_cli import __version__ as _HERMES_VERSION + +_AI_GATEWAY_HEADERS = { + "HTTP-Referer": "https://hermes-agent.nousresearch.com", + "X-Title": "Hermes Agent", + "User-Agent": f"HermesAgent/{_HERMES_VERSION}", +} + # Nous Portal extra_body for product attribution. # Callers should pass this as extra_body in chat.completions.create() # when the auxiliary client is backed by Nous Portal. @@ -132,8 +183,6 @@ def _normalize_aux_provider(provider: Optional[str]) -> str: # Default auxiliary models per provider _OPENROUTER_MODEL = "google/gemini-3-flash-preview" _NOUS_MODEL = "google/gemini-3-flash-preview" -_NOUS_FREE_TIER_VISION_MODEL = "xiaomi/mimo-v2-omni" -_NOUS_FREE_TIER_AUX_MODEL = "xiaomi/mimo-v2-pro" _NOUS_DEFAULT_BASE_URL = "https://inference-api.nousresearch.com/v1" _ANTHROPIC_DEFAULT_BASE_URL = "https://api.anthropic.com" _AUTH_JSON_PATH = get_hermes_home() / "auth.json" @@ -147,6 +196,45 @@ def _normalize_aux_provider(provider: Optional[str]) -> str: _CODEX_AUX_BASE_URL = "https://chatgpt.com/backend-api/codex" +def _codex_cloudflare_headers(access_token: str) -> Dict[str, str]: + """Headers required to avoid Cloudflare 403s on chatgpt.com/backend-api/codex. + + The Cloudflare layer in front of the Codex endpoint whitelists a small set of + first-party originators (``codex_cli_rs``, ``codex_vscode``, ``codex_sdk_ts``, + anything starting with ``Codex``). Requests from non-residential IPs (VPS, + server-hosted agents) that don't advertise an allowed originator are served + a 403 with ``cf-mitigated: challenge`` regardless of auth correctness. + + We pin ``originator: codex_cli_rs`` to match the upstream codex-rs CLI, set + ``User-Agent`` to a codex_cli_rs-shaped string (beats SDK fingerprinting), + and extract ``ChatGPT-Account-ID`` (canonical casing, from codex-rs + ``auth.rs``) out of the OAuth JWT's ``chatgpt_account_id`` claim. + + Malformed tokens are tolerated — we drop the account-ID header rather than + raise, so a bad token still surfaces as an auth error (401) instead of a + crash at client construction. + """ + headers = { + "User-Agent": "codex_cli_rs/0.0.0 (Hermes Agent)", + "originator": "codex_cli_rs", + } + if not isinstance(access_token, str) or not access_token.strip(): + return headers + try: + import base64 + parts = access_token.split(".") + if len(parts) < 2: + return headers + payload_b64 = parts[1] + "=" * (-len(parts[1]) % 4) + claims = json.loads(base64.urlsafe_b64decode(payload_b64)) + acct_id = claims.get("https://api.openai.com/auth", {}).get("chatgpt_account_id") + if isinstance(acct_id, str) and acct_id: + headers["ChatGPT-Account-ID"] = acct_id + except Exception: + pass + return headers + + def _to_openai_base_url(base_url: str) -> str: """Normalize an Anthropic-style base URL to OpenAI-compatible format. @@ -485,7 +573,8 @@ def __init__(self, real_client: Any, model: str, is_oauth: bool = False): self._is_oauth = is_oauth def create(self, **kwargs) -> Any: - from agent.anthropic_adapter import build_anthropic_kwargs, normalize_anthropic_response + from agent.anthropic_adapter import build_anthropic_kwargs + from agent.transports import get_transport messages = kwargs.get("messages", []) model = kwargs.get("model", self._model) @@ -513,11 +602,28 @@ def create(self, **kwargs) -> Any: tool_choice=normalized_tool_choice, is_oauth=self._is_oauth, ) + # Opus 4.7+ rejects any non-default temperature/top_p/top_k; only set + # temperature for models that still accept it. build_anthropic_kwargs + # additionally strips these keys as a safety net — keep both layers. if temperature is not None: - anthropic_kwargs["temperature"] = temperature + from agent.anthropic_adapter import _forbids_sampling_params + if not _forbids_sampling_params(model): + anthropic_kwargs["temperature"] = temperature response = self._client.messages.create(**anthropic_kwargs) - assistant_message, finish_reason = normalize_anthropic_response(response) + _transport = get_transport("anthropic_messages") + _nr = _transport.normalize_response( + response, strip_tool_prefix=self._is_oauth + ) + + # ToolCall already duck-types as OpenAI shape (.type, .function.name, + # .function.arguments) via properties, so no wrapping needed. + assistant_message = SimpleNamespace( + content=_nr.content, + tool_calls=_nr.tool_calls, + reasoning=_nr.reasoning, + ) + finish_reason = _nr.finish_reason usage = None if hasattr(response, "usage") and response.usage: @@ -634,6 +740,33 @@ def _nous_base_url() -> str: return os.getenv("NOUS_INFERENCE_BASE_URL", _NOUS_DEFAULT_BASE_URL) +def _resolve_nous_runtime_api(*, force_refresh: bool = False) -> Optional[tuple[str, str]]: + """Return fresh Nous runtime credentials when available. + + This mirrors the main agent's 401 recovery path and keeps auxiliary + clients aligned with the singleton auth store + mint flow instead of + relying only on whatever raw tokens happen to be sitting in auth.json + or the credential pool. + """ + try: + from hermes_cli.auth import resolve_nous_runtime_credentials + + creds = resolve_nous_runtime_credentials( + min_key_ttl_seconds=max(60, int(os.getenv("HERMES_NOUS_MIN_KEY_TTL_SECONDS", "1800"))), + timeout_seconds=float(os.getenv("HERMES_NOUS_TIMEOUT_SECONDS", "15")), + force_mint=force_refresh, + ) + except Exception as exc: + logger.debug("Auxiliary Nous runtime credential resolution failed: %s", exc) + return None + + api_key = str(creds.get("api_key") or "").strip() + base_url = str(creds.get("base_url") or "").strip().rstrip("/") + if not api_key or not base_url: + return None + return api_key, base_url + + def _read_codex_access_token() -> Optional[str]: """Read a valid, non-expired Codex OAuth access token from Hermes auth store. @@ -717,10 +850,15 @@ def _resolve_api_key_provider() -> Tuple[Optional[OpenAI], Optional[str]]: if model is None: continue # skip provider if we don't know a valid aux model logger.debug("Auxiliary text client: %s (%s) via pool", pconfig.name, model) + if provider_id == "gemini": + from agent.gemini_native_adapter import GeminiNativeClient, is_native_gemini_base_url + + if is_native_gemini_base_url(base_url): + return GeminiNativeClient(api_key=api_key, base_url=base_url), model extra = {} - if "api.kimi.com" in base_url.lower(): - extra["default_headers"] = {"User-Agent": "KimiCLI/1.30.0"} - elif "api.githubcopilot.com" in base_url.lower(): + if base_url_host_matches(base_url, "api.kimi.com"): + extra["default_headers"] = {"User-Agent": "claude-code/0.1.0"} + elif base_url_host_matches(base_url, "api.githubcopilot.com"): from hermes_cli.models import copilot_default_headers extra["default_headers"] = copilot_default_headers() @@ -738,10 +876,15 @@ def _resolve_api_key_provider() -> Tuple[Optional[OpenAI], Optional[str]]: if model is None: continue # skip provider if we don't know a valid aux model logger.debug("Auxiliary text client: %s (%s)", pconfig.name, model) + if provider_id == "gemini": + from agent.gemini_native_adapter import GeminiNativeClient, is_native_gemini_base_url + + if is_native_gemini_base_url(base_url): + return GeminiNativeClient(api_key=api_key, base_url=base_url), model extra = {} - if "api.kimi.com" in base_url.lower(): - extra["default_headers"] = {"User-Agent": "KimiCLI/1.30.0"} - elif "api.githubcopilot.com" in base_url.lower(): + if base_url_host_matches(base_url, "api.kimi.com"): + extra["default_headers"] = {"User-Agent": "claude-code/0.1.0"} + elif base_url_host_matches(base_url, "api.githubcopilot.com"): from hermes_cli.models import copilot_default_headers extra["default_headers"] = copilot_default_headers() @@ -773,31 +916,80 @@ def _try_openrouter() -> Tuple[Optional[OpenAI], Optional[str]]: default_headers=_OR_HEADERS), _OPENROUTER_MODEL +def _describe_openrouter_unavailable() -> str: + """Return a more precise OpenRouter auth failure reason for logs.""" + pool_present, entry = _select_pool_entry("openrouter") + if pool_present: + if entry is None: + return "OpenRouter credential pool has no usable entries (credentials may be exhausted)" + if not _pool_runtime_api_key(entry): + return "OpenRouter credential pool entry is missing a runtime API key" + if not str(os.getenv("OPENROUTER_API_KEY") or "").strip(): + return "OPENROUTER_API_KEY not set" + return "no usable OpenRouter credentials found" + + def _try_nous(vision: bool = False) -> Tuple[Optional[OpenAI], Optional[str]]: + # Check cross-session rate limit guard before attempting Nous — + # if another session already recorded a 429, skip Nous entirely + # to avoid piling more requests onto the tapped RPH bucket. + try: + from agent.nous_rate_guard import nous_rate_limit_remaining + _remaining = nous_rate_limit_remaining() + if _remaining is not None and _remaining > 0: + logger.debug( + "Auxiliary: skipping Nous Portal (rate-limited, resets in %.0fs)", + _remaining, + ) + return None, None + except Exception: + pass + nous = _read_nous_auth() - if not nous: + runtime = _resolve_nous_runtime_api(force_refresh=False) + if runtime is None and not nous: return None, None global auxiliary_is_nous auxiliary_is_nous = True logger.debug("Auxiliary client: Nous Portal") - if nous.get("source") == "pool": - model = "gemini-3-flash" - else: - model = _NOUS_MODEL - # Free-tier users can't use paid auxiliary models — use the free - # models instead: mimo-v2-omni for vision, mimo-v2-pro for text tasks. + + # Ask the Portal which model it currently recommends for this task type. + # The /api/nous/recommended-models endpoint is the authoritative source: + # it distinguishes paid vs free tier recommendations, and get_nous_recommended_aux_model + # auto-detects the caller's tier via check_nous_free_tier(). Fall back to + # _NOUS_MODEL (google/gemini-3-flash-preview) when the Portal is unreachable + # or returns a null recommendation for this task type. + model = _NOUS_MODEL try: - from hermes_cli.models import check_nous_free_tier - if check_nous_free_tier(): - model = _NOUS_FREE_TIER_VISION_MODEL if vision else _NOUS_FREE_TIER_AUX_MODEL - logger.debug("Free-tier Nous account — using %s for auxiliary/%s", - model, "vision" if vision else "text") - except Exception: - pass + from hermes_cli.models import get_nous_recommended_aux_model + recommended = get_nous_recommended_aux_model(vision=vision) + if recommended: + model = recommended + logger.debug( + "Auxiliary/%s: using Portal-recommended model %s", + "vision" if vision else "text", model, + ) + else: + logger.debug( + "Auxiliary/%s: no Portal recommendation, falling back to %s", + "vision" if vision else "text", model, + ) + except Exception as exc: + logger.debug( + "Auxiliary/%s: recommended-models lookup failed (%s); " + "falling back to %s", + "vision" if vision else "text", exc, model, + ) + + if runtime is not None: + api_key, base_url = runtime + else: + api_key = _nous_api_key(nous or {}) + base_url = str((nous or {}).get("inference_base_url") or _nous_base_url()).rstrip("/") return ( OpenAI( - api_key=_nous_api_key(nous), - base_url=str(nous.get("inference_base_url") or _nous_base_url()).rstrip("/"), + api_key=api_key, + base_url=base_url, ), model, ) @@ -875,7 +1067,7 @@ def _resolve_custom_runtime() -> Tuple[Optional[str], Optional[str], Optional[st return None, None, None custom_base = custom_base.strip().rstrip("/") - if "openrouter.ai" in custom_base.lower(): + if base_url_host_matches(custom_base, "openrouter.ai"): # requested='custom' falls back to OpenRouter when no custom endpoint is # configured. Treat that as "no custom endpoint" for auxiliary routing. return None, None, None @@ -898,7 +1090,54 @@ def _current_custom_base_url() -> str: return custom_base or "" -def _try_custom_endpoint() -> Tuple[Optional[OpenAI], Optional[str]]: +def _validate_proxy_env_urls() -> None: + """Fail fast with a clear error when proxy env vars have malformed URLs. + + Common cause: shell config (e.g. .zshrc) with a typo like + ``export HTTP_PROXY=http://127.0.0.1:6153export NEXT_VAR=...`` + which concatenates 'export' into the port number. Without this + check the OpenAI/httpx client raises a cryptic ``Invalid port`` + error that doesn't name the offending env var. + """ + from urllib.parse import urlparse + + normalize_proxy_env_vars() + + for key in ("HTTPS_PROXY", "HTTP_PROXY", "ALL_PROXY", + "https_proxy", "http_proxy", "all_proxy"): + value = str(os.environ.get(key) or "").strip() + if not value: + continue + try: + parsed = urlparse(value) + if parsed.scheme: + _ = parsed.port # raises ValueError for e.g. '6153export' + except ValueError as exc: + raise RuntimeError( + f"Malformed proxy environment variable {key}={value!r}. " + "Fix or unset your proxy settings and try again." + ) from exc + + +def _validate_base_url(base_url: str) -> None: + """Reject obviously broken custom endpoint URLs before they reach httpx.""" + from urllib.parse import urlparse + + candidate = str(base_url or "").strip() + if not candidate or candidate.startswith("acp://"): + return + try: + parsed = urlparse(candidate) + if parsed.scheme in {"http", "https"}: + _ = parsed.port # raises ValueError for malformed ports + except ValueError as exc: + raise RuntimeError( + f"Malformed custom endpoint URL: {candidate!r}. " + "Run `hermes setup` or `hermes model` and enter a valid http(s) base URL." + ) from exc + + +def _try_custom_endpoint() -> Tuple[Optional[Any], Optional[str]]: runtime = _resolve_custom_runtime() if len(runtime) == 2: custom_base, custom_key = runtime @@ -914,6 +1153,23 @@ def _try_custom_endpoint() -> Tuple[Optional[OpenAI], Optional[str]]: if custom_mode == "codex_responses": real_client = OpenAI(api_key=custom_key, base_url=custom_base) return CodexAuxiliaryClient(real_client, model), model + if custom_mode == "anthropic_messages": + # Third-party Anthropic-compatible gateway (MiniMax, Zhipu GLM, + # LiteLLM proxies, etc.). Must NEVER be treated as OAuth — + # Anthropic OAuth claims only apply to api.anthropic.com. + try: + from agent.anthropic_adapter import build_anthropic_client + real_client = build_anthropic_client(custom_key, custom_base) + except ImportError: + logger.warning( + "Custom endpoint declares api_mode=anthropic_messages but the " + "anthropic SDK is not installed — falling back to OpenAI-wire." + ) + return OpenAI(api_key=custom_key, base_url=custom_base), model + return ( + AnthropicAuxiliaryClient(real_client, model, custom_key, custom_base, is_oauth=False), + model, + ) return OpenAI(api_key=custom_key, base_url=custom_base), model @@ -934,7 +1190,11 @@ def _try_codex() -> Tuple[Optional[Any], Optional[str]]: return None, None base_url = _CODEX_AUX_BASE_URL logger.debug("Auxiliary client: Codex OAuth (%s via Responses API)", _CODEX_AUX_MODEL) - real_client = OpenAI(api_key=codex_token, base_url=base_url) + real_client = OpenAI( + api_key=codex_token, + base_url=base_url, + default_headers=_codex_cloudflare_headers(codex_token), + ) return CodexAuxiliaryClient(real_client, _CODEX_AUX_MODEL), _CODEX_AUX_MODEL @@ -994,8 +1254,6 @@ def _try_anthropic() -> Tuple[Optional[Any], Optional[str]]: "_resolve_api_key_provider": "api-key", } -_AGGREGATOR_PROVIDERS = frozenset({"openrouter", "nous"}) - _MAIN_RUNTIME_FIELDS = ("provider", "model", "base_url", "api_key", "api_mode") @@ -1075,6 +1333,15 @@ def _is_connection_error(exc: Exception) -> bool: return False +def _is_auth_error(exc: Exception) -> bool: + """Detect auth failures that should trigger provider-specific refresh.""" + status = getattr(exc, "status_code", None) + if status == 401: + return True + err_lower = str(exc).lower() + return "error code: 401" in err_lower or "authenticationerror" in type(exc).__name__.lower() + + def _try_payment_fallback( failed_provider: str, task: str = None, @@ -1126,11 +1393,15 @@ def _resolve_auto(main_runtime: Optional[Dict[str, Any]] = None) -> Tuple[Option """Full auto-detection chain. Priority: - 1. If the user's main provider is NOT an aggregator (OpenRouter / Nous), - use their main provider + main model directly. This ensures users on - Alibaba, DeepSeek, ZAI, etc. get auxiliary tasks handled by the same - provider they already have credentials for — no OpenRouter key needed. - 2. OpenRouter → Nous → custom → Codex → API-key providers (original chain). + 1. User's main provider + main model, regardless of provider type. + This means auxiliary tasks (compression, vision, web extraction, + session search, etc.) use the same model the user configured for + chat. Users on OpenRouter/Nous get their chosen chat model; users + on DeepSeek/ZAI/Alibaba get theirs; etc. Running aux tasks on the + user's picked model keeps behavior predictable — no surprise + switches to a cheap fallback model for side tasks. + 2. OpenRouter → Nous → custom → Codex → API-key providers (fallback + chain, only used when the main provider has no working client). """ global auxiliary_is_nous, _stale_base_url_warned auxiliary_is_nous = False # Reset — _try_nous() will set True if it wins @@ -1160,11 +1431,16 @@ def _resolve_auto(main_runtime: Optional[Dict[str, Any]] = None) -> Tuple[Option ) _stale_base_url_warned = True - # ── Step 1: non-aggregator main provider → use main model directly ── + # ── Step 1: main provider + main model → use them directly ── + # + # This is the primary aux backend for every user. "auto" means + # "use my main chat model for side tasks as well" — including users + # on aggregators (OpenRouter, Nous) who previously got routed to a + # cheap provider-side default. Explicit per-task overrides set via + # config.yaml (auxiliary..provider) still win over this. main_provider = runtime_provider or _read_main_provider() main_model = runtime_model or _read_main_model() if (main_provider and main_model - and main_provider not in _AGGREGATOR_PROVIDERS and main_provider not in ("auto", "")): resolved_provider = main_provider explicit_base_url = None @@ -1223,20 +1499,33 @@ def _to_async_client(sync_client, model: str): return AsyncCodexAuxiliaryClient(sync_client), model if isinstance(sync_client, AnthropicAuxiliaryClient): return AsyncAnthropicAuxiliaryClient(sync_client), model + try: + from agent.gemini_native_adapter import GeminiNativeClient, AsyncGeminiNativeClient + + if isinstance(sync_client, GeminiNativeClient): + return AsyncGeminiNativeClient(sync_client), model + except ImportError: + pass + try: + from agent.copilot_acp_client import CopilotACPClient + if isinstance(sync_client, CopilotACPClient): + return sync_client, model + except ImportError: + pass async_kwargs = { "api_key": sync_client.api_key, "base_url": str(sync_client.base_url), } - base_lower = str(sync_client.base_url).lower() - if "openrouter" in base_lower: + sync_base_url = str(sync_client.base_url) + if base_url_host_matches(sync_base_url, "openrouter.ai"): async_kwargs["default_headers"] = dict(_OR_HEADERS) - elif "api.githubcopilot.com" in base_lower: + elif base_url_host_matches(sync_base_url, "api.githubcopilot.com"): from hermes_cli.models import copilot_default_headers async_kwargs["default_headers"] = copilot_default_headers() - elif "api.kimi.com" in base_lower: - async_kwargs["default_headers"] = {"User-Agent": "KimiCLI/1.30.0"} + elif base_url_host_matches(sync_base_url, "api.kimi.com"): + async_kwargs["default_headers"] = {"User-Agent": "claude-code/0.1.0"} return AsyncOpenAI(**async_kwargs), model @@ -1292,6 +1581,7 @@ def resolve_provider_client( Returns: (client, resolved_model) or (None, None) if auth is unavailable. """ + _validate_proxy_env_urls() # Normalise aliases provider = _normalize_aux_provider(provider) @@ -1311,8 +1601,7 @@ def _needs_codex_wrap(client_obj, base_url_str: str, model_str: str) -> bool: # Auto-detect: api.openai.com + codex model name pattern if api_mode and api_mode != "codex_responses": return False # explicit non-codex mode - normalized_base = (base_url_str or "").strip().lower() - if "api.openai.com" in normalized_base and "openrouter" not in normalized_base: + if base_url_hostname(base_url_str) == "api.openai.com": model_lower = (model_str or "").lower() if "codex" in model_lower: return True @@ -1351,8 +1640,10 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = ""): if provider == "openrouter": client, default = _try_openrouter() if client is None: - logger.warning("resolve_provider_client: openrouter requested " - "but OPENROUTER_API_KEY not set") + logger.warning( + "resolve_provider_client: openrouter requested but %s", + _describe_openrouter_unavailable(), + ) return None, None final_model = _normalize_resolved_model(model or default, provider) return (_to_async_client(client, final_model) if async_mode @@ -1360,7 +1651,13 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = ""): # ── Nous Portal (OAuth) ────────────────────────────────────────── if provider == "nous": - client, default = _try_nous() + # Detect vision tasks: either explicit model override from + # _PROVIDER_VISION_MODELS, or caller passed a known vision model. + _is_vision = ( + model in _PROVIDER_VISION_MODELS.values() + or (model or "").strip().lower() == "mimo-v2-omni" + ) + client, default = _try_nous(vision=_is_vision) if client is None: logger.warning("resolve_provider_client: nous requested " "but Nous Portal not configured (run: hermes auth)") @@ -1380,7 +1677,11 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = ""): "but no Codex OAuth token found (run: hermes model)") return None, None final_model = _normalize_resolved_model(model or _CODEX_AUX_MODEL, provider) - raw_client = OpenAI(api_key=codex_token, base_url=_CODEX_AUX_BASE_URL) + raw_client = OpenAI( + api_key=codex_token, + base_url=_CODEX_AUX_BASE_URL, + default_headers=_codex_cloudflare_headers(codex_token), + ) return (raw_client, final_model) # Standard path: wrap in CodexAuxiliaryClient adapter client, default = _try_codex() @@ -1412,9 +1713,9 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = ""): provider, ) extra = {} - if "api.kimi.com" in custom_base.lower(): - extra["default_headers"] = {"User-Agent": "KimiCLI/1.30.0"} - elif "api.githubcopilot.com" in custom_base.lower(): + if base_url_host_matches(custom_base, "api.kimi.com"): + extra["default_headers"] = {"User-Agent": "claude-code/0.1.0"} + elif base_url_host_matches(custom_base, "api.githubcopilot.com"): from hermes_cli.models import copilot_default_headers extra["default_headers"] = copilot_default_headers() client = OpenAI(api_key=custom_key, base_url=custom_base, **extra) @@ -1467,7 +1768,11 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = ""): # ── API-key providers from PROVIDER_REGISTRY ───────────────────── try: - from hermes_cli.auth import PROVIDER_REGISTRY, resolve_api_key_provider_credentials + from hermes_cli.auth import ( + PROVIDER_REGISTRY, + resolve_api_key_provider_credentials, + resolve_external_process_provider_credentials, + ) except ImportError: logger.debug("hermes_cli.auth not available for provider %s", provider) return None, None @@ -1504,15 +1809,23 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = ""): default_model = _API_KEY_PROVIDER_AUX_MODELS.get(provider, "") final_model = _normalize_resolved_model(model or default_model, provider) + if provider == "gemini": + from agent.gemini_native_adapter import GeminiNativeClient, is_native_gemini_base_url + + if is_native_gemini_base_url(base_url): + client = GeminiNativeClient(api_key=api_key, base_url=base_url) + logger.debug("resolve_provider_client: %s (%s)", provider, final_model) + return (_to_async_client(client, final_model) if async_mode + else (client, final_model)) + # Provider-specific headers headers = {} - if "api.kimi.com" in base_url.lower(): - headers["User-Agent"] = "KimiCLI/1.30.0" - elif "api.githubcopilot.com" in base_url.lower(): + if base_url_host_matches(base_url, "api.kimi.com"): + headers["User-Agent"] = "claude-code/0.1.0" + elif base_url_host_matches(base_url, "api.githubcopilot.com"): from hermes_cli.models import copilot_default_headers headers.update(copilot_default_headers()) - client = OpenAI(api_key=api_key, base_url=base_url, **({"default_headers": headers} if headers else {})) @@ -1541,6 +1854,41 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = ""): return (_to_async_client(client, final_model) if async_mode else (client, final_model)) + if pconfig.auth_type == "external_process": + creds = resolve_external_process_provider_credentials(provider) + final_model = _normalize_resolved_model(model or _read_main_model(), provider) + if provider == "copilot-acp": + api_key = str(creds.get("api_key", "")).strip() + base_url = str(creds.get("base_url", "")).strip() + command = str(creds.get("command", "")).strip() or None + args = list(creds.get("args") or []) + if not final_model: + logger.warning( + "resolve_provider_client: copilot-acp requested but no model " + "was provided or configured" + ) + return None, None + if not api_key or not base_url: + logger.warning( + "resolve_provider_client: copilot-acp requested but external " + "process credentials are incomplete" + ) + return None, None + from agent.copilot_acp_client import CopilotACPClient + + client = CopilotACPClient( + api_key=api_key, + base_url=base_url, + command=command, + args=args, + ) + logger.debug("resolve_provider_client: %s (%s)", provider, final_model) + return (_to_async_client(client, final_model) if async_mode + else (client, final_model)) + logger.warning("resolve_provider_client: external-process provider %s not " + "directly supported", provider) + return None, None + elif pconfig.auth_type in ("oauth_device_code", "oauth_external"): # OAuth providers — route through their specific try functions if provider == "nous": @@ -1701,34 +2049,42 @@ def _finalize(resolved_provider: str, sync_client: Any, default_model: Optional[ if requested == "auto": # Vision auto-detection order: - # 1. Active provider + model (user's main chat config) - # 2. OpenRouter (known vision-capable default model) - # 3. Nous Portal (known vision-capable default model) + # 1. User's main provider + main model (including aggregators). + # _PROVIDER_VISION_MODELS provides per-provider vision model + # overrides when the provider has a dedicated multimodal model + # that differs from the chat model (e.g. xiaomi → mimo-v2-omni, + # zai → glm-5v-turbo). Nous is the exception: it has a dedicated + # strict vision backend with tier-aware defaults, so it must not + # fall through to the user's text chat model here. + # 2. OpenRouter (vision-capable aggregator fallback) + # 3. Nous Portal (vision-capable aggregator fallback) # 4. Stop main_provider = _read_main_provider() main_model = _read_main_model() if main_provider and main_provider not in ("auto", ""): - if main_provider in _VISION_AUTO_PROVIDER_ORDER: - # Known strict backend — use its defaults. + if main_provider == "nous": sync_client, default_model = _resolve_strict_vision_backend(main_provider) if sync_client is not None: + logger.info( + "Vision auto-detect: using main provider %s (%s)", + main_provider, default_model or resolved_model or main_model, + ) return _finalize(main_provider, sync_client, default_model) else: - # Exotic provider (DeepSeek, Alibaba, Xiaomi, named custom, etc.) - # Use provider-specific vision model if available, otherwise main model. vision_model = _PROVIDER_VISION_MODELS.get(main_provider, main_model) rpc_client, rpc_model = resolve_provider_client( main_provider, vision_model, api_mode=resolved_api_mode) if rpc_client is not None: logger.info( - "Vision auto-detect: using active provider %s (%s)", + "Vision auto-detect: using main provider %s (%s)", main_provider, rpc_model or vision_model, ) return _finalize( main_provider, rpc_client, rpc_model or vision_model) - # Fall back through aggregators. + # Fall back through aggregators (uses their dedicated vision model, + # not the user's main model) when main provider has no client. for candidate in _VISION_AUTO_PROVIDER_ORDER: if candidate == main_provider: continue # already tried above @@ -1772,7 +2128,7 @@ def auxiliary_max_tokens_param(value: int) -> dict: # Only use max_completion_tokens for direct OpenAI custom endpoints if (not or_key and _read_nous_auth() is None - and "api.openai.com" in custom_base.lower()): + and base_url_hostname(custom_base) == "api.openai.com"): return {"max_completion_tokens": value} return {"max_tokens": value} @@ -1789,9 +2145,85 @@ def auxiliary_max_tokens_param(value: int) -> dict: # Every auxiliary LLM consumer should use these instead of manually # constructing clients and calling .chat.completions.create(). -# Client cache: (provider, async_mode, base_url, api_key) -> (client, default_model) +# Client cache: (provider, async_mode, base_url, api_key, api_mode, runtime_key) -> (client, default_model, loop) +# NOTE: loop identity is NOT part of the key. On async cache hits we check +# whether the cached loop is the *current* loop; if not, the stale entry is +# replaced in-place. This bounds cache growth to one entry per unique +# provider config rather than one per (config × event-loop), which previously +# caused unbounded fd accumulation in long-running gateway processes (#10200). _client_cache: Dict[tuple, tuple] = {} _client_cache_lock = threading.Lock() +_CLIENT_CACHE_MAX_SIZE = 64 # safety belt — evict oldest when exceeded + + +def _client_cache_key( + provider: str, + *, + async_mode: bool, + base_url: Optional[str] = None, + api_key: Optional[str] = None, + api_mode: Optional[str] = None, + main_runtime: Optional[Dict[str, Any]] = None, +) -> tuple: + runtime = _normalize_main_runtime(main_runtime) + runtime_key = tuple(runtime.get(field, "") for field in _MAIN_RUNTIME_FIELDS) if provider == "auto" else () + return (provider, async_mode, base_url or "", api_key or "", api_mode or "", runtime_key) + + +def _store_cached_client(cache_key: tuple, client: Any, default_model: Optional[str], *, bound_loop: Any = None) -> None: + with _client_cache_lock: + old_entry = _client_cache.get(cache_key) + if old_entry is not None and old_entry[0] is not client: + _force_close_async_httpx(old_entry[0]) + try: + close_fn = getattr(old_entry[0], "close", None) + if callable(close_fn): + close_fn() + except Exception: + pass + _client_cache[cache_key] = (client, default_model, bound_loop) + + +def _refresh_nous_auxiliary_client( + *, + cache_provider: str, + model: Optional[str], + async_mode: bool, + base_url: Optional[str] = None, + api_key: Optional[str] = None, + api_mode: Optional[str] = None, + main_runtime: Optional[Dict[str, Any]] = None, +) -> Tuple[Optional[Any], Optional[str]]: + """Refresh Nous runtime creds, rebuild the client, and replace the cache entry.""" + runtime = _resolve_nous_runtime_api(force_refresh=True) + if runtime is None: + return None, model + + fresh_key, fresh_base_url = runtime + sync_client = OpenAI(api_key=fresh_key, base_url=fresh_base_url) + final_model = model + + current_loop = None + if async_mode: + try: + import asyncio as _aio + current_loop = _aio.get_event_loop() + except RuntimeError: + pass + client, final_model = _to_async_client(sync_client, final_model or "") + else: + client = sync_client + + cache_key = _client_cache_key( + cache_provider, + async_mode=async_mode, + base_url=base_url, + api_key=api_key, + api_mode=api_mode, + main_runtime=main_runtime, + ) + _store_cached_client(cache_key, client, final_model, bound_loop=current_loop) + return client, final_model def neuter_async_httpx_del() -> None: @@ -1895,7 +2327,7 @@ def cleanup_stale_async_clients() -> None: def _is_openrouter_client(client: Any) -> bool: for obj in (client, getattr(client, "_client", None), getattr(client, "client", None)): - if obj and "openrouter" in str(getattr(obj, "base_url", "") or "").lower(): + if obj and base_url_host_matches(str(getattr(obj, "base_url", "") or ""), "openrouter.ai"): return True return False @@ -1924,39 +2356,55 @@ def _get_cached_client( Async clients (AsyncOpenAI) use httpx.AsyncClient internally, which binds to the event loop that was current when the client was created. Using such a client on a *different* loop causes deadlocks or - RuntimeError. To prevent cross-loop issues (especially in gateway - mode where _run_async() may spawn fresh loops in worker threads), the - cache key for async clients includes the current event loop's identity - so each loop gets its own client instance. + RuntimeError. To prevent cross-loop issues, the cache validates on + every async hit that the cached loop is the *current, open* loop. + If the loop changed (e.g. a new gateway worker-thread loop), the stale + entry is replaced in-place rather than creating an additional entry. + + This keeps cache size bounded to one entry per unique provider config, + preventing the fd-exhaustion that previously occurred in long-running + gateways where recycled worker threads created unbounded entries (#10200). """ - # Include loop identity for async clients to prevent cross-loop reuse. - # httpx.AsyncClient (inside AsyncOpenAI) is bound to the loop where it - # was created — reusing it on a different loop causes deadlocks (#2681). - loop_id = 0 + # Resolve the current event loop for async clients so we can validate + # cached entries. Loop identity is NOT in the cache key — instead we + # check at hit time whether the cached loop is still current and open. + # This prevents unbounded cache growth from recycled worker-thread loops + # while still guaranteeing we never reuse a client on the wrong loop + # (which causes deadlocks, see #2681). current_loop = None if async_mode: try: import asyncio as _aio current_loop = _aio.get_event_loop() - loop_id = id(current_loop) except RuntimeError: pass runtime = _normalize_main_runtime(main_runtime) - runtime_key = tuple(runtime.get(field, "") for field in _MAIN_RUNTIME_FIELDS) if provider == "auto" else () - cache_key = (provider, async_mode, base_url or "", api_key or "", api_mode or "", loop_id, runtime_key) + cache_key = _client_cache_key( + provider, + async_mode=async_mode, + base_url=base_url, + api_key=api_key, + api_mode=api_mode, + main_runtime=main_runtime, + ) with _client_cache_lock: if cache_key in _client_cache: cached_client, cached_default, cached_loop = _client_cache[cache_key] if async_mode: - # A cached async client whose loop has been closed will raise - # "Event loop is closed" when httpx tries to clean up its - # transport. Discard the stale client and create a fresh one. - if cached_loop is not None and cached_loop.is_closed(): - _force_close_async_httpx(cached_client) - del _client_cache[cache_key] - else: + # Validate: the cached client must be bound to the CURRENT, + # OPEN loop. If the loop changed or was closed, the httpx + # transport inside is dead — force-close and replace. + loop_ok = ( + cached_loop is not None + and cached_loop is current_loop + and not cached_loop.is_closed() + ) + if loop_ok: effective = _compat_model(cached_client, model, cached_default) return cached_client, effective + # Stale — evict and fall through to create a new client. + _force_close_async_httpx(cached_client) + del _client_cache[cache_key] else: effective = _compat_model(cached_client, model, cached_default) return cached_client, effective @@ -1976,6 +2424,12 @@ def _get_cached_client( bound_loop = current_loop with _client_cache_lock: if cache_key not in _client_cache: + # Safety belt: if the cache has grown beyond the max, evict + # the oldest entries (FIFO — dict preserves insertion order). + while len(_client_cache) >= _CLIENT_CACHE_MAX_SIZE: + evict_key, evict_entry = next(iter(_client_cache.items())) + _force_close_async_httpx(evict_entry[0]) + del _client_cache[evict_key] _client_cache[cache_key] = (client, default_model, bound_loop) else: client, default_model, _ = _client_cache[cache_key] @@ -2001,7 +2455,6 @@ def _resolve_task_provider_model( to "custom" and the task uses that direct endpoint. api_mode is one of "chat_completions", "codex_responses", or None (auto-detect). """ - config = {} cfg_provider = None cfg_model = None cfg_base_url = None @@ -2009,16 +2462,7 @@ def _resolve_task_provider_model( cfg_api_mode = None if task: - try: - from hermes_cli.config import load_config - config = load_config() - except ImportError: - config = {} - - aux = config.get("auxiliary", {}) if isinstance(config, dict) else {} - task_config = aux.get(task, {}) if isinstance(aux, dict) else {} - if not isinstance(task_config, dict): - task_config = {} + task_config = _get_auxiliary_task_config(task) cfg_provider = str(task_config.get("provider", "")).strip() or None cfg_model = str(task_config.get("model", "")).strip() or None cfg_base_url = str(task_config.get("base_url", "")).strip() or None @@ -2048,17 +2492,25 @@ def _resolve_task_provider_model( _DEFAULT_AUX_TIMEOUT = 30.0 -def _get_task_timeout(task: str, default: float = _DEFAULT_AUX_TIMEOUT) -> float: - """Read timeout from auxiliary.{task}.timeout in config, falling back to *default*.""" +def _get_auxiliary_task_config(task: str) -> Dict[str, Any]: + """Return the config dict for auxiliary., or {} when unavailable.""" if not task: - return default + return {} try: from hermes_cli.config import load_config config = load_config() except ImportError: - return default + return {} aux = config.get("auxiliary", {}) if isinstance(config, dict) else {} task_config = aux.get(task, {}) if isinstance(aux, dict) else {} + return task_config if isinstance(task_config, dict) else {} + + +def _get_task_timeout(task: str, default: float = _DEFAULT_AUX_TIMEOUT) -> float: + """Read timeout from auxiliary.{task}.timeout in config, falling back to *default*.""" + if not task: + return default + task_config = _get_auxiliary_task_config(task) raw = task_config.get("timeout") if raw is not None: try: @@ -2068,6 +2520,15 @@ def _get_task_timeout(task: str, default: float = _DEFAULT_AUX_TIMEOUT) -> float return default +def _get_task_extra_body(task: str) -> Dict[str, Any]: + """Read auxiliary..extra_body and return a shallow copy when valid.""" + task_config = _get_auxiliary_task_config(task) + raw = task_config.get("extra_body") + if isinstance(raw, dict): + return dict(raw) + return {} + + # --------------------------------------------------------------------------- # Anthropic-compatible endpoint detection + image block conversion # --------------------------------------------------------------------------- @@ -2155,6 +2616,21 @@ def _build_call_kwargs( "timeout": timeout, } + fixed_temperature = _fixed_temperature_for_model(model, base_url) + if fixed_temperature is OMIT_TEMPERATURE: + temperature = None # strip — let server choose + elif fixed_temperature is not None: + temperature = fixed_temperature + + # Opus 4.7+ rejects any non-default temperature/top_p/top_k — silently + # drop here so auxiliary callers that hardcode temperature (e.g. 0.3 on + # flush_memories, 0 on structured-JSON extraction) don't 400 the moment + # the aux model is flipped to 4.7. + if temperature is not None: + from agent.anthropic_adapter import _forbids_sampling_params + if _forbids_sampling_params(model): + temperature = None + if temperature is not None: kwargs["temperature"] = temperature @@ -2163,7 +2639,7 @@ def _build_call_kwargs( # Direct OpenAI api.openai.com with newer models needs max_completion_tokens. if provider == "custom": custom_base = base_url or _current_custom_base_url() - if "api.openai.com" in custom_base.lower(): + if base_url_hostname(custom_base) == "api.openai.com": kwargs["max_completion_tokens"] = max_tokens else: kwargs["max_tokens"] = max_tokens @@ -2255,13 +2731,15 @@ def call_llm( """ resolved_provider, resolved_model, resolved_base_url, resolved_api_key, resolved_api_mode = _resolve_task_provider_model( task, provider, model, base_url, api_key) + effective_extra_body = _get_task_extra_body(task) + effective_extra_body.update(extra_body or {}) if task == "vision": effective_provider, client, final_model = resolve_vision_provider_client( - provider=provider, - model=model, - base_url=base_url, - api_key=api_key, + provider=resolved_provider if resolved_provider != "auto" else provider, + model=resolved_model or model, + base_url=resolved_base_url or base_url, + api_key=resolved_api_key or api_key, async_mode=False, ) if client is None and resolved_provider != "auto" and not resolved_base_url: @@ -2323,11 +2801,14 @@ def call_llm( task, resolved_provider or "auto", final_model or "default", f" at {_base_info}" if _base_info and "openrouter" not in _base_info else "") + # Pass the client's actual base_url (not just resolved_base_url) so + # endpoint-specific temperature overrides can distinguish + # api.moonshot.ai vs api.kimi.com/coding even on auto-detected routes. kwargs = _build_call_kwargs( resolved_provider, final_model, messages, temperature=temperature, max_tokens=max_tokens, - tools=tools, timeout=effective_timeout, extra_body=extra_body, - base_url=resolved_base_url) + tools=tools, timeout=effective_timeout, extra_body=effective_extra_body, + base_url=_base_info or resolved_base_url) # Convert image blocks for Anthropic-compatible endpoints (e.g. MiniMax) _client_base = str(getattr(client, "base_url", "") or "") @@ -2353,6 +2834,29 @@ def call_llm( raise first_err = retry_err + # ── Nous auth refresh parity with main agent ────────────────── + client_is_nous = ( + resolved_provider == "nous" + or base_url_host_matches(_base_info, "inference-api.nousresearch.com") + ) + if _is_auth_error(first_err) and client_is_nous: + refreshed_client, refreshed_model = _refresh_nous_auxiliary_client( + cache_provider=resolved_provider or "nous", + model=final_model, + async_mode=False, + base_url=resolved_base_url, + api_key=resolved_api_key, + api_mode=resolved_api_mode, + main_runtime=main_runtime, + ) + if refreshed_client is not None: + logger.info("Auxiliary %s: refreshed Nous runtime credentials after 401, retrying", + task or "call") + if refreshed_model and refreshed_model != kwargs.get("model"): + kwargs["model"] = refreshed_model + return _validate_llm_response( + refreshed_client.chat.completions.create(**kwargs), task) + # ── Payment / credit exhaustion fallback ────────────────────── # When the resolved provider returns 402 or a credit-related error, # try alternative providers instead of giving up. This handles the @@ -2381,7 +2885,8 @@ def call_llm( fb_label, fb_model, messages, temperature=temperature, max_tokens=max_tokens, tools=tools, timeout=effective_timeout, - extra_body=extra_body) + extra_body=effective_extra_body, + base_url=str(getattr(fb_client, "base_url", "") or "")) return _validate_llm_response( fb_client.chat.completions.create(**fb_kwargs), task) raise @@ -2463,13 +2968,15 @@ async def async_call_llm( """ resolved_provider, resolved_model, resolved_base_url, resolved_api_key, resolved_api_mode = _resolve_task_provider_model( task, provider, model, base_url, api_key) + effective_extra_body = _get_task_extra_body(task) + effective_extra_body.update(extra_body or {}) if task == "vision": effective_provider, client, final_model = resolve_vision_provider_client( - provider=provider, - model=model, - base_url=base_url, - api_key=api_key, + provider=resolved_provider if resolved_provider != "auto" else provider, + model=resolved_model or model, + base_url=resolved_base_url or base_url, + api_key=resolved_api_key or api_key, async_mode=True, ) if client is None and resolved_provider != "auto" and not resolved_base_url: @@ -2516,14 +3023,17 @@ async def async_call_llm( effective_timeout = timeout if timeout is not None else _get_task_timeout(task) + # Pass the client's actual base_url (not just resolved_base_url) so + # endpoint-specific temperature overrides can distinguish + # api.moonshot.ai vs api.kimi.com/coding even on auto-detected routes. + _client_base = str(getattr(client, "base_url", "") or "") kwargs = _build_call_kwargs( resolved_provider, final_model, messages, temperature=temperature, max_tokens=max_tokens, - tools=tools, timeout=effective_timeout, extra_body=extra_body, - base_url=resolved_base_url) + tools=tools, timeout=effective_timeout, extra_body=effective_extra_body, + base_url=_client_base or resolved_base_url) # Convert image blocks for Anthropic-compatible endpoints (e.g. MiniMax) - _client_base = str(getattr(client, "base_url", "") or "") if _is_anthropic_compat_endpoint(resolved_provider, _client_base): kwargs["messages"] = _convert_openai_images_to_anthropic(kwargs["messages"]) @@ -2545,6 +3055,28 @@ async def async_call_llm( raise first_err = retry_err + # ── Nous auth refresh parity with main agent ────────────────── + client_is_nous = ( + resolved_provider == "nous" + or base_url_host_matches(_client_base, "inference-api.nousresearch.com") + ) + if _is_auth_error(first_err) and client_is_nous: + refreshed_client, refreshed_model = _refresh_nous_auxiliary_client( + cache_provider=resolved_provider or "nous", + model=final_model, + async_mode=True, + base_url=resolved_base_url, + api_key=resolved_api_key, + api_mode=resolved_api_mode, + ) + if refreshed_client is not None: + logger.info("Auxiliary %s (async): refreshed Nous runtime credentials after 401, retrying", + task or "call") + if refreshed_model and refreshed_model != kwargs.get("model"): + kwargs["model"] = refreshed_model + return _validate_llm_response( + await refreshed_client.chat.completions.create(**kwargs), task) + # ── Payment / connection fallback (mirrors sync call_llm) ───── should_fallback = _is_payment_error(first_err) or _is_connection_error(first_err) is_auto = resolved_provider in ("auto", "", None) @@ -2559,7 +3091,8 @@ async def async_call_llm( fb_label, fb_model, messages, temperature=temperature, max_tokens=max_tokens, tools=tools, timeout=effective_timeout, - extra_body=extra_body) + extra_body=effective_extra_body, + base_url=str(getattr(fb_client, "base_url", "") or "")) # Convert sync fallback client to async async_fb, async_fb_model = _to_async_client(fb_client, fb_model or "") if async_fb_model and async_fb_model != fb_kwargs.get("model"): diff --git a/agent/bedrock_adapter.py b/agent/bedrock_adapter.py new file mode 100644 index 000000000000..9e4297581de0 --- /dev/null +++ b/agent/bedrock_adapter.py @@ -0,0 +1,1098 @@ +"""AWS Bedrock Converse API adapter for Hermes Agent. + +Provides native integration with Amazon Bedrock using the Converse API, +bypassing the OpenAI-compatible endpoint in favor of direct AWS SDK calls. +This enables full access to the Bedrock ecosystem: + + - **Native Converse API**: Unified interface for all Bedrock models + (Claude, Nova, Llama, Mistral, etc.) with streaming support. + - **AWS credential chain**: IAM roles, SSO profiles, environment variables, + instance metadata — zero API key management for AWS-native environments. + - **Dynamic model discovery**: Auto-discovers available foundation models + and cross-region inference profiles via the Bedrock control plane. + - **Guardrails support**: Optional Bedrock Guardrails configuration for + content filtering and safety policies. + - **Inference profiles**: Supports cross-region inference profiles + (us.anthropic.claude-*, global.anthropic.claude-*) for better capacity + and automatic failover. + +Architecture follows the same pattern as ``anthropic_adapter.py``: + - All Bedrock-specific logic is isolated in this module. + - Messages/tools are converted between OpenAI format and Converse format. + - Responses are normalized back to OpenAI-compatible objects for the agent loop. + +Reference: OpenClaw's ``extensions/amazon-bedrock/`` plugin, which implements +the same Converse API integration in TypeScript via ``@aws-sdk/client-bedrock``. + +Requires: ``boto3`` (optional dependency — only needed when using the Bedrock provider). +""" + +import json +import logging +import os +import re +from types import SimpleNamespace +from typing import Any, Dict, List, Optional, Tuple + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Lazy boto3 import — only loaded when the Bedrock provider is actually used. +# This keeps startup fast for users who don't use Bedrock. +# --------------------------------------------------------------------------- + +_bedrock_runtime_client_cache: Dict[str, Any] = {} +_bedrock_control_client_cache: Dict[str, Any] = {} + + +def _require_boto3(): + """Import boto3, raising a clear error if not installed.""" + try: + import boto3 + return boto3 + except ImportError: + raise ImportError( + "The 'boto3' package is required for the AWS Bedrock provider. " + "Install it with: pip install boto3\n" + "Or install Hermes with Bedrock support: pip install -e '.[bedrock]'" + ) + + +def _get_bedrock_runtime_client(region: str): + """Get or create a cached ``bedrock-runtime`` client for the given region. + + Uses the default AWS credential chain (env vars → profile → instance role). + """ + if region not in _bedrock_runtime_client_cache: + boto3 = _require_boto3() + _bedrock_runtime_client_cache[region] = boto3.client( + "bedrock-runtime", region_name=region, + ) + return _bedrock_runtime_client_cache[region] + + +def _get_bedrock_control_client(region: str): + """Get or create a cached ``bedrock`` control-plane client for model discovery.""" + if region not in _bedrock_control_client_cache: + boto3 = _require_boto3() + _bedrock_control_client_cache[region] = boto3.client( + "bedrock", region_name=region, + ) + return _bedrock_control_client_cache[region] + + +def reset_client_cache(): + """Clear cached boto3 clients. Used in tests and profile switches.""" + _bedrock_runtime_client_cache.clear() + _bedrock_control_client_cache.clear() + + +# --------------------------------------------------------------------------- +# AWS credential detection +# --------------------------------------------------------------------------- + +# Priority order matches OpenClaw's resolveAwsSdkEnvVarName(): +# 1. AWS_BEARER_TOKEN_BEDROCK (Bedrock-specific bearer token) +# 2. AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY (explicit IAM credentials) +# 3. AWS_PROFILE (named profile → SSO, assume-role, etc.) +# 4. Implicit: instance role, ECS task role, Lambda execution role +_AWS_CREDENTIAL_ENV_VARS = [ + "AWS_BEARER_TOKEN_BEDROCK", + "AWS_ACCESS_KEY_ID", + "AWS_PROFILE", + # These are checked by boto3's default chain but we list them for + # has_aws_credentials() detection: + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", + "AWS_WEB_IDENTITY_TOKEN_FILE", +] + + +def resolve_aws_auth_env_var(env: Optional[Dict[str, str]] = None) -> Optional[str]: + """Return the name of the AWS auth source that is active, or None. + + Checks environment variables first, then falls back to boto3's credential + chain for implicit sources (EC2 IMDS, ECS task role, etc.). + + This mirrors OpenClaw's ``resolveAwsSdkEnvVarName()`` — used to detect + whether the user has any AWS credentials configured without actually + attempting to authenticate. + """ + env = env if env is not None else os.environ + # Bearer token takes highest priority + if env.get("AWS_BEARER_TOKEN_BEDROCK", "").strip(): + return "AWS_BEARER_TOKEN_BEDROCK" + # Explicit access key pair + if (env.get("AWS_ACCESS_KEY_ID", "").strip() + and env.get("AWS_SECRET_ACCESS_KEY", "").strip()): + return "AWS_ACCESS_KEY_ID" + # Named profile (SSO, assume-role, etc.) + if env.get("AWS_PROFILE", "").strip(): + return "AWS_PROFILE" + # Container credentials (ECS, CodeBuild) + if env.get("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", "").strip(): + return "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" + # Web identity (EKS IRSA) + if env.get("AWS_WEB_IDENTITY_TOKEN_FILE", "").strip(): + return "AWS_WEB_IDENTITY_TOKEN_FILE" + # No env vars — check if boto3 can resolve credentials via IMDS or other + # implicit sources (EC2 instance role, ECS task role, Lambda, etc.) + try: + import botocore.session + session = botocore.session.get_session() + credentials = session.get_credentials() + if credentials is not None: + resolved = credentials.get_frozen_credentials() + if resolved and resolved.access_key: + return "iam-role" + except Exception: + pass + return None + + +def has_aws_credentials(env: Optional[Dict[str, str]] = None) -> bool: + """Return True if any AWS credential source is detected. + + Checks environment variables first (fast, no I/O), then falls back to + boto3's credential chain which covers EC2 instance roles, ECS task roles, + Lambda execution roles, and other IMDS-based sources that don't set + environment variables. + + This two-tier approach mirrors the pattern from OpenClaw PR #62673: + cloud environments (EC2, ECS, Lambda) provide credentials via instance + metadata, not environment variables. The env-var check is a fast path + for local development; the boto3 fallback covers all cloud deployments. + """ + if resolve_aws_auth_env_var(env) is not None: + return True + # Fall back to boto3's credential resolver — this covers EC2 instance + # metadata (IMDS), ECS container credentials, and other implicit sources + # that don't set environment variables. + try: + import botocore.session + session = botocore.session.get_session() + credentials = session.get_credentials() + if credentials is not None: + resolved = credentials.get_frozen_credentials() + if resolved and resolved.access_key: + return True + except Exception: + pass + return False + + +def resolve_bedrock_region(env: Optional[Dict[str, str]] = None) -> str: + """Resolve the AWS region for Bedrock API calls. + + Priority: AWS_REGION → AWS_DEFAULT_REGION → us-east-1 (fallback). + """ + env = env if env is not None else os.environ + return ( + env.get("AWS_REGION", "").strip() + or env.get("AWS_DEFAULT_REGION", "").strip() + or "us-east-1" + ) + + +# --------------------------------------------------------------------------- +# Tool-calling capability detection +# --------------------------------------------------------------------------- +# Some Bedrock models don't support tool/function calling. Sending toolConfig +# to these models causes ValidationException. We maintain a denylist of known +# non-tool-calling model patterns and strip tools for them. +# +# This is a conservative approach: unknown models are assumed to support tools. +# If a model fails with a tool-related ValidationException, add it here. + +_NON_TOOL_CALLING_PATTERNS = [ + "deepseek.r1", # DeepSeek R1 — reasoning only, no tool support + "deepseek-r1", # Alternate ID format + "stability.", # Image generation models + "cohere.embed", # Embedding models + "amazon.titan-embed", # Embedding models +] + + +def _model_supports_tool_use(model_id: str) -> bool: + """Return True if the model is expected to support tool/function calling. + + Models in the denylist are known to reject toolConfig in the Converse API. + Unknown models default to True (assume tool support). + """ + model_lower = model_id.lower() + return not any(pattern in model_lower for pattern in _NON_TOOL_CALLING_PATTERNS) + + +def is_anthropic_bedrock_model(model_id: str) -> bool: + """Return True if the model is an Anthropic Claude model on Bedrock. + + These models should use the AnthropicBedrock SDK path for full feature + parity (prompt caching, thinking budgets, adaptive thinking). + Non-Claude models use the Converse API path. + + Matches: + - ``anthropic.claude-*`` (foundation model IDs) + - ``us.anthropic.claude-*`` (US inference profiles) + - ``global.anthropic.claude-*`` (global inference profiles) + - ``eu.anthropic.claude-*`` (EU inference profiles) + """ + model_lower = model_id.lower() + # Strip regional prefix if present + for prefix in ("us.", "global.", "eu.", "ap.", "jp."): + if model_lower.startswith(prefix): + model_lower = model_lower[len(prefix):] + break + return model_lower.startswith("anthropic.claude") + + +# --------------------------------------------------------------------------- +# Message format conversion: OpenAI → Bedrock Converse +# --------------------------------------------------------------------------- + +def convert_tools_to_converse(tools: List[Dict]) -> List[Dict]: + """Convert OpenAI-format tool definitions to Bedrock Converse ``toolConfig``. + + OpenAI format:: + + {"type": "function", "function": {"name": "...", "description": "...", + "parameters": {"type": "object", "properties": {...}}}} + + Converse format:: + + {"toolSpec": {"name": "...", "description": "...", + "inputSchema": {"json": {"type": "object", "properties": {...}}}}} + """ + if not tools: + return [] + result = [] + for t in tools: + fn = t.get("function", {}) + name = fn.get("name", "") + description = fn.get("description", "") + parameters = fn.get("parameters", {"type": "object", "properties": {}}) + result.append({ + "toolSpec": { + "name": name, + "description": description, + "inputSchema": {"json": parameters}, + } + }) + return result + + +def _convert_content_to_converse(content) -> List[Dict]: + """Convert OpenAI message content (string or list) to Converse content blocks. + + Handles: + - Plain text strings → [{"text": "..."}] + - Content arrays with text/image_url parts → mixed text/image blocks + + Filters out empty text blocks — Bedrock's Converse API rejects messages + where a text content block has an empty ``text`` field (ValidationException: + "text content blocks must be non-empty"). Ref: issue #9486. + """ + if content is None: + return [{"text": " "}] + if isinstance(content, str): + return [{"text": content}] if content.strip() else [{"text": " "}] + if isinstance(content, list): + blocks = [] + for part in content: + if isinstance(part, str): + blocks.append({"text": part}) + continue + if not isinstance(part, dict): + continue + part_type = part.get("type", "") + if part_type == "text": + text = part.get("text", "") + blocks.append({"text": text if text else " "}) + elif part_type == "image_url": + image_url = part.get("image_url", {}) + url = image_url.get("url", "") if isinstance(image_url, dict) else "" + if url.startswith("data:"): + # data:image/jpeg;base64,/9j/4AAQ... + header, _, data = url.partition(",") + media_type = "image/jpeg" + if header.startswith("data:"): + mime_part = header[5:].split(";")[0] + if mime_part: + media_type = mime_part + blocks.append({ + "image": { + "format": media_type.split("/")[-1] if "/" in media_type else "jpeg", + "source": {"bytes": data}, + } + }) + else: + # Remote URL — Converse doesn't support URLs directly, + # include as text reference for the model. + blocks.append({"text": f"[Image: {url}]"}) + return blocks if blocks else [{"text": " "}] + return [{"text": str(content)}] + + +def convert_messages_to_converse( + messages: List[Dict], +) -> Tuple[Optional[List[Dict]], List[Dict]]: + """Convert OpenAI-format messages to Bedrock Converse format. + + Returns ``(system_prompt, converse_messages)`` where: + - ``system_prompt`` is a list of system content blocks (or None) + - ``converse_messages`` is the conversation in Converse format + + Handles: + - System messages → extracted as system prompt + - User messages → ``{"role": "user", "content": [...]}`` + - Assistant messages → ``{"role": "assistant", "content": [...]}`` + - Tool calls → ``{"toolUse": {"toolUseId": ..., "name": ..., "input": ...}}`` + - Tool results → ``{"toolResult": {"toolUseId": ..., "content": [...]}}`` + + Converse requires strict user/assistant alternation. Consecutive messages + with the same role are merged into a single message. + """ + system_blocks: List[Dict] = [] + converse_msgs: List[Dict] = [] + + for msg in messages: + role = msg.get("role", "") + content = msg.get("content") + + if role == "system": + # System messages become the system prompt + if isinstance(content, str) and content.strip(): + system_blocks.append({"text": content}) + elif isinstance(content, list): + for part in content: + if isinstance(part, dict) and part.get("type") == "text": + system_blocks.append({"text": part.get("text", "")}) + elif isinstance(part, str): + system_blocks.append({"text": part}) + continue + + if role == "tool": + # Tool result messages → merge into the preceding user turn + tool_call_id = msg.get("tool_call_id", "") + result_content = content if isinstance(content, str) else json.dumps(content) + tool_result_block = { + "toolResult": { + "toolUseId": tool_call_id, + "content": [{"text": result_content}], + } + } + # In Converse, tool results go in a "user" role message + if converse_msgs and converse_msgs[-1]["role"] == "user": + converse_msgs[-1]["content"].append(tool_result_block) + else: + converse_msgs.append({ + "role": "user", + "content": [tool_result_block], + }) + continue + + if role == "assistant": + content_blocks = [] + # Convert text content + if isinstance(content, str) and content.strip(): + content_blocks.append({"text": content}) + elif isinstance(content, list): + content_blocks.extend(_convert_content_to_converse(content)) + + # Convert tool calls + tool_calls = msg.get("tool_calls", []) + for tc in (tool_calls or []): + fn = tc.get("function", {}) + args_str = fn.get("arguments", "{}") + try: + args_dict = json.loads(args_str) if isinstance(args_str, str) else args_str + except (json.JSONDecodeError, TypeError): + args_dict = {} + content_blocks.append({ + "toolUse": { + "toolUseId": tc.get("id", ""), + "name": fn.get("name", ""), + "input": args_dict, + } + }) + + if not content_blocks: + content_blocks = [{"text": " "}] + + # Merge with previous assistant message if needed (strict alternation) + if converse_msgs and converse_msgs[-1]["role"] == "assistant": + converse_msgs[-1]["content"].extend(content_blocks) + else: + converse_msgs.append({ + "role": "assistant", + "content": content_blocks, + }) + continue + + if role == "user": + content_blocks = _convert_content_to_converse(content) + # Merge with previous user message if needed (strict alternation) + if converse_msgs and converse_msgs[-1]["role"] == "user": + converse_msgs[-1]["content"].extend(content_blocks) + else: + converse_msgs.append({ + "role": "user", + "content": content_blocks, + }) + continue + + # Converse requires the first message to be from the user + if converse_msgs and converse_msgs[0]["role"] != "user": + converse_msgs.insert(0, {"role": "user", "content": [{"text": " "}]}) + + # Converse requires the last message to be from the user + if converse_msgs and converse_msgs[-1]["role"] != "user": + converse_msgs.append({"role": "user", "content": [{"text": " "}]}) + + return (system_blocks if system_blocks else None, converse_msgs) + + +# --------------------------------------------------------------------------- +# Response format conversion: Bedrock Converse → OpenAI +# --------------------------------------------------------------------------- + +def _converse_stop_reason_to_openai(stop_reason: str) -> str: + """Map Bedrock Converse stop reasons to OpenAI finish_reason values.""" + mapping = { + "end_turn": "stop", + "stop_sequence": "stop", + "tool_use": "tool_calls", + "max_tokens": "length", + "content_filtered": "content_filter", + "guardrail_intervened": "content_filter", + } + return mapping.get(stop_reason, "stop") + + +def normalize_converse_response(response: Dict) -> SimpleNamespace: + """Convert a Bedrock Converse API response to an OpenAI-compatible object. + + The agent loop in ``run_agent.py`` expects responses shaped like + ``openai.ChatCompletion`` — this function bridges the gap. + + Returns a SimpleNamespace with: + - ``.choices[0].message.content`` — text response + - ``.choices[0].message.tool_calls`` — tool call list (if any) + - ``.choices[0].finish_reason`` — stop/tool_calls/length + - ``.usage`` — token usage stats + """ + output = response.get("output", {}) + message = output.get("message", {}) + content_blocks = message.get("content", []) + stop_reason = response.get("stopReason", "end_turn") + + text_parts = [] + tool_calls = [] + + for block in content_blocks: + if "text" in block: + text_parts.append(block["text"]) + elif "toolUse" in block: + tu = block["toolUse"] + tool_calls.append(SimpleNamespace( + id=tu.get("toolUseId", ""), + type="function", + function=SimpleNamespace( + name=tu.get("name", ""), + arguments=json.dumps(tu.get("input", {})), + ), + )) + + # Build the message object + msg = SimpleNamespace( + role="assistant", + content="\n".join(text_parts) if text_parts else None, + tool_calls=tool_calls if tool_calls else None, + ) + + # Build usage stats + usage_data = response.get("usage", {}) + usage = SimpleNamespace( + prompt_tokens=usage_data.get("inputTokens", 0), + completion_tokens=usage_data.get("outputTokens", 0), + total_tokens=( + usage_data.get("inputTokens", 0) + usage_data.get("outputTokens", 0) + ), + ) + + finish_reason = _converse_stop_reason_to_openai(stop_reason) + if tool_calls and finish_reason == "stop": + finish_reason = "tool_calls" + + choice = SimpleNamespace( + index=0, + message=msg, + finish_reason=finish_reason, + ) + + return SimpleNamespace( + choices=[choice], + usage=usage, + model=response.get("modelId", ""), + ) + + +# --------------------------------------------------------------------------- +# Streaming response conversion +# --------------------------------------------------------------------------- + +def normalize_converse_stream_events(event_stream) -> SimpleNamespace: + """Consume a Bedrock ConverseStream event stream and build an OpenAI-compatible response. + + Processes the stream events in order: + - ``messageStart`` — role info + - ``contentBlockStart`` — new text or toolUse block + - ``contentBlockDelta`` — incremental text or toolUse input + - ``contentBlockStop`` — block complete + - ``messageStop`` — stop reason + - ``metadata`` — usage stats + + Returns the same shape as ``normalize_converse_response()``. + """ + return stream_converse_with_callbacks(event_stream) + + +def stream_converse_with_callbacks( + event_stream, + on_text_delta=None, + on_tool_start=None, + on_reasoning_delta=None, + on_interrupt_check=None, +) -> SimpleNamespace: + """Process a Bedrock ConverseStream event stream with real-time callbacks. + + This is the core streaming function that powers both the CLI's live token + display and the gateway's progressive message updates. + + Args: + event_stream: The boto3 ``converse_stream()`` response containing a + ``stream`` key with an iterable of events. + on_text_delta: Called with each text chunk as it arrives. Only fires + when no tool_use blocks have been seen (same semantics as the + Anthropic and chat_completions streaming paths). + on_tool_start: Called with the tool name when a toolUse block begins. + Lets the TUI show a spinner while tool arguments are generated. + on_reasoning_delta: Called with reasoning/thinking text chunks. + Bedrock surfaces thinking via ``reasoning`` content block deltas + on supported models (Claude 4.6+). + on_interrupt_check: Called on each event. Should return True if the + agent has been interrupted and streaming should stop. + + Returns: + An OpenAI-compatible SimpleNamespace response, identical in shape to + ``normalize_converse_response()``. + """ + text_parts: List[str] = [] + tool_calls: List[SimpleNamespace] = [] + current_tool: Optional[Dict] = None + current_text_buffer: List[str] = [] + has_tool_use = False + stop_reason = "end_turn" + usage_data: Dict[str, int] = {} + + for event in event_stream.get("stream", []): + # Check for interrupt + if on_interrupt_check and on_interrupt_check(): + break + + if "contentBlockStart" in event: + start = event["contentBlockStart"].get("start", {}) + if "toolUse" in start: + has_tool_use = True + # Flush any accumulated text + if current_text_buffer: + text_parts.append("".join(current_text_buffer)) + current_text_buffer = [] + current_tool = { + "toolUseId": start["toolUse"].get("toolUseId", ""), + "name": start["toolUse"].get("name", ""), + "input_json": "", + } + if on_tool_start: + on_tool_start(current_tool["name"]) + + elif "contentBlockDelta" in event: + delta = event["contentBlockDelta"].get("delta", {}) + if "text" in delta: + text = delta["text"] + current_text_buffer.append(text) + # Fire text delta callback only when no tool calls are present + # (same semantics as Anthropic/chat_completions streaming) + if on_text_delta and not has_tool_use: + on_text_delta(text) + elif "toolUse" in delta: + if current_tool is not None: + current_tool["input_json"] += delta["toolUse"].get("input", "") + elif "reasoningContent" in delta: + # Claude 4.6+ on Bedrock surfaces thinking via reasoningContent + reasoning = delta["reasoningContent"] + if isinstance(reasoning, dict): + thinking_text = reasoning.get("text", "") + if thinking_text and on_reasoning_delta: + on_reasoning_delta(thinking_text) + + elif "contentBlockStop" in event: + if current_tool is not None: + try: + input_dict = json.loads(current_tool["input_json"]) if current_tool["input_json"] else {} + except (json.JSONDecodeError, TypeError): + input_dict = {} + tool_calls.append(SimpleNamespace( + id=current_tool["toolUseId"], + type="function", + function=SimpleNamespace( + name=current_tool["name"], + arguments=json.dumps(input_dict), + ), + )) + current_tool = None + elif current_text_buffer: + text_parts.append("".join(current_text_buffer)) + current_text_buffer = [] + + elif "messageStop" in event: + stop_reason = event["messageStop"].get("stopReason", "end_turn") + + elif "metadata" in event: + meta_usage = event["metadata"].get("usage", {}) + usage_data = { + "inputTokens": meta_usage.get("inputTokens", 0), + "outputTokens": meta_usage.get("outputTokens", 0), + } + + # Flush remaining text + if current_text_buffer: + text_parts.append("".join(current_text_buffer)) + + msg = SimpleNamespace( + role="assistant", + content="\n".join(text_parts) if text_parts else None, + tool_calls=tool_calls if tool_calls else None, + ) + + usage = SimpleNamespace( + prompt_tokens=usage_data.get("inputTokens", 0), + completion_tokens=usage_data.get("outputTokens", 0), + total_tokens=( + usage_data.get("inputTokens", 0) + usage_data.get("outputTokens", 0) + ), + ) + + finish_reason = _converse_stop_reason_to_openai(stop_reason) + if tool_calls and finish_reason == "stop": + finish_reason = "tool_calls" + + choice = SimpleNamespace( + index=0, + message=msg, + finish_reason=finish_reason, + ) + + return SimpleNamespace( + choices=[choice], + usage=usage, + model="", + ) + + +# --------------------------------------------------------------------------- +# High-level API: call Bedrock Converse +# --------------------------------------------------------------------------- + +def build_converse_kwargs( + model: str, + messages: List[Dict], + tools: Optional[List[Dict]] = None, + max_tokens: int = 4096, + temperature: Optional[float] = None, + top_p: Optional[float] = None, + stop_sequences: Optional[List[str]] = None, + guardrail_config: Optional[Dict] = None, +) -> Dict[str, Any]: + """Build kwargs for ``bedrock-runtime.converse()`` or ``converse_stream()``. + + Converts OpenAI-format inputs to Converse API parameters. + """ + system_prompt, converse_messages = convert_messages_to_converse(messages) + + kwargs: Dict[str, Any] = { + "modelId": model, + "messages": converse_messages, + "inferenceConfig": { + "maxTokens": max_tokens, + }, + } + + if system_prompt: + kwargs["system"] = system_prompt + + if temperature is not None: + kwargs["inferenceConfig"]["temperature"] = temperature + + if top_p is not None: + kwargs["inferenceConfig"]["topP"] = top_p + + if stop_sequences: + kwargs["inferenceConfig"]["stopSequences"] = stop_sequences + + if tools: + converse_tools = convert_tools_to_converse(tools) + if converse_tools: + # Some Bedrock models don't support tool/function calling (e.g. + # DeepSeek R1, reasoning-only models). Sending toolConfig to + # these models causes a ValidationException → retry loop → failure. + # Strip tools for known non-tool-calling models and warn the user. + # Ref: PR #7920 feedback from @ptlally, pattern from PR #4346. + if _model_supports_tool_use(model): + kwargs["toolConfig"] = {"tools": converse_tools} + else: + logger.warning( + "Model %s does not support tool calling — tools stripped. " + "The agent will operate in text-only mode.", model + ) + + if guardrail_config: + kwargs["guardrailConfig"] = guardrail_config + + return kwargs + + +def call_converse( + region: str, + model: str, + messages: List[Dict], + tools: Optional[List[Dict]] = None, + max_tokens: int = 4096, + temperature: Optional[float] = None, + top_p: Optional[float] = None, + stop_sequences: Optional[List[str]] = None, + guardrail_config: Optional[Dict] = None, +) -> SimpleNamespace: + """Call Bedrock Converse API (non-streaming) and return an OpenAI-compatible response. + + This is the primary entry point for the agent loop when using the Bedrock provider. + """ + client = _get_bedrock_runtime_client(region) + kwargs = build_converse_kwargs( + model=model, + messages=messages, + tools=tools, + max_tokens=max_tokens, + temperature=temperature, + top_p=top_p, + stop_sequences=stop_sequences, + guardrail_config=guardrail_config, + ) + + response = client.converse(**kwargs) + return normalize_converse_response(response) + + +def call_converse_stream( + region: str, + model: str, + messages: List[Dict], + tools: Optional[List[Dict]] = None, + max_tokens: int = 4096, + temperature: Optional[float] = None, + top_p: Optional[float] = None, + stop_sequences: Optional[List[str]] = None, + guardrail_config: Optional[Dict] = None, +) -> SimpleNamespace: + """Call Bedrock ConverseStream API and return an OpenAI-compatible response. + + Consumes the full stream and returns the assembled response. For true + streaming with delta callbacks, use ``iter_converse_stream()`` instead. + """ + client = _get_bedrock_runtime_client(region) + kwargs = build_converse_kwargs( + model=model, + messages=messages, + tools=tools, + max_tokens=max_tokens, + temperature=temperature, + top_p=top_p, + stop_sequences=stop_sequences, + guardrail_config=guardrail_config, + ) + + response = client.converse_stream(**kwargs) + return normalize_converse_stream_events(response) + + +# --------------------------------------------------------------------------- +# Model discovery +# --------------------------------------------------------------------------- + +_discovery_cache: Dict[str, Any] = {} +_DISCOVERY_CACHE_TTL_SECONDS = 3600 + + +def reset_discovery_cache(): + """Clear the model discovery cache. Used in tests.""" + _discovery_cache.clear() + + +def discover_bedrock_models( + region: str, + provider_filter: Optional[List[str]] = None, +) -> List[Dict[str, Any]]: + """Discover available Bedrock foundation models and inference profiles. + + Returns a list of model info dicts with keys: + - ``id``: Model ID (e.g. "anthropic.claude-sonnet-4-6-20250514-v1:0") + - ``name``: Human-readable name + - ``provider``: Model provider (e.g. "Anthropic", "Amazon", "Meta") + - ``input_modalities``: List of input types (e.g. ["TEXT", "IMAGE"]) + - ``output_modalities``: List of output types + - ``streaming``: Whether streaming is supported + + Caches results for 1 hour per region to avoid repeated API calls. + + Mirrors OpenClaw's ``discoverBedrockModels()`` in + ``extensions/amazon-bedrock/discovery.ts``. + """ + import time + + cache_key = f"{region}:{','.join(sorted(provider_filter or []))}" + cached = _discovery_cache.get(cache_key) + if cached and (time.time() - cached["timestamp"]) < _DISCOVERY_CACHE_TTL_SECONDS: + return cached["models"] + + try: + client = _get_bedrock_control_client(region) + except Exception as e: + logger.warning("Failed to create Bedrock client for model discovery: %s", e) + return [] + + models = [] + seen_ids = set() + filter_set = {f.lower() for f in (provider_filter or [])} + + # 1. Discover foundation models + try: + response = client.list_foundation_models() + for summary in response.get("modelSummaries", []): + model_id = (summary.get("modelId") or "").strip() + if not model_id: + continue + + # Apply provider filter + if filter_set: + provider_name = (summary.get("providerName") or "").lower() + model_prefix = model_id.split(".")[0].lower() if "." in model_id else "" + if provider_name not in filter_set and model_prefix not in filter_set: + continue + + # Only include active, streaming-capable, text-output models + lifecycle = summary.get("modelLifecycle", {}) + if lifecycle.get("status", "").upper() != "ACTIVE": + continue + if not summary.get("responseStreamingSupported", False): + continue + output_mods = summary.get("outputModalities", []) + if "TEXT" not in output_mods: + continue + + models.append({ + "id": model_id, + "name": (summary.get("modelName") or model_id).strip(), + "provider": (summary.get("providerName") or "").strip(), + "input_modalities": summary.get("inputModalities", []), + "output_modalities": output_mods, + "streaming": True, + }) + seen_ids.add(model_id.lower()) + except Exception as e: + logger.warning("Failed to list Bedrock foundation models: %s", e) + + # 2. Discover inference profiles (cross-region, better capacity) + try: + profiles = [] + next_token = None + while True: + kwargs = {} + if next_token: + kwargs["nextToken"] = next_token + response = client.list_inference_profiles(**kwargs) + for profile in response.get("inferenceProfileSummaries", []): + profiles.append(profile) + next_token = response.get("nextToken") + if not next_token: + break + + for profile in profiles: + profile_id = (profile.get("inferenceProfileId") or "").strip() + if not profile_id: + continue + if profile.get("status") != "ACTIVE": + continue + if profile_id.lower() in seen_ids: + continue + + # Apply provider filter to underlying models + if filter_set: + profile_models = profile.get("models", []) + matches = any( + _extract_provider_from_arn(m.get("modelArn", "")).lower() in filter_set + for m in profile_models + ) + if not matches: + continue + + models.append({ + "id": profile_id, + "name": (profile.get("inferenceProfileName") or profile_id).strip(), + "provider": "inference-profile", + "input_modalities": ["TEXT"], + "output_modalities": ["TEXT"], + "streaming": True, + }) + seen_ids.add(profile_id.lower()) + except Exception as e: + logger.debug("Skipping inference profile discovery: %s", e) + + # Sort: global cross-region profiles first (recommended), then alphabetical + models.sort(key=lambda m: ( + 0 if m["id"].startswith("global.") else 1, + m["name"].lower(), + )) + + _discovery_cache[cache_key] = { + "timestamp": time.time(), + "models": models, + } + return models + + +def _extract_provider_from_arn(arn: str) -> str: + """Extract the model provider from a Bedrock model ARN. + + Example: "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-v2" + → "anthropic" + """ + match = re.search(r"foundation-model/([^.]+)", arn) + return match.group(1) if match else "" + + +def get_bedrock_model_ids(region: str) -> List[str]: + """Return a flat list of available Bedrock model IDs for the given region. + + Convenience wrapper around ``discover_bedrock_models()`` for use in + the model selection UI. + """ + models = discover_bedrock_models(region) + return [m["id"] for m in models] + + +# --------------------------------------------------------------------------- +# Error classification — Bedrock-specific exceptions +# --------------------------------------------------------------------------- +# Mirrors OpenClaw's classifyFailoverReason() and matchesContextOverflowError() +# in extensions/amazon-bedrock/register.sync.runtime.ts. + +# Patterns that indicate the input context exceeded the model's token limit. +# Used by run_agent.py to trigger context compression instead of retrying. +CONTEXT_OVERFLOW_PATTERNS = [ + re.compile(r"ValidationException.*(?:input is too long|max input token|input token.*exceed)", re.IGNORECASE), + re.compile(r"ValidationException.*(?:exceeds? the (?:maximum|max) (?:number of )?(?:input )?tokens)", re.IGNORECASE), + re.compile(r"ModelStreamErrorException.*(?:Input is too long|too many input tokens)", re.IGNORECASE), +] + +# Patterns for throttling / rate limit errors — should trigger backoff + retry. +THROTTLE_PATTERNS = [ + re.compile(r"ThrottlingException", re.IGNORECASE), + re.compile(r"Too many concurrent requests", re.IGNORECASE), + re.compile(r"ServiceQuotaExceededException", re.IGNORECASE), +] + +# Patterns for transient overload — model is temporarily unavailable. +OVERLOAD_PATTERNS = [ + re.compile(r"ModelNotReadyException", re.IGNORECASE), + re.compile(r"ModelTimeoutException", re.IGNORECASE), + re.compile(r"InternalServerException", re.IGNORECASE), +] + + +def is_context_overflow_error(error_message: str) -> bool: + """Return True if the error indicates the input context was too large. + + When this returns True, the agent should compress context and retry + rather than treating it as a fatal error. + """ + return any(p.search(error_message) for p in CONTEXT_OVERFLOW_PATTERNS) + + +def classify_bedrock_error(error_message: str) -> str: + """Classify a Bedrock error for retry/failover decisions. + + Returns: + - ``"context_overflow"`` — input too long, compress and retry + - ``"rate_limit"`` — throttled, backoff and retry + - ``"overloaded"`` — model temporarily unavailable, retry with delay + - ``"unknown"`` — unclassified error + """ + if is_context_overflow_error(error_message): + return "context_overflow" + if any(p.search(error_message) for p in THROTTLE_PATTERNS): + return "rate_limit" + if any(p.search(error_message) for p in OVERLOAD_PATTERNS): + return "overloaded" + return "unknown" + + +# --------------------------------------------------------------------------- +# Bedrock model context lengths +# --------------------------------------------------------------------------- +# Static fallback table for models where the Bedrock API doesn't expose +# context window sizes. Used by agent/model_metadata.py when dynamic +# detection is unavailable. + +BEDROCK_CONTEXT_LENGTHS: Dict[str, int] = { + # Anthropic Claude models on Bedrock + "anthropic.claude-opus-4-6": 200_000, + "anthropic.claude-sonnet-4-6": 200_000, + "anthropic.claude-sonnet-4-5": 200_000, + "anthropic.claude-haiku-4-5": 200_000, + "anthropic.claude-opus-4": 200_000, + "anthropic.claude-sonnet-4": 200_000, + "anthropic.claude-3-5-sonnet": 200_000, + "anthropic.claude-3-5-haiku": 200_000, + "anthropic.claude-3-opus": 200_000, + "anthropic.claude-3-sonnet": 200_000, + "anthropic.claude-3-haiku": 200_000, + # Amazon Nova + "amazon.nova-pro": 300_000, + "amazon.nova-lite": 300_000, + "amazon.nova-micro": 128_000, + # Meta Llama + "meta.llama4-maverick": 128_000, + "meta.llama4-scout": 128_000, + "meta.llama3-3-70b-instruct": 128_000, + # Mistral + "mistral.mistral-large": 128_000, + # DeepSeek + "deepseek.v3": 128_000, +} + +# Default for unknown Bedrock models +BEDROCK_DEFAULT_CONTEXT_LENGTH = 128_000 + + +def get_bedrock_context_length(model_id: str) -> int: + """Look up the context window size for a Bedrock model. + + Uses substring matching so versioned IDs like + ``anthropic.claude-sonnet-4-6-20250514-v1:0`` resolve correctly. + """ + model_lower = model_id.lower() + best_key = "" + best_val = BEDROCK_DEFAULT_CONTEXT_LENGTH + for key, val in BEDROCK_CONTEXT_LENGTHS.items(): + if key in model_lower and len(key) > len(best_key): + best_key = key + best_val = val + return best_val diff --git a/agent/codex_responses_adapter.py b/agent/codex_responses_adapter.py new file mode 100644 index 000000000000..4d3e5590be44 --- /dev/null +++ b/agent/codex_responses_adapter.py @@ -0,0 +1,813 @@ +"""Codex Responses API adapter. + +Pure format-conversion and normalization logic for the OpenAI Responses API +(used by OpenAI Codex, xAI, GitHub Models, and other Responses-compatible endpoints). + +Extracted from run_agent.py to isolate Responses API-specific logic from the +core agent loop. All functions are stateless — they operate on the data passed +in and return transformed results. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import re +import uuid +from types import SimpleNamespace +from typing import Any, Dict, List, Optional + +from agent.prompt_builder import DEFAULT_AGENT_IDENTITY + +logger = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# Multimodal content helpers +# --------------------------------------------------------------------------- + +def _chat_content_to_responses_parts(content: Any) -> List[Dict[str, Any]]: + """Convert chat-style multimodal content to Responses API input parts. + + Input: ``[{"type":"text"|"image_url", ...}]`` (native OpenAI Chat format) + Output: ``[{"type":"input_text"|"input_image", ...}]`` (Responses format) + + Returns an empty list when ``content`` is not a list or contains no + recognized parts — callers fall back to the string path. + """ + if not isinstance(content, list): + return [] + converted: List[Dict[str, Any]] = [] + for part in content: + if isinstance(part, str): + if part: + converted.append({"type": "input_text", "text": part}) + continue + if not isinstance(part, dict): + continue + ptype = str(part.get("type") or "").strip().lower() + if ptype in {"text", "input_text", "output_text"}: + text = part.get("text") + if isinstance(text, str) and text: + converted.append({"type": "input_text", "text": text}) + continue + if ptype in {"image_url", "input_image"}: + image_ref = part.get("image_url") + detail = part.get("detail") + if isinstance(image_ref, dict): + url = image_ref.get("url") + detail = image_ref.get("detail", detail) + else: + url = image_ref + if not isinstance(url, str) or not url: + continue + image_part: Dict[str, Any] = {"type": "input_image", "image_url": url} + if isinstance(detail, str) and detail.strip(): + image_part["detail"] = detail.strip() + converted.append(image_part) + return converted + + +def _summarize_user_message_for_log(content: Any) -> str: + """Return a short text summary of a user message for logging/trajectory. + + Multimodal messages arrive as a list of ``{type:"text"|"image_url", ...}`` + parts from the API server. Logging, spinner previews, and trajectory + files all want a plain string — this helper extracts the first chunk of + text and notes any attached images. Returns an empty string for empty + lists and ``str(content)`` for unexpected scalar types. + """ + if content is None: + return "" + if isinstance(content, str): + return content + if isinstance(content, list): + text_bits: List[str] = [] + image_count = 0 + for part in content: + if isinstance(part, str): + if part: + text_bits.append(part) + continue + if not isinstance(part, dict): + continue + ptype = str(part.get("type") or "").strip().lower() + if ptype in {"text", "input_text", "output_text"}: + text = part.get("text") + if isinstance(text, str) and text: + text_bits.append(text) + elif ptype in {"image_url", "input_image"}: + image_count += 1 + summary = " ".join(text_bits).strip() + if image_count: + note = f"[{image_count} image{'s' if image_count != 1 else ''}]" + summary = f"{note} {summary}" if summary else note + return summary + try: + return str(content) + except Exception: + return "" + + +# --------------------------------------------------------------------------- +# ID helpers +# --------------------------------------------------------------------------- + +def _deterministic_call_id(fn_name: str, arguments: str, index: int = 0) -> str: + """Generate a deterministic call_id from tool call content. + + Used as a fallback when the API doesn't provide a call_id. + Deterministic IDs prevent cache invalidation — random UUIDs would + make every API call's prefix unique, breaking OpenAI's prompt cache. + """ + seed = f"{fn_name}:{arguments}:{index}" + digest = hashlib.sha256(seed.encode("utf-8", errors="replace")).hexdigest()[:12] + return f"call_{digest}" + + +def _split_responses_tool_id(raw_id: Any) -> tuple[Optional[str], Optional[str]]: + """Split a stored tool id into (call_id, response_item_id).""" + if not isinstance(raw_id, str): + return None, None + value = raw_id.strip() + if not value: + return None, None + if "|" in value: + call_id, response_item_id = value.split("|", 1) + call_id = call_id.strip() or None + response_item_id = response_item_id.strip() or None + return call_id, response_item_id + if value.startswith("fc_"): + return None, value + return value, None + + +def _derive_responses_function_call_id( + call_id: str, + response_item_id: Optional[str] = None, +) -> str: + """Build a valid Responses `function_call.id` (must start with `fc_`).""" + if isinstance(response_item_id, str): + candidate = response_item_id.strip() + if candidate.startswith("fc_"): + return candidate + + source = (call_id or "").strip() + if source.startswith("fc_"): + return source + if source.startswith("call_") and len(source) > len("call_"): + return f"fc_{source[len('call_'):]}" + + sanitized = re.sub(r"[^A-Za-z0-9_-]", "", source) + if sanitized.startswith("fc_"): + return sanitized + if sanitized.startswith("call_") and len(sanitized) > len("call_"): + return f"fc_{sanitized[len('call_'):]}" + if sanitized: + return f"fc_{sanitized[:48]}" + + seed = source or str(response_item_id or "") or uuid.uuid4().hex + digest = hashlib.sha1(seed.encode("utf-8")).hexdigest()[:24] + return f"fc_{digest}" + + +# --------------------------------------------------------------------------- +# Schema conversion +# --------------------------------------------------------------------------- + +def _responses_tools(tools: Optional[List[Dict[str, Any]]] = None) -> Optional[List[Dict[str, Any]]]: + """Convert chat-completions tool schemas to Responses function-tool schemas.""" + if not tools: + return None + + converted: List[Dict[str, Any]] = [] + for item in tools: + fn = item.get("function", {}) if isinstance(item, dict) else {} + name = fn.get("name") + if not isinstance(name, str) or not name.strip(): + continue + converted.append({ + "type": "function", + "name": name, + "description": fn.get("description", ""), + "strict": False, + "parameters": fn.get("parameters", {"type": "object", "properties": {}}), + }) + return converted or None + + +# --------------------------------------------------------------------------- +# Message format conversion +# --------------------------------------------------------------------------- + +def _chat_messages_to_responses_input(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Convert internal chat-style messages to Responses input items.""" + items: List[Dict[str, Any]] = [] + seen_item_ids: set = set() + + for msg in messages: + if not isinstance(msg, dict): + continue + role = msg.get("role") + if role == "system": + continue + + if role in {"user", "assistant"}: + content = msg.get("content", "") + if isinstance(content, list): + content_parts = _chat_content_to_responses_parts(content) + content_text = "".join( + p.get("text", "") for p in content_parts if p.get("type") == "input_text" + ) + else: + content_parts = [] + content_text = str(content) if content is not None else "" + + if role == "assistant": + # Replay encrypted reasoning items from previous turns + # so the API can maintain coherent reasoning chains. + codex_reasoning = msg.get("codex_reasoning_items") + has_codex_reasoning = False + if isinstance(codex_reasoning, list): + for ri in codex_reasoning: + if isinstance(ri, dict) and ri.get("encrypted_content"): + item_id = ri.get("id") + if item_id and item_id in seen_item_ids: + continue + # Strip the "id" field — with store=False the + # Responses API cannot look up items by ID and + # returns 404. The encrypted_content blob is + # self-contained for reasoning chain continuity. + replay_item = {k: v for k, v in ri.items() if k != "id"} + items.append(replay_item) + if item_id: + seen_item_ids.add(item_id) + has_codex_reasoning = True + + if content_parts: + items.append({"role": "assistant", "content": content_parts}) + elif content_text.strip(): + items.append({"role": "assistant", "content": content_text}) + elif has_codex_reasoning: + # The Responses API requires a following item after each + # reasoning item (otherwise: missing_following_item error). + # When the assistant produced only reasoning with no visible + # content, emit an empty assistant message as the required + # following item. + items.append({"role": "assistant", "content": ""}) + + tool_calls = msg.get("tool_calls") + if isinstance(tool_calls, list): + for tc in tool_calls: + if not isinstance(tc, dict): + continue + fn = tc.get("function", {}) + fn_name = fn.get("name") + if not isinstance(fn_name, str) or not fn_name.strip(): + continue + + embedded_call_id, embedded_response_item_id = _split_responses_tool_id( + tc.get("id") + ) + call_id = tc.get("call_id") + if not isinstance(call_id, str) or not call_id.strip(): + call_id = embedded_call_id + if not isinstance(call_id, str) or not call_id.strip(): + if ( + isinstance(embedded_response_item_id, str) + and embedded_response_item_id.startswith("fc_") + and len(embedded_response_item_id) > len("fc_") + ): + call_id = f"call_{embedded_response_item_id[len('fc_'):]}" + else: + _raw_args = str(fn.get("arguments", "{}")) + call_id = _deterministic_call_id(fn_name, _raw_args, len(items)) + call_id = call_id.strip() + + arguments = fn.get("arguments", "{}") + if isinstance(arguments, dict): + arguments = json.dumps(arguments, ensure_ascii=False) + elif not isinstance(arguments, str): + arguments = str(arguments) + arguments = arguments.strip() or "{}" + + items.append({ + "type": "function_call", + "call_id": call_id, + "name": fn_name, + "arguments": arguments, + }) + continue + + # Non-assistant (user) role: emit multimodal parts when present, + # otherwise fall back to the text payload. + if content_parts: + items.append({"role": role, "content": content_parts}) + else: + items.append({"role": role, "content": content_text}) + continue + + if role == "tool": + raw_tool_call_id = msg.get("tool_call_id") + call_id, _ = _split_responses_tool_id(raw_tool_call_id) + if not isinstance(call_id, str) or not call_id.strip(): + if isinstance(raw_tool_call_id, str) and raw_tool_call_id.strip(): + call_id = raw_tool_call_id.strip() + if not isinstance(call_id, str) or not call_id.strip(): + continue + items.append({ + "type": "function_call_output", + "call_id": call_id, + "output": str(msg.get("content", "") or ""), + }) + + return items + + +# --------------------------------------------------------------------------- +# Input preflight / validation +# --------------------------------------------------------------------------- + +def _preflight_codex_input_items(raw_items: Any) -> List[Dict[str, Any]]: + if not isinstance(raw_items, list): + raise ValueError("Codex Responses input must be a list of input items.") + + normalized: List[Dict[str, Any]] = [] + seen_ids: set = set() + for idx, item in enumerate(raw_items): + if not isinstance(item, dict): + raise ValueError(f"Codex Responses input[{idx}] must be an object.") + + item_type = item.get("type") + if item_type == "function_call": + call_id = item.get("call_id") + name = item.get("name") + if not isinstance(call_id, str) or not call_id.strip(): + raise ValueError(f"Codex Responses input[{idx}] function_call is missing call_id.") + if not isinstance(name, str) or not name.strip(): + raise ValueError(f"Codex Responses input[{idx}] function_call is missing name.") + + arguments = item.get("arguments", "{}") + if isinstance(arguments, dict): + arguments = json.dumps(arguments, ensure_ascii=False) + elif not isinstance(arguments, str): + arguments = str(arguments) + arguments = arguments.strip() or "{}" + + normalized.append( + { + "type": "function_call", + "call_id": call_id.strip(), + "name": name.strip(), + "arguments": arguments, + } + ) + continue + + if item_type == "function_call_output": + call_id = item.get("call_id") + if not isinstance(call_id, str) or not call_id.strip(): + raise ValueError(f"Codex Responses input[{idx}] function_call_output is missing call_id.") + output = item.get("output", "") + if output is None: + output = "" + if not isinstance(output, str): + output = str(output) + + normalized.append( + { + "type": "function_call_output", + "call_id": call_id.strip(), + "output": output, + } + ) + continue + + if item_type == "reasoning": + encrypted = item.get("encrypted_content") + if isinstance(encrypted, str) and encrypted: + item_id = item.get("id") + if isinstance(item_id, str) and item_id: + if item_id in seen_ids: + continue + seen_ids.add(item_id) + reasoning_item = {"type": "reasoning", "encrypted_content": encrypted} + # Do NOT include the "id" in the outgoing item — with + # store=False (our default) the API tries to resolve the + # id server-side and returns 404. The id is still used + # above for local deduplication via seen_ids. + summary = item.get("summary") + if isinstance(summary, list): + reasoning_item["summary"] = summary + else: + reasoning_item["summary"] = [] + normalized.append(reasoning_item) + continue + + role = item.get("role") + if role in {"user", "assistant"}: + content = item.get("content", "") + if content is None: + content = "" + if isinstance(content, list): + # Multimodal content from ``_chat_messages_to_responses_input`` + # is already in Responses format (``input_text`` / ``input_image``). + # Validate each part and pass through. + validated: List[Dict[str, Any]] = [] + for part_idx, part in enumerate(content): + if isinstance(part, str): + if part: + validated.append({"type": "input_text", "text": part}) + continue + if not isinstance(part, dict): + raise ValueError( + f"Codex Responses input[{idx}].content[{part_idx}] must be an object or string." + ) + ptype = str(part.get("type") or "").strip().lower() + if ptype in {"input_text", "text", "output_text"}: + text = part.get("text", "") + if not isinstance(text, str): + text = str(text or "") + validated.append({"type": "input_text", "text": text}) + elif ptype in {"input_image", "image_url"}: + image_ref = part.get("image_url", "") + detail = part.get("detail") + if isinstance(image_ref, dict): + url = image_ref.get("url", "") + detail = image_ref.get("detail", detail) + else: + url = image_ref + if not isinstance(url, str): + url = str(url or "") + image_part: Dict[str, Any] = {"type": "input_image", "image_url": url} + if isinstance(detail, str) and detail.strip(): + image_part["detail"] = detail.strip() + validated.append(image_part) + else: + raise ValueError( + f"Codex Responses input[{idx}].content[{part_idx}] has unsupported type {part.get('type')!r}." + ) + normalized.append({"role": role, "content": validated}) + continue + if not isinstance(content, str): + content = str(content) + + normalized.append({"role": role, "content": content}) + continue + + raise ValueError( + f"Codex Responses input[{idx}] has unsupported item shape (type={item_type!r}, role={role!r})." + ) + + return normalized + + +def _preflight_codex_api_kwargs( + api_kwargs: Any, + *, + allow_stream: bool = False, +) -> Dict[str, Any]: + if not isinstance(api_kwargs, dict): + raise ValueError("Codex Responses request must be a dict.") + + required = {"model", "instructions", "input"} + missing = [key for key in required if key not in api_kwargs] + if missing: + raise ValueError(f"Codex Responses request missing required field(s): {', '.join(sorted(missing))}.") + + model = api_kwargs.get("model") + if not isinstance(model, str) or not model.strip(): + raise ValueError("Codex Responses request 'model' must be a non-empty string.") + model = model.strip() + + instructions = api_kwargs.get("instructions") + if instructions is None: + instructions = "" + if not isinstance(instructions, str): + instructions = str(instructions) + instructions = instructions.strip() or DEFAULT_AGENT_IDENTITY + + normalized_input = _preflight_codex_input_items(api_kwargs.get("input")) + + tools = api_kwargs.get("tools") + normalized_tools = None + if tools is not None: + if not isinstance(tools, list): + raise ValueError("Codex Responses request 'tools' must be a list when provided.") + normalized_tools = [] + for idx, tool in enumerate(tools): + if not isinstance(tool, dict): + raise ValueError(f"Codex Responses tools[{idx}] must be an object.") + if tool.get("type") != "function": + raise ValueError(f"Codex Responses tools[{idx}] has unsupported type {tool.get('type')!r}.") + + name = tool.get("name") + parameters = tool.get("parameters") + if not isinstance(name, str) or not name.strip(): + raise ValueError(f"Codex Responses tools[{idx}] is missing a valid name.") + if not isinstance(parameters, dict): + raise ValueError(f"Codex Responses tools[{idx}] is missing valid parameters.") + + description = tool.get("description", "") + if description is None: + description = "" + if not isinstance(description, str): + description = str(description) + + strict = tool.get("strict", False) + if not isinstance(strict, bool): + strict = bool(strict) + + normalized_tools.append( + { + "type": "function", + "name": name.strip(), + "description": description, + "strict": strict, + "parameters": parameters, + } + ) + + store = api_kwargs.get("store", False) + if store is not False: + raise ValueError("Codex Responses contract requires 'store' to be false.") + + allowed_keys = { + "model", "instructions", "input", "tools", "store", + "reasoning", "include", "max_output_tokens", "temperature", + "tool_choice", "parallel_tool_calls", "prompt_cache_key", "service_tier", + "extra_headers", + } + normalized: Dict[str, Any] = { + "model": model, + "instructions": instructions, + "input": normalized_input, + "store": False, + } + if normalized_tools is not None: + normalized["tools"] = normalized_tools + + # Pass through reasoning config + reasoning = api_kwargs.get("reasoning") + if isinstance(reasoning, dict): + normalized["reasoning"] = reasoning + include = api_kwargs.get("include") + if isinstance(include, list): + normalized["include"] = include + service_tier = api_kwargs.get("service_tier") + if isinstance(service_tier, str) and service_tier.strip(): + normalized["service_tier"] = service_tier.strip() + + # Pass through max_output_tokens and temperature + max_output_tokens = api_kwargs.get("max_output_tokens") + if isinstance(max_output_tokens, (int, float)) and max_output_tokens > 0: + normalized["max_output_tokens"] = int(max_output_tokens) + temperature = api_kwargs.get("temperature") + if isinstance(temperature, (int, float)): + normalized["temperature"] = float(temperature) + + # Pass through tool_choice, parallel_tool_calls, prompt_cache_key + for passthrough_key in ("tool_choice", "parallel_tool_calls", "prompt_cache_key"): + val = api_kwargs.get(passthrough_key) + if val is not None: + normalized[passthrough_key] = val + + extra_headers = api_kwargs.get("extra_headers") + if extra_headers is not None: + if not isinstance(extra_headers, dict): + raise ValueError("Codex Responses request 'extra_headers' must be an object.") + normalized_headers: Dict[str, str] = {} + for key, value in extra_headers.items(): + if not isinstance(key, str) or not key.strip(): + raise ValueError("Codex Responses request 'extra_headers' keys must be non-empty strings.") + if value is None: + continue + normalized_headers[key.strip()] = str(value) + if normalized_headers: + normalized["extra_headers"] = normalized_headers + + if allow_stream: + stream = api_kwargs.get("stream") + if stream is not None and stream is not True: + raise ValueError("Codex Responses 'stream' must be true when set.") + if stream is True: + normalized["stream"] = True + allowed_keys.add("stream") + elif "stream" in api_kwargs: + raise ValueError("Codex Responses stream flag is only allowed in fallback streaming requests.") + + unexpected = sorted(key for key in api_kwargs if key not in allowed_keys) + if unexpected: + raise ValueError( + f"Codex Responses request has unsupported field(s): {', '.join(unexpected)}." + ) + + return normalized + + +# --------------------------------------------------------------------------- +# Response extraction helpers +# --------------------------------------------------------------------------- + +def _extract_responses_message_text(item: Any) -> str: + """Extract assistant text from a Responses message output item.""" + content = getattr(item, "content", None) + if not isinstance(content, list): + return "" + + chunks: List[str] = [] + for part in content: + ptype = getattr(part, "type", None) + if ptype not in {"output_text", "text"}: + continue + text = getattr(part, "text", None) + if isinstance(text, str) and text: + chunks.append(text) + return "".join(chunks).strip() + + +def _extract_responses_reasoning_text(item: Any) -> str: + """Extract a compact reasoning text from a Responses reasoning item.""" + summary = getattr(item, "summary", None) + if isinstance(summary, list): + chunks: List[str] = [] + for part in summary: + text = getattr(part, "text", None) + if isinstance(text, str) and text: + chunks.append(text) + if chunks: + return "\n".join(chunks).strip() + text = getattr(item, "text", None) + if isinstance(text, str) and text: + return text.strip() + return "" + + +# --------------------------------------------------------------------------- +# Full response normalization +# --------------------------------------------------------------------------- + +def _normalize_codex_response(response: Any) -> tuple[Any, str]: + """Normalize a Responses API object to an assistant_message-like object.""" + output = getattr(response, "output", None) + if not isinstance(output, list) or not output: + # The Codex backend can return empty output when the answer was + # delivered entirely via stream events. Check output_text as a + # last-resort fallback before raising. + out_text = getattr(response, "output_text", None) + if isinstance(out_text, str) and out_text.strip(): + logger.debug( + "Codex response has empty output but output_text is present (%d chars); " + "synthesizing output item.", len(out_text.strip()), + ) + output = [SimpleNamespace( + type="message", role="assistant", status="completed", + content=[SimpleNamespace(type="output_text", text=out_text.strip())], + )] + response.output = output + else: + raise RuntimeError("Responses API returned no output items") + + response_status = getattr(response, "status", None) + if isinstance(response_status, str): + response_status = response_status.strip().lower() + else: + response_status = None + + if response_status in {"failed", "cancelled"}: + error_obj = getattr(response, "error", None) + if isinstance(error_obj, dict): + error_msg = error_obj.get("message") or str(error_obj) + else: + error_msg = str(error_obj) if error_obj else f"Responses API returned status '{response_status}'" + raise RuntimeError(error_msg) + + content_parts: List[str] = [] + reasoning_parts: List[str] = [] + reasoning_items_raw: List[Dict[str, Any]] = [] + tool_calls: List[Any] = [] + has_incomplete_items = response_status in {"queued", "in_progress", "incomplete"} + saw_commentary_phase = False + saw_final_answer_phase = False + + for item in output: + item_type = getattr(item, "type", None) + item_status = getattr(item, "status", None) + if isinstance(item_status, str): + item_status = item_status.strip().lower() + else: + item_status = None + + if item_status in {"queued", "in_progress", "incomplete"}: + has_incomplete_items = True + + if item_type == "message": + item_phase = getattr(item, "phase", None) + if isinstance(item_phase, str): + normalized_phase = item_phase.strip().lower() + if normalized_phase in {"commentary", "analysis"}: + saw_commentary_phase = True + elif normalized_phase in {"final_answer", "final"}: + saw_final_answer_phase = True + message_text = _extract_responses_message_text(item) + if message_text: + content_parts.append(message_text) + elif item_type == "reasoning": + reasoning_text = _extract_responses_reasoning_text(item) + if reasoning_text: + reasoning_parts.append(reasoning_text) + # Capture the full reasoning item for multi-turn continuity. + # encrypted_content is an opaque blob the API needs back on + # subsequent turns to maintain coherent reasoning chains. + encrypted = getattr(item, "encrypted_content", None) + if isinstance(encrypted, str) and encrypted: + raw_item = {"type": "reasoning", "encrypted_content": encrypted} + item_id = getattr(item, "id", None) + if isinstance(item_id, str) and item_id: + raw_item["id"] = item_id + # Capture summary — required by the API when replaying reasoning items + summary = getattr(item, "summary", None) + if isinstance(summary, list): + raw_summary = [] + for part in summary: + text = getattr(part, "text", None) + if isinstance(text, str): + raw_summary.append({"type": "summary_text", "text": text}) + raw_item["summary"] = raw_summary + reasoning_items_raw.append(raw_item) + elif item_type == "function_call": + if item_status in {"queued", "in_progress", "incomplete"}: + continue + fn_name = getattr(item, "name", "") or "" + arguments = getattr(item, "arguments", "{}") + if not isinstance(arguments, str): + arguments = json.dumps(arguments, ensure_ascii=False) + raw_call_id = getattr(item, "call_id", None) + raw_item_id = getattr(item, "id", None) + embedded_call_id, _ = _split_responses_tool_id(raw_item_id) + call_id = raw_call_id if isinstance(raw_call_id, str) and raw_call_id.strip() else embedded_call_id + if not isinstance(call_id, str) or not call_id.strip(): + call_id = _deterministic_call_id(fn_name, arguments, len(tool_calls)) + call_id = call_id.strip() + response_item_id = raw_item_id if isinstance(raw_item_id, str) else None + response_item_id = _derive_responses_function_call_id(call_id, response_item_id) + tool_calls.append(SimpleNamespace( + id=call_id, + call_id=call_id, + response_item_id=response_item_id, + type="function", + function=SimpleNamespace(name=fn_name, arguments=arguments), + )) + elif item_type == "custom_tool_call": + fn_name = getattr(item, "name", "") or "" + arguments = getattr(item, "input", "{}") + if not isinstance(arguments, str): + arguments = json.dumps(arguments, ensure_ascii=False) + raw_call_id = getattr(item, "call_id", None) + raw_item_id = getattr(item, "id", None) + embedded_call_id, _ = _split_responses_tool_id(raw_item_id) + call_id = raw_call_id if isinstance(raw_call_id, str) and raw_call_id.strip() else embedded_call_id + if not isinstance(call_id, str) or not call_id.strip(): + call_id = _deterministic_call_id(fn_name, arguments, len(tool_calls)) + call_id = call_id.strip() + response_item_id = raw_item_id if isinstance(raw_item_id, str) else None + response_item_id = _derive_responses_function_call_id(call_id, response_item_id) + tool_calls.append(SimpleNamespace( + id=call_id, + call_id=call_id, + response_item_id=response_item_id, + type="function", + function=SimpleNamespace(name=fn_name, arguments=arguments), + )) + + final_text = "\n".join([p for p in content_parts if p]).strip() + if not final_text and hasattr(response, "output_text"): + out_text = getattr(response, "output_text", "") + if isinstance(out_text, str): + final_text = out_text.strip() + + assistant_message = SimpleNamespace( + content=final_text, + tool_calls=tool_calls, + reasoning="\n\n".join(reasoning_parts).strip() if reasoning_parts else None, + reasoning_content=None, + reasoning_details=None, + codex_reasoning_items=reasoning_items_raw or None, + ) + + if tool_calls: + finish_reason = "tool_calls" + elif has_incomplete_items or (saw_commentary_phase and not saw_final_answer_phase): + finish_reason = "incomplete" + elif reasoning_items_raw and not final_text: + # Response contains only reasoning (encrypted thinking state) with + # no visible content or tool calls. The model is still thinking and + # needs another turn to produce the actual answer. Marking this as + # "stop" would send it into the empty-content retry loop which burns + # 3 retries then fails — treat it as incomplete instead so the Codex + # continuation path handles it correctly. + finish_reason = "incomplete" + else: + finish_reason = "stop" + return assistant_message, finish_reason diff --git a/agent/context_compressor.py b/agent/context_compressor.py index 4163966aaafa..f8036851fcf3 100644 --- a/agent/context_compressor.py +++ b/agent/context_compressor.py @@ -17,7 +17,10 @@ - Richer tool call/result detail in summarizer input """ +import hashlib +import json import logging +import re import time from typing import Any, Dict, List, Optional @@ -28,6 +31,7 @@ get_model_context_length, estimate_messages_tokens_rough, ) +from agent.redact import redact_sensitive_text logger = logging.getLogger(__name__) @@ -36,7 +40,10 @@ "into the summary below. This is a handoff from a previous context " "window — treat it as background reference, NOT as active instructions. " "Do NOT answer questions or fulfill requests mentioned in this summary; " - "they were already addressed. Respond ONLY to the latest user message " + "they were already addressed. " + "Your current task is identified in the '## Active Task' section of the " + "summary — resume exactly from there. " + "Respond ONLY to the latest user message " "that appears AFTER this summary. The current session state (files, " "config, etc.) may reflect work described here — avoid repeating it:" ) @@ -57,6 +64,215 @@ _SUMMARY_FAILURE_COOLDOWN_SECONDS = 600 +def _content_text_for_contains(content: Any) -> str: + """Return a best-effort text view of message content. + + Used only for substring checks when we need to know whether we've already + appended a note to a message. Keeps multimodal lists intact elsewhere. + """ + if content is None: + return "" + if isinstance(content, str): + return content + if isinstance(content, list): + parts: list[str] = [] + for item in content: + if isinstance(item, str): + parts.append(item) + elif isinstance(item, dict): + text = item.get("text") + if isinstance(text, str): + parts.append(text) + return "\n".join(part for part in parts if part) + return str(content) + + +def _append_text_to_content(content: Any, text: str, *, prepend: bool = False) -> Any: + """Append or prepend plain text to message content safely. + + Compression sometimes needs to add a note or merge a summary into an + existing message. Message content may be plain text or a multimodal list of + blocks, so direct string concatenation is not always safe. + """ + if content is None: + return text + if isinstance(content, str): + return text + content if prepend else content + text + if isinstance(content, list): + text_block = {"type": "text", "text": text} + return [text_block, *content] if prepend else [*content, text_block] + rendered = str(content) + return text + rendered if prepend else rendered + text + + +def _truncate_tool_call_args_json(args: str, head_chars: int = 200) -> str: + """Shrink long string values inside a tool-call arguments JSON blob while + preserving JSON validity. + + The ``function.arguments`` field on a tool call is a JSON-encoded string + passed through to the LLM provider; downstream providers strictly + validate it and return a non-retryable 400 when it is not well-formed. + An earlier implementation sliced the raw JSON at a fixed byte offset and + appended ``...[truncated]`` — which routinely produced strings like:: + + {"path": "/foo/bar", "content": "# long markdown + ...[truncated] + + i.e. an unterminated string and a missing closing brace. MiniMax, for + example, rejects this with ``invalid function arguments json string`` + and the session gets stuck re-sending the same broken history on every + turn. See issue #11762 for the observed loop. + + This helper parses the arguments, shrinks long string leaves inside the + parsed structure, and re-serialises. Non-string values (paths, ints, + booleans) are preserved intact. If the arguments are not valid JSON + to begin with — some model backends use non-JSON tool arguments — the + original string is returned unchanged rather than replaced with + something neither we nor the backend can parse. + """ + try: + parsed = json.loads(args) + except (ValueError, TypeError): + return args + + def _shrink(obj: Any) -> Any: + if isinstance(obj, str): + if len(obj) > head_chars: + return obj[:head_chars] + "...[truncated]" + return obj + if isinstance(obj, dict): + return {k: _shrink(v) for k, v in obj.items()} + if isinstance(obj, list): + return [_shrink(v) for v in obj] + return obj + + shrunken = _shrink(parsed) + # ensure_ascii=False preserves CJK/emoji instead of bloating with \uXXXX + return json.dumps(shrunken, ensure_ascii=False) + + +def _summarize_tool_result(tool_name: str, tool_args: str, tool_content: str) -> str: + """Create an informative 1-line summary of a tool call + result. + + Used during the pre-compression pruning pass to replace large tool + outputs with a short but useful description of what the tool did, + rather than a generic placeholder that carries zero information. + + Returns strings like:: + + [terminal] ran `npm test` -> exit 0, 47 lines output + [read_file] read config.py from line 1 (1,200 chars) + [search_files] content search for 'compress' in agent/ -> 12 matches + """ + try: + args = json.loads(tool_args) if tool_args else {} + except (json.JSONDecodeError, TypeError): + args = {} + + content = tool_content or "" + content_len = len(content) + line_count = content.count("\n") + 1 if content.strip() else 0 + + if tool_name == "terminal": + cmd = args.get("command", "") + if len(cmd) > 80: + cmd = cmd[:77] + "..." + exit_match = re.search(r'"exit_code"\s*:\s*(-?\d+)', content) + exit_code = exit_match.group(1) if exit_match else "?" + return f"[terminal] ran `{cmd}` -> exit {exit_code}, {line_count} lines output" + + if tool_name == "read_file": + path = args.get("path", "?") + offset = args.get("offset", 1) + return f"[read_file] read {path} from line {offset} ({content_len:,} chars)" + + if tool_name == "write_file": + path = args.get("path", "?") + written_lines = args.get("content", "").count("\n") + 1 if args.get("content") else "?" + return f"[write_file] wrote to {path} ({written_lines} lines)" + + if tool_name == "search_files": + pattern = args.get("pattern", "?") + path = args.get("path", ".") + target = args.get("target", "content") + match_count = re.search(r'"total_count"\s*:\s*(\d+)', content) + count = match_count.group(1) if match_count else "?" + return f"[search_files] {target} search for '{pattern}' in {path} -> {count} matches" + + if tool_name == "patch": + path = args.get("path", "?") + mode = args.get("mode", "replace") + return f"[patch] {mode} in {path} ({content_len:,} chars result)" + + if tool_name in ("browser_navigate", "browser_click", "browser_snapshot", + "browser_type", "browser_scroll", "browser_vision"): + url = args.get("url", "") + ref = args.get("ref", "") + detail = f" {url}" if url else (f" ref={ref}" if ref else "") + return f"[{tool_name}]{detail} ({content_len:,} chars)" + + if tool_name == "web_search": + query = args.get("query", "?") + return f"[web_search] query='{query}' ({content_len:,} chars result)" + + if tool_name == "web_extract": + urls = args.get("urls", []) + url_desc = urls[0] if isinstance(urls, list) and urls else "?" + if isinstance(urls, list) and len(urls) > 1: + url_desc += f" (+{len(urls) - 1} more)" + return f"[web_extract] {url_desc} ({content_len:,} chars)" + + if tool_name == "delegate_task": + goal = args.get("goal", "") + if len(goal) > 60: + goal = goal[:57] + "..." + return f"[delegate_task] '{goal}' ({content_len:,} chars result)" + + if tool_name == "execute_code": + code_preview = (args.get("code") or "")[:60].replace("\n", " ") + if len(args.get("code", "")) > 60: + code_preview += "..." + return f"[execute_code] `{code_preview}` ({line_count} lines output)" + + if tool_name in ("skill_view", "skills_list", "skill_manage"): + name = args.get("name", "?") + return f"[{tool_name}] name={name} ({content_len:,} chars)" + + if tool_name == "vision_analyze": + question = args.get("question", "")[:50] + return f"[vision_analyze] '{question}' ({content_len:,} chars)" + + if tool_name == "memory": + action = args.get("action", "?") + target = args.get("target", "?") + return f"[memory] {action} on {target}" + + if tool_name == "todo": + return "[todo] updated task list" + + if tool_name == "clarify": + return "[clarify] asked user a question" + + if tool_name == "text_to_speech": + return f"[text_to_speech] generated audio ({content_len:,} chars)" + + if tool_name == "cronjob": + action = args.get("action", "?") + return f"[cronjob] {action}" + + if tool_name == "process": + action = args.get("action", "?") + sid = args.get("session_id", "?") + return f"[process] {action} session={sid}" + + # Generic fallback + first_arg = "" + for k, v in list(args.items())[:2]: + sv = str(v)[:40] + first_arg += f" {k}={sv}" + return f"[{tool_name}]{first_arg} ({content_len:,} chars result)" + + class ContextCompressor(ContextEngine): """Default context engine — compresses conversation context via lossy summarization. @@ -78,6 +294,8 @@ def on_session_reset(self) -> None: self._context_probed = False self._context_probe_persistable = False self._previous_summary = None + self._last_compression_savings_pct = 100.0 + self._ineffective_compression_count = 0 def update_model( self, @@ -167,6 +385,9 @@ def __init__( # Stores the previous compaction summary for iterative updates self._previous_summary: Optional[str] = None + # Anti-thrashing: track whether last compression was effective + self._last_compression_savings_pct: float = 100.0 + self._ineffective_compression_count: int = 0 self._summary_failure_cooldown_until: float = 0.0 def update_from_response(self, usage: Dict[str, Any]): @@ -175,9 +396,26 @@ def update_from_response(self, usage: Dict[str, Any]): self.last_completion_tokens = usage.get("completion_tokens", 0) def should_compress(self, prompt_tokens: int = None) -> bool: - """Check if context exceeds the compression threshold.""" + """Check if context exceeds the compression threshold. + + Includes anti-thrashing protection: if the last two compressions + each saved less than 10%, skip compression to avoid infinite loops + where each pass removes only 1-2 messages. + """ tokens = prompt_tokens if prompt_tokens is not None else self.last_prompt_tokens - return tokens >= self.threshold_tokens + if tokens < self.threshold_tokens: + return False + # Anti-thrashing: back off if recent compressions were ineffective + if self._ineffective_compression_count >= 2: + if not self.quiet_mode: + logger.warning( + "Compression skipped — last %d compressions saved <10%% each. " + "Consider /new to start a fresh session, or /compress " + "for focused compression.", + self._ineffective_compression_count, + ) + return False + return True # ------------------------------------------------------------------ # Tool output pruning (cheap pre-pass, no LLM call) @@ -187,7 +425,16 @@ def _prune_old_tool_results( self, messages: List[Dict[str, Any]], protect_tail_count: int, protect_tail_tokens: int | None = None, ) -> tuple[List[Dict[str, Any]], int]: - """Replace old tool result contents with a short placeholder. + """Replace old tool result contents with informative 1-line summaries. + + Instead of a generic placeholder, generates a summary like:: + + [terminal] ran `npm test` -> exit 0, 47 lines output + [read_file] read config.py from line 1 (3,400 chars) + + Also deduplicates identical tool results (e.g. reading the same file + 5x keeps only the newest full copy) and truncates large tool_call + arguments in assistant messages outside the protected tail. Walks backward from the end, protecting the most recent messages that fall within ``protect_tail_tokens`` (when provided) OR the last @@ -203,6 +450,22 @@ def _prune_old_tool_results( result = [m.copy() for m in messages] pruned = 0 + # Build index: tool_call_id -> (tool_name, arguments_json) + call_id_to_tool: Dict[str, tuple] = {} + for msg in result: + if msg.get("role") == "assistant": + for tc in msg.get("tool_calls") or []: + if isinstance(tc, dict): + cid = tc.get("id", "") + fn = tc.get("function", {}) + call_id_to_tool[cid] = (fn.get("name", "unknown"), fn.get("arguments", "")) + else: + cid = getattr(tc, "id", "") or "" + fn = getattr(tc, "function", None) + name = getattr(fn, "name", "unknown") if fn else "unknown" + args_str = getattr(fn, "arguments", "") if fn else "" + call_id_to_tool[cid] = (name, args_str) + # Determine the prune boundary if protect_tail_tokens is not None and protect_tail_tokens > 0: # Token-budget approach: walk backward accumulating tokens @@ -211,7 +474,8 @@ def _prune_old_tool_results( min_protect = min(protect_tail_count, len(result) - 1) for i in range(len(result) - 1, -1, -1): msg = result[i] - content_len = len(msg.get("content") or "") + raw_content = msg.get("content") or "" + content_len = sum(len(p.get("text", "")) for p in raw_content) if isinstance(raw_content, list) else len(raw_content) msg_tokens = content_len // _CHARS_PER_TOKEN + 10 for tc in msg.get("tool_calls") or []: if isinstance(tc, dict): @@ -226,18 +490,76 @@ def _prune_old_tool_results( else: prune_boundary = len(result) - protect_tail_count + # Pass 1: Deduplicate identical tool results. + # When the same file is read multiple times, keep only the most recent + # full copy and replace older duplicates with a back-reference. + content_hashes: dict = {} # hash -> (index, tool_call_id) + for i in range(len(result) - 1, -1, -1): + msg = result[i] + if msg.get("role") != "tool": + continue + content = msg.get("content") or "" + # Skip multimodal content (list of content blocks) + if isinstance(content, list): + continue + if len(content) < 200: + continue + h = hashlib.md5(content.encode("utf-8", errors="replace")).hexdigest()[:12] + if h in content_hashes: + # This is an older duplicate — replace with back-reference + result[i] = {**msg, "content": "[Duplicate tool output — same content as a more recent call]"} + pruned += 1 + else: + content_hashes[h] = (i, msg.get("tool_call_id", "?")) + + # Pass 2: Replace old tool results with informative summaries for i in range(prune_boundary): msg = result[i] if msg.get("role") != "tool": continue content = msg.get("content", "") + # Skip multimodal content (list of content blocks) + if isinstance(content, list): + continue if not content or content == _PRUNED_TOOL_PLACEHOLDER: continue + # Skip already-deduplicated or previously-summarized results + if content.startswith("[Duplicate tool output"): + continue # Only prune if the content is substantial (>200 chars) if len(content) > 200: - result[i] = {**msg, "content": _PRUNED_TOOL_PLACEHOLDER} + call_id = msg.get("tool_call_id", "") + tool_name, tool_args = call_id_to_tool.get(call_id, ("unknown", "")) + summary = _summarize_tool_result(tool_name, tool_args, content) + result[i] = {**msg, "content": summary} pruned += 1 + # Pass 3: Truncate large tool_call arguments in assistant messages + # outside the protected tail. write_file with 50KB content, for + # example, survives pruning entirely without this. + # + # The shrinking is done inside the parsed JSON structure so the + # result remains valid JSON — otherwise downstream providers 400 + # on every subsequent turn until the broken call falls out of + # the window. See ``_truncate_tool_call_args_json`` docstring. + for i in range(prune_boundary): + msg = result[i] + if msg.get("role") != "assistant" or not msg.get("tool_calls"): + continue + new_tcs = [] + modified = False + for tc in msg["tool_calls"]: + if isinstance(tc, dict): + args = tc.get("function", {}).get("arguments", "") + if len(args) > 500: + new_args = _truncate_tool_call_args_json(args) + if new_args != args: + tc = {**tc, "function": {**tc["function"], "arguments": new_args}} + modified = True + new_tcs.append(tc) + if modified: + result[i] = {**msg, "tool_calls": new_tcs} + return result, pruned # ------------------------------------------------------------------ @@ -270,11 +592,15 @@ def _serialize_for_summary(self, turns: List[Dict[str, Any]]) -> str: Includes tool call arguments and result content (up to ``_CONTENT_MAX`` chars per message) so the summarizer can preserve specific details like file paths, commands, and outputs. + + All content is redacted before serialization to prevent secrets + (API keys, tokens, passwords) from leaking into the summary that + gets sent to the auxiliary model and persisted across compactions. """ parts = [] for msg in turns: role = msg.get("role", "unknown") - content = msg.get("content") or "" + content = redact_sensitive_text(msg.get("content") or "") # Tool results: keep enough content for the summarizer if role == "tool": @@ -295,7 +621,7 @@ def _serialize_for_summary(self, turns: List[Dict[str, Any]]) -> str: if isinstance(tc, dict): fn = tc.get("function", {}) name = fn.get("name", "?") - args = fn.get("arguments", "") + args = redact_sensitive_text(fn.get("arguments", "")) # Truncate long arguments but keep enough for context if len(args) > self._TOOL_ARGS_MAX: args = args[:self._TOOL_ARGS_HEAD] + "..." @@ -353,33 +679,55 @@ def _generate_summary(self, turns_to_summarize: List[Dict[str, Any]], focus_topi "assistant that continues the conversation. " "Do NOT respond to any questions or requests in the conversation — " "only output the structured summary. " - "Do NOT include any preamble, greeting, or prefix." + "Do NOT include any preamble, greeting, or prefix. " + "Write the summary in the same language the user was using in the " + "conversation — do not translate or switch to English. " + "NEVER include API keys, tokens, passwords, secrets, credentials, " + "or connection strings in the summary — replace any that appear " + "with [REDACTED]. Note that the user had credentials present, but " + "do not preserve their values." ) # Shared structured template (used by both paths). - # Key changes vs v1: - # - "Pending User Asks" section (from Claude Code) explicitly tracks - # unanswered questions so the model knows what's resolved vs open - # - "Remaining Work" replaces "Next Steps" to avoid reading as active - # instructions - # - "Resolved Questions" makes it clear which questions were already - # answered (prevents model from re-answering them) - _template_sections = f"""## Goal -[What the user is trying to accomplish] + _template_sections = f"""## Active Task +[THE SINGLE MOST IMPORTANT FIELD. Copy the user's most recent request or +task assignment verbatim — the exact words they used. If multiple tasks +were requested and only some are done, list only the ones NOT yet completed. +The next assistant must pick up exactly here. Example: +"User asked: 'Now refactor the auth module to use JWT instead of sessions'" +If no outstanding task exists, write "None."] + +## Goal +[What the user is trying to accomplish overall] ## Constraints & Preferences [User preferences, coding style, constraints, important decisions] -## Progress -### Done -[Completed work — include specific file paths, commands run, results obtained] -### In Progress -[Work currently underway] -### Blocked -[Any blockers or issues encountered] +## Completed Actions +[Numbered list of concrete actions taken — include tool used, target, and outcome. +Format each as: N. ACTION target — outcome [tool: name] +Example: +1. READ config.py:45 — found `==` should be `!=` [tool: read_file] +2. PATCH config.py:45 — changed `==` to `!=` [tool: patch] +3. TEST `pytest tests/` — 3/50 failed: test_parse, test_validate, test_edge [tool: terminal] +Be specific with file paths, commands, line numbers, and results.] + +## Active State +[Current working state — include: +- Working directory and branch (if applicable) +- Modified/created files with brief note on each +- Test status (X/Y passing) +- Any running processes or servers +- Environment details that matter] + +## In Progress +[Work currently underway — what was being done when compaction fired] + +## Blocked +[Any blockers, errors, or issues not yet resolved. Include exact error messages.] ## Key Decisions -[Important technical decisions and why they were made] +[Important technical decisions and WHY they were made] ## Resolved Questions [Questions the user asked that were ALREADY answered — include the answer so the next assistant does not re-answer them] @@ -394,12 +742,9 @@ def _generate_summary(self, turns_to_summarize: List[Dict[str, Any]], focus_topi [What remains to be done — framed as context, not instructions] ## Critical Context -[Any specific values, error messages, configuration details, or data that would be lost without explicit preservation] - -## Tools & Patterns -[Which tools were used, how they were used effectively, and any tool-specific discoveries] +[Any specific values, error messages, configuration details, or data that would be lost without explicit preservation. NEVER include API keys, tokens, passwords, or credentials — write [REDACTED] instead.] -Target ~{summary_budget} tokens. Be specific — include file paths, command outputs, error messages, and concrete values rather than vague descriptions. +Target ~{summary_budget} tokens. Be CONCRETE — include file paths, command outputs, error messages, line numbers, and specific values. Avoid vague descriptions like "made some changes" — say exactly what changed. Write only the summary body. Do not include any preamble or prefix.""" @@ -415,7 +760,7 @@ def _generate_summary(self, turns_to_summarize: List[Dict[str, Any]], focus_topi NEW TURNS TO INCORPORATE: {content_to_summarize} -Update the summary using this exact structure. PRESERVE all existing information that is still relevant. ADD new progress. Move items from "In Progress" to "Done" when completed. Move answered questions to "Resolved Questions". Remove information only if it is clearly obsolete. +Update the summary using this exact structure. PRESERVE all existing information that is still relevant. ADD new completed actions to the numbered list (continue numbering). Move items from "In Progress" to "Completed Actions" when done. Move answered questions to "Resolved Questions". Update "Active State" to reflect current state. Remove information only if it is clearly obsolete. CRITICAL: Update "## Active Task" to reflect the user's most recent unfulfilled request — this is the most important field for task continuity. {_template_sections}""" else: @@ -437,7 +782,7 @@ def _generate_summary(self, turns_to_summarize: List[Dict[str, Any]], focus_topi prompt += f""" FOCUS TOPIC: "{focus_topic}" -The user has requested that this compaction PRIORITISE preserving all information related to the focus topic above. For content related to "{focus_topic}", include full detail — exact values, file paths, command outputs, error messages, and decisions. For content NOT related to the focus topic, summarise more aggressively (brief one-liners or omit if truly irrelevant). The focus topic sections should receive roughly 60-70% of the summary token budget.""" +The user has requested that this compaction PRIORITISE preserving all information related to the focus topic above. For content related to "{focus_topic}", include full detail — exact values, file paths, command outputs, error messages, and decisions. For content NOT related to the focus topic, summarise more aggressively (brief one-liners or omit if truly irrelevant). The focus topic sections should receive roughly 60-70% of the summary token budget. Even for the focus topic, NEVER preserve API keys, tokens, passwords, or credentials — use [REDACTED].""" try: call_kwargs = { @@ -450,7 +795,7 @@ def _generate_summary(self, turns_to_summarize: List[Dict[str, Any]], focus_topi "api_mode": self.api_mode, }, "messages": [{"role": "user", "content": prompt}], - "max_tokens": summary_budget * 2, + "max_tokens": int(summary_budget * 1.3), # timeout resolved from auxiliary.compression.timeout config by call_llm } if self.summary_model: @@ -460,12 +805,16 @@ def _generate_summary(self, turns_to_summarize: List[Dict[str, Any]], focus_topi # Handle cases where content is not a string (e.g., dict from llama.cpp) if not isinstance(content, str): content = str(content) if content else "" - summary = content.strip() + # Redact the summary output as well — the summarizer LLM may + # ignore prompt instructions and echo back secrets verbatim. + summary = redact_sensitive_text(content.strip()) # Store for iterative updates on next compaction self._previous_summary = summary self._summary_failure_cooldown_until = 0.0 + self._summary_model_fallen_back = False return self._with_summary_prefix(summary) except RuntimeError: + # No provider configured — long cooldown, unlikely to self-resolve self._summary_failure_cooldown_until = time.monotonic() + _SUMMARY_FAILURE_COOLDOWN_SECONDS logging.warning("Context compression: no provider available for " "summary. Middle turns will be dropped without summary " @@ -473,12 +822,42 @@ def _generate_summary(self, turns_to_summarize: List[Dict[str, Any]], focus_topi _SUMMARY_FAILURE_COOLDOWN_SECONDS) return None except Exception as e: - self._summary_failure_cooldown_until = time.monotonic() + _SUMMARY_FAILURE_COOLDOWN_SECONDS + # If the summary model is different from the main model and the + # error looks permanent (model not found, 503, 404), fall back to + # using the main model instead of entering cooldown that leaves + # context growing unbounded. (#8620 sub-issue 4) + _status = getattr(e, "status_code", None) or getattr(getattr(e, "response", None), "status_code", None) + _err_str = str(e).lower() + _is_model_not_found = ( + _status in (404, 503) + or "model_not_found" in _err_str + or "does not exist" in _err_str + or "no available channel" in _err_str + ) + if ( + _is_model_not_found + and self.summary_model + and self.summary_model != self.model + and not getattr(self, "_summary_model_fallen_back", False) + ): + self._summary_model_fallen_back = True + logging.warning( + "Summary model '%s' not available (%s). " + "Falling back to main model '%s' for compression.", + self.summary_model, e, self.model, + ) + self.summary_model = "" # empty = use main model + self._summary_failure_cooldown_until = 0.0 # no cooldown + return self._generate_summary(turns_to_summarize, focus_topic=focus_topic) # retry immediately + + # Transient errors (timeout, rate limit, network) — shorter cooldown + _transient_cooldown = 60 + self._summary_failure_cooldown_until = time.monotonic() + _transient_cooldown logging.warning( "Failed to generate context summary: %s. " "Further summary attempts paused for %d seconds.", e, - _SUMMARY_FAILURE_COOLDOWN_SECONDS, + _transient_cooldown, ) return None @@ -601,6 +980,62 @@ def _align_boundary_backward(self, messages: List[Dict[str, Any]], idx: int) -> # Tail protection by token budget # ------------------------------------------------------------------ + def _find_last_user_message_idx( + self, messages: List[Dict[str, Any]], head_end: int + ) -> int: + """Return the index of the last user-role message at or after *head_end*, or -1.""" + for i in range(len(messages) - 1, head_end - 1, -1): + if messages[i].get("role") == "user": + return i + return -1 + + def _ensure_last_user_message_in_tail( + self, + messages: List[Dict[str, Any]], + cut_idx: int, + head_end: int, + ) -> int: + """Guarantee the most recent user message is in the protected tail. + + Context compressor bug (#10896): ``_align_boundary_backward`` can pull + ``cut_idx`` past a user message when it tries to keep tool_call/result + groups together. If the last user message ends up in the *compressed* + middle region the LLM summariser writes it into "Pending User Asks", + but ``SUMMARY_PREFIX`` tells the next model to respond only to user + messages *after* the summary — so the task effectively disappears from + the active context, causing the agent to stall, repeat completed work, + or silently drop the user's latest request. + + Fix: if the last user-role message is not already in the tail + (``messages[cut_idx:]``), walk ``cut_idx`` back to include it. We + then re-align backward one more time to avoid splitting any + tool_call/result group that immediately precedes the user message. + """ + last_user_idx = self._find_last_user_message_idx(messages, head_end) + if last_user_idx < 0: + # No user message found beyond head — nothing to anchor. + return cut_idx + + if last_user_idx >= cut_idx: + # Already in the tail; nothing to do. + return cut_idx + + # The last user message is in the middle (compressed) region. + # Pull cut_idx back to it directly — a user message is already a + # clean boundary (no tool_call/result splitting risk), so there is no + # need to call _align_boundary_backward here; doing so would + # unnecessarily pull the cut further back into the preceding + # assistant + tool_calls group. + if not self.quiet_mode: + logger.debug( + "Anchoring tail cut to last user message at index %d " + "(was %d) to prevent active-task loss after compression", + last_user_idx, + cut_idx, + ) + # Safety: never go back into the head region. + return max(last_user_idx, head_end + 1) + def _find_tail_cut_by_tokens( self, messages: List[Dict[str, Any]], head_end: int, token_budget: int | None = None, @@ -618,7 +1053,8 @@ def _find_tail_cut_by_tokens( read, etc.). If even the minimum 3 messages exceed 1.5x the budget the cut is placed right after the head so compression still runs. - Never cuts inside a tool_call/result group. + Never cuts inside a tool_call/result group. Always ensures the most + recent user message is in the tail (see ``_ensure_last_user_message_in_tail``). """ if token_budget is None: token_budget = self.tail_token_budget @@ -657,6 +1093,10 @@ def _find_tail_cut_by_tokens( # Align to avoid splitting tool groups cut_idx = self._align_boundary_backward(messages, cut_idx) + # Ensure the most recent user message is always in the tail so the + # active task is never lost to compression (fixes #10896). + cut_idx = self._ensure_last_user_message_in_tail(messages, cut_idx, head_end) + return max(cut_idx, head_end + 1) # ------------------------------------------------------------------ @@ -744,11 +1184,14 @@ def compress(self, messages: List[Dict[str, Any]], current_tokens: int = None, f compressed = [] for i in range(compress_start): msg = messages[i].copy() - if i == 0 and msg.get("role") == "system" and self.compression_count == 0: - msg["content"] = ( - (msg.get("content") or "") - + "\n\n[Note: Some earlier conversation turns have been compacted into a handoff summary to preserve context space. The current session state may still reflect earlier work, so build on that summary and state rather than re-doing work.]" - ) + if i == 0 and msg.get("role") == "system": + existing = msg.get("content") + _compression_note = "[Note: Some earlier conversation turns have been compacted into a handoff summary to preserve context space. The current session state may still reflect earlier work, so build on that summary and state rather than re-doing work.]" + if _compression_note not in _content_text_for_contains(existing): + msg["content"] = _append_text_to_content( + existing, + "\n\n" + _compression_note if isinstance(existing, str) and existing else _compression_note, + ) compressed.append(msg) # If LLM summary failed, insert a static fallback so the model @@ -792,12 +1235,15 @@ def compress(self, messages: List[Dict[str, Any]], current_tokens: int = None, f for i in range(compress_end, n_messages): msg = messages[i].copy() if _merge_summary_into_tail and i == compress_end: - original = msg.get("content") or "" - msg["content"] = ( + merged_prefix = ( summary + "\n\n--- END OF CONTEXT SUMMARY — " "respond to the message below, not the summary above ---\n\n" - + original + ) + msg["content"] = _append_text_to_content( + msg.get("content"), + merged_prefix, + prepend=True, ) _merge_summary_into_tail = False compressed.append(msg) @@ -806,14 +1252,24 @@ def compress(self, messages: List[Dict[str, Any]], current_tokens: int = None, f compressed = self._sanitize_tool_pairs(compressed) + new_estimate = estimate_messages_tokens_rough(compressed) + saved_estimate = display_tokens - new_estimate + + # Anti-thrashing: track compression effectiveness + savings_pct = (saved_estimate / display_tokens * 100) if display_tokens > 0 else 0 + self._last_compression_savings_pct = savings_pct + if savings_pct < 10: + self._ineffective_compression_count += 1 + else: + self._ineffective_compression_count = 0 + if not self.quiet_mode: - new_estimate = estimate_messages_tokens_rough(compressed) - saved_estimate = display_tokens - new_estimate logger.info( - "Compressed: %d -> %d messages (~%d tokens saved)", + "Compressed: %d -> %d messages (~%d tokens saved, %.0f%%)", n_messages, len(compressed), saved_estimate, + savings_pct, ) logger.info("Compression #%d complete", self.compression_count) diff --git a/agent/context_engine.py b/agent/context_engine.py index 6cd7275fe9b3..6ae90b6cdf6b 100644 --- a/agent/context_engine.py +++ b/agent/context_engine.py @@ -26,7 +26,7 @@ """ from abc import ABC, abstractmethod -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List class ContextEngine(ABC): diff --git a/agent/context_references.py b/agent/context_references.py index 7ecb90c497f3..50a33a1d7577 100644 --- a/agent/context_references.py +++ b/agent/context_references.py @@ -483,9 +483,7 @@ def _rg_files(path: Path, cwd: Path, limit: int) -> list[Path] | None: text=True, timeout=10, ) - except FileNotFoundError: - return None - except subprocess.TimeoutExpired: + except (FileNotFoundError, OSError, subprocess.TimeoutExpired): return None if result.returncode != 0: return None diff --git a/agent/copilot_acp_client.py b/agent/copilot_acp_client.py index 235fd9a1a5fb..783f949567b5 100644 --- a/agent/copilot_acp_client.py +++ b/agent/copilot_acp_client.py @@ -21,6 +21,9 @@ from types import SimpleNamespace from typing import Any +from agent.file_safety import get_read_block_error, is_write_denied +from agent.redact import redact_sensitive_text + ACP_MARKER_BASE_URL = "acp://copilot" _DEFAULT_TIMEOUT_SECONDS = 900.0 @@ -54,6 +57,18 @@ def _jsonrpc_error(message_id: Any, code: int, message: str) -> dict[str, Any]: } +def _permission_denied(message_id: Any) -> dict[str, Any]: + return { + "jsonrpc": "2.0", + "id": message_id, + "result": { + "outcome": { + "outcome": "cancelled", + } + }, + } + + def _format_messages_as_prompt( messages: list[dict[str, Any]], model: str | None = None, @@ -313,9 +328,25 @@ def _create_chat_completion( tools=tools, tool_choice=tool_choice, ) + # Normalise timeout: run_agent.py may pass an httpx.Timeout object + # (used natively by the OpenAI SDK) rather than a plain float. + if timeout is None: + _effective_timeout = _DEFAULT_TIMEOUT_SECONDS + elif isinstance(timeout, (int, float)): + _effective_timeout = float(timeout) + else: + # httpx.Timeout or similar — pick the largest component so the + # subprocess has enough wall-clock time for the full response. + _candidates = [ + getattr(timeout, attr, None) + for attr in ("read", "write", "connect", "pool", "timeout") + ] + _numeric = [float(v) for v in _candidates if isinstance(v, (int, float))] + _effective_timeout = max(_numeric) if _numeric else _DEFAULT_TIMEOUT_SECONDS + response_text, reasoning_text = self._run_prompt( prompt_text, - timeout_seconds=float(timeout or _DEFAULT_TIMEOUT_SECONDS), + timeout_seconds=_effective_timeout, ) tool_calls, cleaned_text = _extract_tool_calls_from_text(response_text) @@ -370,6 +401,8 @@ def _run_prompt(self, prompt_text: str, *, timeout_seconds: float) -> tuple[str, stderr_tail: deque[str] = deque(maxlen=40) def _stdout_reader() -> None: + if proc.stdout is None: + return for line in proc.stdout: try: inbox.put(json.loads(line)) @@ -517,18 +550,13 @@ def _handle_server_message( params = msg.get("params") or {} if method == "session/request_permission": - response = { - "jsonrpc": "2.0", - "id": message_id, - "result": { - "outcome": { - "outcome": "allow_once", - } - }, - } + response = _permission_denied(message_id) elif method == "fs/read_text_file": try: path = _ensure_path_within_cwd(str(params.get("path") or ""), cwd) + block_error = get_read_block_error(str(path)) + if block_error: + raise PermissionError(block_error) content = path.read_text() if path.exists() else "" line = params.get("line") limit = params.get("limit") @@ -537,6 +565,8 @@ def _handle_server_message( start = line - 1 end = start + limit if isinstance(limit, int) and limit > 0 else None content = "".join(lines[start:end]) + if content: + content = redact_sensitive_text(content) response = { "jsonrpc": "2.0", "id": message_id, @@ -549,6 +579,10 @@ def _handle_server_message( elif method == "fs/write_text_file": try: path = _ensure_path_within_cwd(str(params.get("path") or ""), cwd) + if is_write_denied(str(path)): + raise PermissionError( + f"Write denied: '{path}' is a protected system/credential file." + ) path.parent.mkdir(parents=True, exist_ok=True) path.write_text(str(params.get("content") or "")) response = { diff --git a/agent/credential_pool.py b/agent/credential_pool.py index ea9ad923298f..de8d03185a8a 100644 --- a/agent/credential_pool.py +++ b/agent/credential_pool.py @@ -18,13 +18,10 @@ from hermes_cli.auth import ( CODEX_ACCESS_TOKEN_REFRESH_SKEW_SECONDS, DEFAULT_AGENT_KEY_MIN_TTL_SECONDS, - KIMI_CODE_BASE_URL, PROVIDER_REGISTRY, _auth_store_lock, _codex_access_token_is_expiring, _decode_jwt_claims, - _import_codex_cli_tokens, - _write_codex_cli_tokens, _load_auth_store, _load_provider_state, _resolve_kimi_base_url, @@ -458,39 +455,6 @@ def _sync_anthropic_entry_from_credentials_file(self, entry: PooledCredential) - logger.debug("Failed to sync from credentials file: %s", exc) return entry - def _sync_codex_entry_from_cli(self, entry: PooledCredential) -> PooledCredential: - """Sync an openai-codex pool entry from ~/.codex/auth.json if tokens differ. - - OpenAI OAuth refresh tokens are single-use and rotate on every refresh. - When the Codex CLI (or another Hermes profile) refreshes its token, - the pool entry's refresh_token becomes stale. This method detects that - by comparing against ~/.codex/auth.json and syncing the fresh pair. - """ - if self.provider != "openai-codex": - return entry - try: - cli_tokens = _import_codex_cli_tokens() - if not cli_tokens: - return entry - cli_refresh = cli_tokens.get("refresh_token", "") - cli_access = cli_tokens.get("access_token", "") - if cli_refresh and cli_refresh != entry.refresh_token: - logger.debug("Pool entry %s: syncing tokens from ~/.codex/auth.json (refresh token changed)", entry.id) - updated = replace( - entry, - access_token=cli_access, - refresh_token=cli_refresh, - last_status=None, - last_status_at=None, - last_error_code=None, - ) - self._replace_entry(entry, updated) - self._persist() - return updated - except Exception as exc: - logger.debug("Failed to sync from ~/.codex/auth.json: %s", exc) - return entry - def _sync_device_code_entry_to_auth_store(self, entry: PooledCredential) -> None: """Write refreshed pool entry tokens back to auth.json providers. @@ -586,13 +550,6 @@ def _refresh_entry(self, entry: PooledCredential, *, force: bool) -> Optional[Po except Exception as wexc: logger.debug("Failed to write refreshed token to credentials file: %s", wexc) elif self.provider == "openai-codex": - # Proactively sync from ~/.codex/auth.json before refresh. - # The Codex CLI (or another Hermes profile) may have already - # consumed our refresh_token. Syncing first avoids a - # "refresh_token_reused" error when the CLI has a newer pair. - synced = self._sync_codex_entry_from_cli(entry) - if synced is not entry: - entry = synced refreshed = auth_mod.refresh_codex_oauth_pure( entry.access_token, entry.refresh_token, @@ -678,45 +635,6 @@ def _refresh_entry(self, entry: PooledCredential, *, force: bool) -> Optional[Po # Credentials file had a valid (non-expired) token — use it directly logger.debug("Credentials file has valid token, using without refresh") return synced - # For openai-codex: the refresh_token may have been consumed by - # the Codex CLI between our proactive sync and the refresh call. - # Re-sync and retry once. - if self.provider == "openai-codex": - synced = self._sync_codex_entry_from_cli(entry) - if synced.refresh_token != entry.refresh_token: - logger.debug("Retrying Codex refresh with synced token from ~/.codex/auth.json") - try: - refreshed = auth_mod.refresh_codex_oauth_pure( - synced.access_token, - synced.refresh_token, - ) - updated = replace( - synced, - access_token=refreshed["access_token"], - refresh_token=refreshed["refresh_token"], - last_refresh=refreshed.get("last_refresh"), - last_status=STATUS_OK, - last_status_at=None, - last_error_code=None, - ) - self._replace_entry(synced, updated) - self._persist() - self._sync_device_code_entry_to_auth_store(updated) - try: - _write_codex_cli_tokens( - updated.access_token, - updated.refresh_token, - last_refresh=updated.last_refresh, - ) - except Exception as wexc: - logger.debug("Failed to write refreshed Codex tokens to CLI file (retry): %s", wexc) - return updated - except Exception as retry_exc: - logger.debug("Codex retry refresh also failed: %s", retry_exc) - elif not self._entry_needs_refresh(synced): - logger.debug("Codex CLI has valid token, using without refresh") - self._sync_device_code_entry_to_auth_store(synced) - return synced self._mark_exhausted(entry, None) return None @@ -735,17 +653,6 @@ def _refresh_entry(self, entry: PooledCredential, *, force: bool) -> Optional[Po # _seed_from_singletons() on the next load_pool() sees fresh state # instead of re-seeding stale/consumed tokens. self._sync_device_code_entry_to_auth_store(updated) - # Write refreshed tokens back to ~/.codex/auth.json so Codex CLI - # and VS Code don't hit "refresh_token_reused" on their next refresh. - if self.provider == "openai-codex": - try: - _write_codex_cli_tokens( - updated.access_token, - updated.refresh_token, - last_refresh=updated.last_refresh, - ) - except Exception as wexc: - logger.debug("Failed to write refreshed Codex tokens to CLI file: %s", wexc) return updated def _entry_needs_refresh(self, entry: PooledCredential) -> bool: @@ -791,16 +698,6 @@ def _available_entries(self, *, clear_expired: bool = False, refresh: bool = Fal if synced is not entry: entry = synced cleared_any = True - # For openai-codex entries, sync from ~/.codex/auth.json before - # any status/refresh checks. This picks up tokens refreshed by - # the Codex CLI or another Hermes profile. - if (self.provider == "openai-codex" - and entry.last_status == STATUS_EXHAUSTED - and entry.refresh_token): - synced = self._sync_codex_entry_from_cli(entry) - if synced is not entry: - entry = synced - cleared_any = True if entry.last_status == STATUS_EXHAUSTED: exhausted_until = _exhausted_until(entry) if exhausted_until is not None and now < exhausted_until: @@ -1086,6 +983,14 @@ def _seed_from_singletons(provider: str, entries: List[PooledCredential]) -> Tup active_sources: Set[str] = set() auth_store = _load_auth_store() + # Shared suppression gate — used at every upsert site so + # `hermes auth remove ` is stable across all source types. + try: + from hermes_cli.auth import is_source_suppressed as _is_suppressed + except ImportError: + def _is_suppressed(_p, _s): # type: ignore[misc] + return False + if provider == "anthropic": # Only auto-discover external credentials (Claude Code, Hermes PKCE) # when the user has explicitly configured anthropic as their provider. @@ -1105,13 +1010,8 @@ def _seed_from_singletons(provider: str, entries: List[PooledCredential]) -> Tup ("claude_code", read_claude_code_credentials()), ): if creds and creds.get("accessToken"): - # Check if user explicitly removed this source - try: - from hermes_cli.auth import is_source_suppressed - if is_source_suppressed(provider, source_name): - continue - except ImportError: - pass + if _is_suppressed(provider, source_name): + continue active_sources.add(source_name) changed |= _upsert_entry( entries, @@ -1129,8 +1029,16 @@ def _seed_from_singletons(provider: str, entries: List[PooledCredential]) -> Tup elif provider == "nous": state = _load_provider_state(auth_store, "nous") - if state: + if state and not _is_suppressed(provider, "device_code"): active_sources.add("device_code") + # Prefer a user-supplied label embedded in the singleton state + # (set by persist_nous_credentials(label=...) when the user ran + # `hermes auth add nous --label `). Fall back to the + # auto-derived token fingerprint for logins that didn't supply one. + custom_label = str(state.get("label") or "").strip() + seeded_label = custom_label or label_from_token( + state.get("access_token", ""), "device_code" + ) changed |= _upsert_entry( entries, provider, @@ -1149,30 +1057,83 @@ def _seed_from_singletons(provider: str, entries: List[PooledCredential]) -> Tup "agent_key": state.get("agent_key"), "agent_key_expires_at": state.get("agent_key_expires_at"), "tls": state.get("tls") if isinstance(state.get("tls"), dict) else None, - "label": label_from_token(state.get("access_token", ""), "device_code"), + "label": seeded_label, }, ) + elif provider == "copilot": + # Copilot tokens are resolved dynamically via `gh auth token` or + # env vars (COPILOT_GITHUB_TOKEN / GH_TOKEN). They don't live in + # the auth store or credential pool, so we resolve them here. + try: + from hermes_cli.copilot_auth import resolve_copilot_token + token, source = resolve_copilot_token() + if token: + source_name = "gh_cli" if "gh" in source.lower() else f"env:{source}" + if not _is_suppressed(provider, source_name): + active_sources.add(source_name) + pconfig = PROVIDER_REGISTRY.get(provider) + changed |= _upsert_entry( + entries, + provider, + source_name, + { + "source": source_name, + "auth_type": AUTH_TYPE_API_KEY, + "access_token": token, + "base_url": pconfig.inference_base_url if pconfig else "", + "label": source, + }, + ) + except Exception as exc: + logger.debug("Copilot token seed failed: %s", exc) + + elif provider == "qwen-oauth": + # Qwen OAuth tokens live in ~/.qwen/oauth_creds.json, written by + # the Qwen CLI (`qwen auth qwen-oauth`). They aren't in the + # Hermes auth store or env vars, so resolve them here. + # Use refresh_if_expiring=False to avoid network calls during + # pool loading / provider discovery. + try: + from hermes_cli.auth import resolve_qwen_runtime_credentials + creds = resolve_qwen_runtime_credentials(refresh_if_expiring=False) + token = creds.get("api_key", "") + if token: + source_name = creds.get("source", "qwen-cli") + if not _is_suppressed(provider, source_name): + active_sources.add(source_name) + changed |= _upsert_entry( + entries, + provider, + source_name, + { + "source": source_name, + "auth_type": AUTH_TYPE_OAUTH, + "access_token": token, + "expires_at_ms": creds.get("expires_at_ms"), + "base_url": creds.get("base_url", ""), + "label": creds.get("auth_file", source_name), + }, + ) + except Exception as exc: + logger.debug("Qwen OAuth token seed failed: %s", exc) + elif provider == "openai-codex": + # Respect user suppression — `hermes auth remove openai-codex` marks + # the device_code source as suppressed so it won't be re-seeded from + # the Hermes auth store. Without this gate the removal is instantly + # undone on the next load_pool() call. + if _is_suppressed(provider, "device_code"): + return changed, active_sources + state = _load_provider_state(auth_store, "openai-codex") tokens = state.get("tokens") if isinstance(state, dict) else None - # Fallback: import from Codex CLI (~/.codex/auth.json) if Hermes auth - # store has no tokens. This mirrors resolve_codex_runtime_credentials() - # so that load_pool() and list_authenticated_providers() detect tokens - # that only exist in the Codex CLI shared file. - if not (isinstance(tokens, dict) and tokens.get("access_token")): - try: - from hermes_cli.auth import _import_codex_cli_tokens, _save_codex_tokens - cli_tokens = _import_codex_cli_tokens() - if cli_tokens: - logger.info("Importing Codex CLI tokens into Hermes auth store.") - _save_codex_tokens(cli_tokens) - # Re-read state after import - auth_store = _load_auth_store() - state = _load_provider_state(auth_store, "openai-codex") - tokens = state.get("tokens") if isinstance(state, dict) else None - except Exception as exc: - logger.debug("Codex CLI token import failed: %s", exc) + # Hermes owns its own Codex auth state — we do NOT auto-import from + # ~/.codex/auth.json at pool-load time. OAuth refresh tokens are + # single-use, so sharing them with Codex CLI / VS Code causes + # refresh_token_reused race failures. Users who want to adopt + # existing Codex CLI credentials get a one-time, explicit prompt + # via `hermes auth openai-codex`. if isinstance(tokens, dict) and tokens.get("access_token"): active_sources.add("device_code") changed |= _upsert_entry( @@ -1196,10 +1157,22 @@ def _seed_from_singletons(provider: str, entries: List[PooledCredential]) -> Tup def _seed_from_env(provider: str, entries: List[PooledCredential]) -> Tuple[bool, Set[str]]: changed = False active_sources: Set[str] = set() + # Honour user suppression — `hermes auth remove ` for an + # env-seeded credential marks the env: source as suppressed so it + # won't be re-seeded from the user's shell environment or ~/.hermes/.env. + # Without this gate the removal is silently undone on the next + # load_pool() call whenever the var is still exported by the shell. + try: + from hermes_cli.auth import is_source_suppressed as _is_source_suppressed + except ImportError: + def _is_source_suppressed(_p, _s): # type: ignore[misc] + return False if provider == "openrouter": token = os.getenv("OPENROUTER_API_KEY", "").strip() if token: source = "env:OPENROUTER_API_KEY" + if _is_source_suppressed(provider, source): + return changed, active_sources active_sources.add(source) changed |= _upsert_entry( entries, @@ -1236,6 +1209,8 @@ def _seed_from_env(provider: str, entries: List[PooledCredential]) -> Tuple[bool if not token: continue source = f"env:{env_var}" + if _is_source_suppressed(provider, source): + continue active_sources.add(source) auth_type = AUTH_TYPE_OAUTH if provider == "anthropic" and not token.startswith("sk-ant-api") else AUTH_TYPE_API_KEY base_url = env_url or pconfig.inference_base_url @@ -1280,6 +1255,13 @@ def _seed_custom_pool(pool_key: str, entries: List[PooledCredential]) -> Tuple[b changed = False active_sources: Set[str] = set() + # Shared suppression gate — same pattern as _seed_from_env/_seed_from_singletons. + try: + from hermes_cli.auth import is_source_suppressed as _is_suppressed + except ImportError: + def _is_suppressed(_p, _s): # type: ignore[misc] + return False + # Seed from the custom_providers config entry's api_key field cp_config = _get_custom_provider_config(pool_key) if cp_config: @@ -1288,19 +1270,20 @@ def _seed_custom_pool(pool_key: str, entries: List[PooledCredential]) -> Tuple[b name = str(cp_config.get("name") or "").strip() if api_key: source = f"config:{name}" - active_sources.add(source) - changed |= _upsert_entry( - entries, - pool_key, - source, - { - "source": source, - "auth_type": AUTH_TYPE_API_KEY, - "access_token": api_key, - "base_url": base_url, - "label": name or source, - }, - ) + if not _is_suppressed(pool_key, source): + active_sources.add(source) + changed |= _upsert_entry( + entries, + pool_key, + source, + { + "source": source, + "auth_type": AUTH_TYPE_API_KEY, + "access_token": api_key, + "base_url": base_url, + "label": name or source, + }, + ) # Seed from model.api_key if model.provider=='custom' and model.base_url matches try: @@ -1320,19 +1303,20 @@ def _seed_custom_pool(pool_key: str, entries: List[PooledCredential]) -> Tuple[b matched_key = get_custom_provider_pool_key(model_base_url) if matched_key == pool_key: source = "model_config" - active_sources.add(source) - changed |= _upsert_entry( - entries, - pool_key, - source, - { - "source": source, - "auth_type": AUTH_TYPE_API_KEY, - "access_token": model_api_key, - "base_url": model_base_url, - "label": "model_config", - }, - ) + if not _is_suppressed(pool_key, source): + active_sources.add(source) + changed |= _upsert_entry( + entries, + pool_key, + source, + { + "source": source, + "auth_type": AUTH_TYPE_API_KEY, + "access_token": model_api_key, + "base_url": model_base_url, + "label": "model_config", + }, + ) except Exception: pass diff --git a/agent/credential_sources.py b/agent/credential_sources.py new file mode 100644 index 000000000000..8ad2fade0b3e --- /dev/null +++ b/agent/credential_sources.py @@ -0,0 +1,401 @@ +"""Unified removal contract for every credential source Hermes reads from. + +Hermes seeds its credential pool from many places: + + env: — os.environ / ~/.hermes/.env + claude_code — ~/.claude/.credentials.json + hermes_pkce — ~/.hermes/.anthropic_oauth.json + device_code — auth.json providers. (nous, openai-codex, ...) + qwen-cli — ~/.qwen/oauth_creds.json + gh_cli — gh auth token + config: — custom_providers config entry + model_config — model.api_key when model.provider == "custom" + manual — user ran `hermes auth add` + +Each source has its own reader inside ``agent.credential_pool._seed_from_*`` +(which keep their existing shape — we haven't restructured them). What we +unify here is **removal**: + + ``hermes auth remove `` must make the pool entry stay gone. + +Before this module, every source had an ad-hoc removal branch in +``auth_remove_command``, and several sources had no branch at all — so +``auth remove`` silently reverted on the next ``load_pool()`` call for +qwen-cli, nous device_code (partial), hermes_pkce, copilot gh_cli, and +custom-config sources. + +Now every source registers a ``RemovalStep`` that does exactly three things +in the same shape: + + 1. Clean up whatever externally-readable state the source reads from + (.env line, auth.json block, OAuth file, etc.) + 2. Suppress the ``(provider, source_id)`` in auth.json so the + corresponding ``_seed_from_*`` branch skips the upsert on re-load + 3. Return ``RemovalResult`` describing what was cleaned and any + diagnostic hints the user should see (shell-exported env vars, + external credential files we deliberately don't delete, etc.) + +Adding a new credential source is: + - wire up a reader branch in ``_seed_from_*`` (existing pattern) + - gate that reader behind ``is_source_suppressed(provider, source_id)`` + - register a ``RemovalStep`` here + +No more per-source if/elif chain in ``auth_remove_command``. +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass, field +from pathlib import Path +from typing import Callable, List, Optional + + +@dataclass +class RemovalResult: + """Outcome of removing a credential source. + + Attributes: + cleaned: Short strings describing external state that was actually + mutated (``"Cleared XAI_API_KEY from .env"``, + ``"Cleared openai-codex OAuth tokens from auth store"``). + Printed as plain lines to the user. + hints: Diagnostic lines ABOUT state the user may need to clean up + themselves or is deliberately left intact (shell-exported env + var, Claude Code credential file we don't delete, etc.). + Printed as plain lines to the user. Always non-destructive. + suppress: Whether to call ``suppress_credential_source`` after + cleanup so future ``load_pool`` calls skip this source. + Default True — almost every source needs this to stay sticky. + The only legitimate False is ``manual`` entries, which aren't + seeded from anywhere external. + """ + + cleaned: List[str] = field(default_factory=list) + hints: List[str] = field(default_factory=list) + suppress: bool = True + + +@dataclass +class RemovalStep: + """How to remove one specific credential source cleanly. + + Attributes: + provider: Provider pool key (``"xai"``, ``"anthropic"``, ``"nous"``, ...). + Special value ``"*"`` means "matches any provider" — used for + sources like ``manual`` that aren't provider-specific. + source_id: Source identifier as it appears in + ``PooledCredential.source``. May be a literal (``"claude_code"``) + or a prefix pattern matched via ``match_fn``. + match_fn: Optional predicate overriding literal ``source_id`` + matching. Gets the removed entry's source string. Used for + ``env:*`` (any env-seeded key), ``config:*`` (any custom + pool), and ``manual:*`` (any manual-source variant). + remove_fn: ``(provider, removed_entry) -> RemovalResult``. Does the + actual cleanup and returns what happened for the user. + description: One-line human-readable description for docs / tests. + """ + + provider: str + source_id: str + remove_fn: Callable[..., RemovalResult] + match_fn: Optional[Callable[[str], bool]] = None + description: str = "" + + def matches(self, provider: str, source: str) -> bool: + if self.provider != "*" and self.provider != provider: + return False + if self.match_fn is not None: + return self.match_fn(source) + return source == self.source_id + + +_REGISTRY: List[RemovalStep] = [] + + +def register(step: RemovalStep) -> RemovalStep: + _REGISTRY.append(step) + return step + + +def find_removal_step(provider: str, source: str) -> Optional[RemovalStep]: + """Return the first matching RemovalStep, or None if unregistered. + + Unregistered sources fall through to the default remove path in + ``auth_remove_command``: the pool entry is already gone (that happens + before dispatch), no external cleanup, no suppression. This is the + correct behaviour for ``manual`` entries — they were only ever stored + in the pool, nothing external to clean up. + """ + for step in _REGISTRY: + if step.matches(provider, source): + return step + return None + + +# --------------------------------------------------------------------------- +# Individual RemovalStep implementations — one per source. +# --------------------------------------------------------------------------- +# Each remove_fn is intentionally small and single-purpose. Adding a new +# credential source means adding ONE entry here — no other changes to +# auth_remove_command. + + +def _remove_env_source(provider: str, removed) -> RemovalResult: + """env: — the most common case. + + Handles three user situations: + 1. Var lives only in ~/.hermes/.env → clear it + 2. Var lives only in the user's shell (shell profile, systemd + EnvironmentFile, launchd plist) → hint them where to unset it + 3. Var lives in both → clear from .env, hint about shell + """ + from hermes_cli.config import get_env_path, remove_env_value + + result = RemovalResult() + env_var = removed.source[len("env:"):] + if not env_var: + return result + + # Detect shell vs .env BEFORE remove_env_value pops os.environ. + env_in_process = bool(os.getenv(env_var)) + env_in_dotenv = False + try: + env_path = get_env_path() + if env_path.exists(): + env_in_dotenv = any( + line.strip().startswith(f"{env_var}=") + for line in env_path.read_text(errors="replace").splitlines() + ) + except OSError: + pass + shell_exported = env_in_process and not env_in_dotenv + + cleared = remove_env_value(env_var) + if cleared: + result.cleaned.append(f"Cleared {env_var} from .env") + + if shell_exported: + result.hints.extend([ + f"Note: {env_var} is still set in your shell environment " + f"(not in ~/.hermes/.env).", + " Unset it there (shell profile, systemd EnvironmentFile, " + "launchd plist, etc.) or it will keep being visible to Hermes.", + f" The pool entry is now suppressed — Hermes will ignore " + f"{env_var} until you run `hermes auth add {provider}`.", + ]) + else: + result.hints.append( + f"Suppressed env:{env_var} — it will not be re-seeded even " + f"if the variable is re-exported later." + ) + return result + + +def _remove_claude_code(provider: str, removed) -> RemovalResult: + """~/.claude/.credentials.json is owned by Claude Code itself. + + We don't delete it — the user's Claude Code install still needs to + work. We just suppress it so Hermes stops reading it. + """ + return RemovalResult(hints=[ + "Suppressed claude_code credential — it will not be re-seeded.", + "Note: Claude Code credentials still live in ~/.claude/.credentials.json", + "Run `hermes auth add anthropic` to re-enable if needed.", + ]) + + +def _remove_hermes_pkce(provider: str, removed) -> RemovalResult: + """~/.hermes/.anthropic_oauth.json is ours — delete it outright.""" + from hermes_constants import get_hermes_home + + result = RemovalResult() + oauth_file = get_hermes_home() / ".anthropic_oauth.json" + if oauth_file.exists(): + try: + oauth_file.unlink() + result.cleaned.append("Cleared Hermes Anthropic OAuth credentials") + except OSError as exc: + result.hints.append(f"Could not delete {oauth_file}: {exc}") + return result + + +def _clear_auth_store_provider(provider: str) -> bool: + """Delete auth_store.providers[provider]. Returns True if deleted.""" + from hermes_cli.auth import ( + _auth_store_lock, + _load_auth_store, + _save_auth_store, + ) + + with _auth_store_lock(): + auth_store = _load_auth_store() + providers_dict = auth_store.get("providers") + if isinstance(providers_dict, dict) and provider in providers_dict: + del providers_dict[provider] + _save_auth_store(auth_store) + return True + return False + + +def _remove_nous_device_code(provider: str, removed) -> RemovalResult: + """Nous OAuth lives in auth.json providers.nous — clear it and suppress. + + We suppress in addition to clearing because nothing else stops the + user's next `hermes login` run from writing providers.nous again + before they decide to. Suppression forces them to go through + `hermes auth add nous` to re-engage, which is the documented re-add + path and clears the suppression atomically. + """ + result = RemovalResult() + if _clear_auth_store_provider(provider): + result.cleaned.append(f"Cleared {provider} OAuth tokens from auth store") + return result + + +def _remove_codex_device_code(provider: str, removed) -> RemovalResult: + """Codex tokens live in TWO places: our auth store AND ~/.codex/auth.json. + + refresh_codex_oauth_pure() writes both every time, so clearing only + the Hermes auth store is not enough — _seed_from_singletons() would + re-import from ~/.codex/auth.json on the next load_pool() call and + the removal would be instantly undone. We suppress instead of + deleting Codex CLI's file, so the Codex CLI itself keeps working. + + The canonical source name in ``_seed_from_singletons`` is + ``"device_code"`` (no prefix). Entries may show up in the pool as + either ``"device_code"`` (seeded) or ``"manual:device_code"`` (added + via ``hermes auth add openai-codex``), but in both cases the re-seed + gate lives at the ``"device_code"`` suppression key. We suppress + that canonical key here; the central dispatcher also suppresses + ``removed.source`` which is fine — belt-and-suspenders, idempotent. + """ + from hermes_cli.auth import suppress_credential_source + + result = RemovalResult() + if _clear_auth_store_provider(provider): + result.cleaned.append(f"Cleared {provider} OAuth tokens from auth store") + # Suppress the canonical re-seed source, not just whatever source the + # removed entry had. Otherwise `manual:device_code` removals wouldn't + # block the `device_code` re-seed path. + suppress_credential_source(provider, "device_code") + result.hints.extend([ + "Suppressed openai-codex device_code source — it will not be re-seeded.", + "Note: Codex CLI credentials still live in ~/.codex/auth.json", + "Run `hermes auth add openai-codex` to re-enable if needed.", + ]) + return result + + +def _remove_qwen_cli(provider: str, removed) -> RemovalResult: + """~/.qwen/oauth_creds.json is owned by the Qwen CLI. + + Same pattern as claude_code — suppress, don't delete. The user's + Qwen CLI install still reads from that file. + """ + return RemovalResult(hints=[ + "Suppressed qwen-cli credential — it will not be re-seeded.", + "Note: Qwen CLI credentials still live in ~/.qwen/oauth_creds.json", + "Run `hermes auth add qwen-oauth` to re-enable if needed.", + ]) + + +def _remove_copilot_gh(provider: str, removed) -> RemovalResult: + """Copilot token comes from `gh auth token` or COPILOT_GITHUB_TOKEN / GH_TOKEN / GITHUB_TOKEN. + + Copilot is special: the same token can be seeded as multiple source + entries (gh_cli from ``_seed_from_singletons`` plus env: from + ``_seed_from_env``), so removing one entry without suppressing the + others lets the duplicates resurrect. We suppress ALL known copilot + sources here so removal is stable regardless of which entry the + user clicked. + + We don't touch the user's gh CLI or shell state — just suppress so + Hermes stops picking the token up. + """ + # Suppress ALL copilot source variants up-front so no path resurrects + # the pool entry. The central dispatcher in auth_remove_command will + # ALSO suppress removed.source, but it's idempotent so double-calling + # is harmless. + from hermes_cli.auth import suppress_credential_source + suppress_credential_source(provider, "gh_cli") + for env_var in ("COPILOT_GITHUB_TOKEN", "GH_TOKEN", "GITHUB_TOKEN"): + suppress_credential_source(provider, f"env:{env_var}") + + return RemovalResult(hints=[ + "Suppressed all copilot token sources (gh_cli + env vars) — they will not be re-seeded.", + "Note: Your gh CLI / shell environment is unchanged.", + "Run `hermes auth add copilot` to re-enable if needed.", + ]) + + +def _remove_custom_config(provider: str, removed) -> RemovalResult: + """Custom provider pools are seeded from custom_providers config or + model.api_key. Both are in config.yaml — modifying that from here + is more invasive than suppression. We suppress; the user can edit + config.yaml if they want to remove the key from disk entirely. + """ + source_label = removed.source + return RemovalResult(hints=[ + f"Suppressed {source_label} — it will not be re-seeded.", + "Note: The underlying value in config.yaml is unchanged. Edit it " + "directly if you want to remove the credential from disk.", + ]) + + +def _register_all_sources() -> None: + """Called once on module import. + + ORDER MATTERS — ``find_removal_step`` returns the first match. Put + provider-specific steps before the generic ``env:*`` step so that e.g. + copilot's ``env:GH_TOKEN`` goes through the copilot removal (which + doesn't touch the user's shell), not the generic env-var removal + (which would try to clear .env). + """ + register(RemovalStep( + provider="copilot", source_id="gh_cli", + match_fn=lambda src: src == "gh_cli" or src.startswith("env:"), + remove_fn=_remove_copilot_gh, + description="gh auth token / COPILOT_GITHUB_TOKEN / GH_TOKEN", + )) + register(RemovalStep( + provider="*", source_id="env:", + match_fn=lambda src: src.startswith("env:"), + remove_fn=_remove_env_source, + description="Any env-seeded credential (XAI_API_KEY, DEEPSEEK_API_KEY, etc.)", + )) + register(RemovalStep( + provider="anthropic", source_id="claude_code", + remove_fn=_remove_claude_code, + description="~/.claude/.credentials.json", + )) + register(RemovalStep( + provider="anthropic", source_id="hermes_pkce", + remove_fn=_remove_hermes_pkce, + description="~/.hermes/.anthropic_oauth.json", + )) + register(RemovalStep( + provider="nous", source_id="device_code", + remove_fn=_remove_nous_device_code, + description="auth.json providers.nous", + )) + register(RemovalStep( + provider="openai-codex", source_id="device_code", + match_fn=lambda src: src == "device_code" or src.endswith(":device_code"), + remove_fn=_remove_codex_device_code, + description="auth.json providers.openai-codex + ~/.codex/auth.json", + )) + register(RemovalStep( + provider="qwen-oauth", source_id="qwen-cli", + remove_fn=_remove_qwen_cli, + description="~/.qwen/oauth_creds.json", + )) + register(RemovalStep( + provider="*", source_id="config:", + match_fn=lambda src: src.startswith("config:") or src == "model_config", + remove_fn=_remove_custom_config, + description="Custom provider config.yaml api_key field", + )) + + +_register_all_sources() diff --git a/agent/display.py b/agent/display.py index 182064576862..474595d76c06 100644 --- a/agent/display.py +++ b/agent/display.py @@ -77,12 +77,6 @@ def _hex_fg(key: str, fallback_rgb: tuple[int, int, int]) -> str: return _diff_colors_cached -def reset_diff_colors() -> None: - """Reset cached diff colors (call after /skin switch).""" - global _diff_colors_cached - _diff_colors_cached = None - - # Module-level helpers — each call resolves from the active skin lazily. def _diff_dim(): return _diff_ansi()["dim"] def _diff_file(): return _diff_ansi()["file"] @@ -231,9 +225,11 @@ def build_tool_preview(tool_name: str, args: dict, max_len: int | None = None) - content = _oneline(args.get("content", "")) return f"+{target}: \"{content[:25]}{'...' if len(content) > 25 else ''}\"" elif action == "replace": - return f"~{target}: \"{_oneline(args.get('old_text', '')[:20])}\"" + old = _oneline(args.get("old_text") or "") or "" + return f"~{target}: \"{old[:20]}\"" elif action == "remove": - return f"-{target}: \"{_oneline(args.get('old_text', '')[:20])}\"" + old = _oneline(args.get("old_text") or "") or "" + return f"-{target}: \"{old[:20]}\"" return action if tool_name == "send_message": @@ -606,6 +602,45 @@ class KawaiiSpinner: "analyzing", "computing", "synthesizing", "formulating", "brainstorming", ] + @classmethod + def get_waiting_faces(cls) -> list: + """Return waiting faces from the active skin, falling back to KAWAII_WAITING.""" + try: + skin = _get_skin() + if skin: + faces = skin.spinner.get("waiting_faces", []) + if faces: + return faces + except Exception: + pass + return cls.KAWAII_WAITING + + @classmethod + def get_thinking_faces(cls) -> list: + """Return thinking faces from the active skin, falling back to KAWAII_THINKING.""" + try: + skin = _get_skin() + if skin: + faces = skin.spinner.get("thinking_faces", []) + if faces: + return faces + except Exception: + pass + return cls.KAWAII_THINKING + + @classmethod + def get_thinking_verbs(cls) -> list: + """Return thinking verbs from the active skin, falling back to THINKING_VERBS.""" + try: + skin = _get_skin() + if skin: + verbs = skin.spinner.get("thinking_verbs", []) + if verbs: + return verbs + except Exception: + pass + return cls.THINKING_VERBS + def __init__(self, message: str = "", spinner_type: str = 'dots', print_fn=None): self.message = message self.spinner_frames = self.SPINNERS.get(spinner_type, self.SPINNERS['dots']) @@ -906,9 +941,13 @@ def _wrap(line: str) -> str: if action == "add": return _wrap(f"┊ 🧠 memory +{target}: \"{_trunc(args.get('content', ''), 30)}\" {dur}") elif action == "replace": - return _wrap(f"┊ 🧠 memory ~{target}: \"{_trunc(args.get('old_text', ''), 20)}\" {dur}") + old = args.get("old_text") or "" + old = old if old else "" + return _wrap(f"┊ 🧠 memory ~{target}: \"{_trunc(old, 20)}\" {dur}") elif action == "remove": - return _wrap(f"┊ 🧠 memory -{target}: \"{_trunc(args.get('old_text', ''), 20)}\" {dur}") + old = args.get("old_text") or "" + old = old if old else "" + return _wrap(f"┊ 🧠 memory -{target}: \"{_trunc(old, 20)}\" {dur}") return _wrap(f"┊ 🧠 memory {action} {dur}") if tool_name == "skills_list": return _wrap(f"┊ 📚 skills list {args.get('category', 'all')} {dur}") @@ -960,84 +999,4 @@ def _wrap(line: str) -> str: # Honcho session line (one-liner with clickable OSC 8 hyperlink) # ========================================================================= -_DIM = "\033[2m" -_SKY_BLUE = "\033[38;5;117m" -_ANSI_RESET = "\033[0m" - - -# ========================================================================= -# Context pressure display (CLI user-facing warnings) -# ========================================================================= - -# ANSI color codes for context pressure tiers -_CYAN = "\033[36m" -_YELLOW = "\033[33m" -_BOLD = "\033[1m" -_DIM_ANSI = "\033[2m" - -# Bar characters -_BAR_FILLED = "▰" -_BAR_EMPTY = "▱" -_BAR_WIDTH = 20 - - -def format_context_pressure( - compaction_progress: float, - threshold_tokens: int, - threshold_percent: float, - compression_enabled: bool = True, -) -> str: - """Build a formatted context pressure line for CLI display. - - The bar and percentage show progress toward the compaction threshold, - NOT the raw context window. 100% = compaction fires. - - Args: - compaction_progress: How close to compaction (0.0–1.0, 1.0 = fires). - threshold_tokens: Compaction threshold in tokens. - threshold_percent: Compaction threshold as a fraction of context window. - compression_enabled: Whether auto-compression is active. - """ - pct_int = min(int(compaction_progress * 100), 100) - filled = min(int(compaction_progress * _BAR_WIDTH), _BAR_WIDTH) - bar = _BAR_FILLED * filled + _BAR_EMPTY * (_BAR_WIDTH - filled) - - threshold_k = f"{threshold_tokens // 1000}k" if threshold_tokens >= 1000 else str(threshold_tokens) - threshold_pct_int = int(threshold_percent * 100) - - color = f"{_BOLD}{_YELLOW}" - icon = "⚠" - if compression_enabled: - hint = "compaction approaching" - else: - hint = "no auto-compaction" - - return ( - f" {color}{icon} context {bar} {pct_int}% to compaction{_ANSI_RESET}" - f" {_DIM_ANSI}{threshold_k} threshold ({threshold_pct_int}%) · {hint}{_ANSI_RESET}" - ) - - -def format_context_pressure_gateway( - compaction_progress: float, - threshold_percent: float, - compression_enabled: bool = True, -) -> str: - """Build a plain-text context pressure notification for messaging platforms. - - No ANSI — just Unicode and plain text suitable for Telegram/Discord/etc. - The percentage shows progress toward the compaction threshold. - """ - pct_int = min(int(compaction_progress * 100), 100) - filled = min(int(compaction_progress * _BAR_WIDTH), _BAR_WIDTH) - bar = _BAR_FILLED * filled + _BAR_EMPTY * (_BAR_WIDTH - filled) - - threshold_pct_int = int(threshold_percent * 100) - - icon = "⚠️" - if compression_enabled: - hint = f"Context compaction approaching (threshold: {threshold_pct_int}% of window)." - else: - hint = "Auto-compaction is disabled — context may be truncated." - return f"{icon} Context: {bar} {pct_int}% to compaction\n{hint}" diff --git a/agent/error_classifier.py b/agent/error_classifier.py index dc5ae6b56f53..04875b6a5458 100644 --- a/agent/error_classifier.py +++ b/agent/error_classifier.py @@ -13,7 +13,6 @@ import enum import logging -import re from dataclasses import dataclass, field from typing import Any, Dict, Optional @@ -113,6 +112,10 @@ def is_auth(self) -> bool: "please retry after", "resource_exhausted", "rate increased too quickly", # Alibaba/DashScope throttling + # AWS Bedrock throttling + "throttlingexception", + "too many concurrent requests", + "servicequotaexceededexception", ] # Usage-limit patterns that need disambiguation (could be billing OR rate_limit) @@ -157,9 +160,26 @@ def is_auth(self) -> bool: "prompt exceeds max length", "max_tokens", "maximum number of tokens", + # vLLM / local inference server patterns + "exceeds the max_model_len", + "max_model_len", + "prompt length", # "engine prompt length X exceeds" + "input is too long", + "maximum model length", + # Ollama patterns + "context length exceeded", + "truncating input", + # llama.cpp / llama-server patterns + "slot context", # "slot context: N tokens, prompt N tokens" + "n_ctx_slot", # Chinese error messages (some providers return these) "超过最大长度", "上下文长度", + # AWS Bedrock Converse API error patterns + "input is too long", + "max input token", + "input token", + "exceeds the maximum number of input tokens", ] # Model not found patterns @@ -200,12 +220,25 @@ def is_auth(self) -> bool: "ConnectionAbortedError", "BrokenPipeError", "TimeoutError", "ReadError", "ServerDisconnectedError", + # SSL/TLS transport errors — transient mid-stream handshake/record + # failures that should retry rather than surface as a stalled session. + # ssl.SSLError subclasses OSError (caught by isinstance) but we list + # the type names here so provider-wrapped SSL errors (e.g. when the + # SDK re-raises without preserving the exception chain) still classify + # as transport rather than falling through to the unknown bucket. + "SSLError", "SSLZeroReturnError", "SSLWantReadError", + "SSLWantWriteError", "SSLEOFError", "SSLSyscallError", # OpenAI SDK errors (not subclasses of Python builtins) "APIConnectionError", "APITimeoutError", }) -# Server disconnect patterns (no status code, but transport-level) +# Server disconnect patterns (no status code, but transport-level). +# These are the "ambiguous" patterns — a plain connection close could be +# transient transport hiccup OR server-side context overflow rejection +# (common when the API gateway disconnects instead of returning an HTTP +# error for oversized requests). A large session + one of these patterns +# triggers the context-overflow-with-compression recovery path. _SERVER_DISCONNECT_PATTERNS = [ "server disconnected", "peer closed connection", @@ -216,6 +249,40 @@ def is_auth(self) -> bool: "incomplete chunked read", ] +# SSL/TLS transient failure patterns — intentionally distinct from +# _SERVER_DISCONNECT_PATTERNS above. +# +# An SSL alert mid-stream is almost always a transport-layer hiccup +# (flaky network, mid-session TLS renegotiation failure, load balancer +# dropping the connection) — NOT a server-side context overflow signal. +# So we want the retry path but NOT the compression path; lumping these +# into _SERVER_DISCONNECT_PATTERNS would trigger unnecessary (and +# expensive) context compression on any large-session SSL hiccup. +# +# The OpenSSL library constructs error codes by prepending a format string +# to the uppercased alert reason; OpenSSL 3.x changed the separator +# (e.g. `SSLV3_ALERT_BAD_RECORD_MAC` → `SSL/TLS_ALERT_BAD_RECORD_MAC`), +# which silently stopped matching anything explicit. Matching on the +# stable substrings (`bad record mac`, `ssl alert`, `tls alert`, etc.) +# survives future OpenSSL format churn without code changes. +_SSL_TRANSIENT_PATTERNS = [ + # Space-separated (human-readable form, Python ssl module, most SDKs) + "bad record mac", + "ssl alert", + "tls alert", + "ssl handshake failure", + "tlsv1 alert", + "sslv3 alert", + # Underscore-separated (OpenSSL error code tokens, e.g. + # `ERR_SSL_SSL/TLS_ALERT_BAD_RECORD_MAC`, `SSLV3_ALERT_BAD_RECORD_MAC`) + "bad_record_mac", + "ssl_alert", + "tls_alert", + "tls_alert_internal_error", + # Python ssl module prefix, e.g. "[SSL: BAD_RECORD_MAC]" + "[ssl:", +] + # ── Classification pipeline ───────────────────────────────────────────── @@ -235,9 +302,10 @@ def classify_api_error( 2. HTTP status code + message-aware refinement 3. Error code classification (from body) 4. Message pattern matching (billing vs rate_limit vs context vs auth) - 5. Transport error heuristics + 5. SSL/TLS transient alert patterns → retry as timeout 6. Server disconnect + large session → context overflow - 7. Fallback: unknown (retryable with backoff) + 7. Transport error heuristics + 8. Fallback: unknown (retryable with backoff) Args: error: The exception from the API call. @@ -270,7 +338,7 @@ def classify_api_error( if isinstance(body, dict): _err_obj = body.get("error", {}) if isinstance(_err_obj, dict): - _body_msg = (_err_obj.get("message") or "").lower() + _body_msg = str(_err_obj.get("message") or "").lower() # Parse metadata.raw for wrapped provider errors _metadata = _err_obj.get("metadata", {}) if isinstance(_metadata, dict): @@ -282,11 +350,11 @@ def classify_api_error( if isinstance(_inner, dict): _inner_err = _inner.get("error", {}) if isinstance(_inner_err, dict): - _metadata_msg = (_inner_err.get("message") or "").lower() + _metadata_msg = str(_inner_err.get("message") or "").lower() except (json.JSONDecodeError, TypeError): pass if not _body_msg: - _body_msg = (body.get("message") or "").lower() + _body_msg = str(body.get("message") or "").lower() # Combine all message sources for pattern matching parts = [_raw_msg] if _body_msg and _body_msg not in _raw_msg: @@ -368,7 +436,18 @@ def _result(reason: FailoverReason, **overrides) -> ClassifiedError: if classified is not None: return classified - # ── 5. Server disconnect + large session → context overflow ───── + # ── 5. SSL/TLS transient errors → retry as timeout (not compression) ── + # SSL alerts mid-stream are transport hiccups, not server-side context + # overflow signals. Classify before the disconnect check so a large + # session doesn't incorrectly trigger context compression when the real + # cause is a flaky TLS handshake. Also matches when the error is + # wrapped in a generic exception whose message string carries the SSL + # alert text but the type isn't ssl.SSLError (happens with some SDKs + # that re-raise without chaining). + if any(p in error_msg for p in _SSL_TRANSIENT_PATTERNS): + return _result(FailoverReason.timeout, retryable=True) + + # ── 6. Server disconnect + large session → context overflow ───── # Must come BEFORE generic transport error catch — a disconnect on # a large session is more likely context overflow than a transient # transport hiccup. Without this ordering, RemoteProtocolError @@ -385,12 +464,12 @@ def _result(reason: FailoverReason, **overrides) -> ClassifiedError: ) return _result(FailoverReason.timeout, retryable=True) - # ── 6. Transport / timeout heuristics ─────────────────────────── + # ── 7. Transport / timeout heuristics ─────────────────────────── if error_type in _TRANSPORT_ERROR_TYPES or isinstance(error, (TimeoutError, ConnectionError, OSError)): return _result(FailoverReason.timeout, retryable=True) - # ── 7. Fallback: unknown ──────────────────────────────────────── + # ── 8. Fallback: unknown ──────────────────────────────────────── return _result(FailoverReason.unknown, retryable=True) @@ -450,11 +529,16 @@ def _classify_by_status( retryable=False, should_fallback=True, ) - # Generic 404 — could be model or endpoint + # Generic 404 with no "model not found" signal — could be a wrong + # endpoint path (common with local llama.cpp / Ollama / vLLM when + # the URL is slightly misconfigured), a proxy routing glitch, or + # a transient backend issue. Classifying these as model_not_found + # silently falls back to a different provider and tells the model + # the model is missing, which is wrong and wastes a turn. Treat + # as unknown so the retry loop surfaces the real error instead. return result_fn( - FailoverReason.model_not_found, - retryable=False, - should_fallback=True, + FailoverReason.unknown, + retryable=True, ) if status_code == 413: @@ -586,10 +670,10 @@ def _classify_400( if isinstance(body, dict): err_obj = body.get("error", {}) if isinstance(err_obj, dict): - err_body_msg = (err_obj.get("message") or "").strip().lower() + err_body_msg = str(err_obj.get("message") or "").strip().lower() # Responses API (and some providers) use flat body: {"message": "..."} if not err_body_msg: - err_body_msg = (body.get("message") or "").strip().lower() + err_body_msg = str(body.get("message") or "").strip().lower() is_generic = len(err_body_msg) < 30 or err_body_msg in ("error", "") is_large = approx_tokens > context_length * 0.4 or approx_tokens > 80000 or num_messages > 80 diff --git a/agent/file_safety.py b/agent/file_safety.py new file mode 100644 index 000000000000..09da46cafdf8 --- /dev/null +++ b/agent/file_safety.py @@ -0,0 +1,111 @@ +"""Shared file safety rules used by both tools and ACP shims.""" + +from __future__ import annotations + +import os +from pathlib import Path +from typing import Optional + + +def _hermes_home_path() -> Path: + """Resolve the active HERMES_HOME (profile-aware) without circular imports.""" + try: + from hermes_constants import get_hermes_home # local import to avoid cycles + return get_hermes_home() + except Exception: + return Path(os.path.expanduser("~/.hermes")) + + +def build_write_denied_paths(home: str) -> set[str]: + """Return exact sensitive paths that must never be written.""" + hermes_home = _hermes_home_path() + return { + os.path.realpath(p) + for p in [ + os.path.join(home, ".ssh", "authorized_keys"), + os.path.join(home, ".ssh", "id_rsa"), + os.path.join(home, ".ssh", "id_ed25519"), + os.path.join(home, ".ssh", "config"), + str(hermes_home / ".env"), + os.path.join(home, ".bashrc"), + os.path.join(home, ".zshrc"), + os.path.join(home, ".profile"), + os.path.join(home, ".bash_profile"), + os.path.join(home, ".zprofile"), + os.path.join(home, ".netrc"), + os.path.join(home, ".pgpass"), + os.path.join(home, ".npmrc"), + os.path.join(home, ".pypirc"), + "/etc/sudoers", + "/etc/passwd", + "/etc/shadow", + ] + } + + +def build_write_denied_prefixes(home: str) -> list[str]: + """Return sensitive directory prefixes that must never be written.""" + return [ + os.path.realpath(p) + os.sep + for p in [ + os.path.join(home, ".ssh"), + os.path.join(home, ".aws"), + os.path.join(home, ".gnupg"), + os.path.join(home, ".kube"), + "/etc/sudoers.d", + "/etc/systemd", + os.path.join(home, ".docker"), + os.path.join(home, ".azure"), + os.path.join(home, ".config", "gh"), + ] + ] + + +def get_safe_write_root() -> Optional[str]: + """Return the resolved HERMES_WRITE_SAFE_ROOT path, or None if unset.""" + root = os.getenv("HERMES_WRITE_SAFE_ROOT", "") + if not root: + return None + try: + return os.path.realpath(os.path.expanduser(root)) + except Exception: + return None + + +def is_write_denied(path: str) -> bool: + """Return True if path is blocked by the write denylist or safe root.""" + home = os.path.realpath(os.path.expanduser("~")) + resolved = os.path.realpath(os.path.expanduser(str(path))) + + if resolved in build_write_denied_paths(home): + return True + for prefix in build_write_denied_prefixes(home): + if resolved.startswith(prefix): + return True + + safe_root = get_safe_write_root() + if safe_root and not (resolved == safe_root or resolved.startswith(safe_root + os.sep)): + return True + + return False + + +def get_read_block_error(path: str) -> Optional[str]: + """Return an error message when a read targets internal Hermes cache files.""" + resolved = Path(path).expanduser().resolve() + hermes_home = _hermes_home_path().resolve() + blocked_dirs = [ + hermes_home / "skills" / ".hub" / "index-cache", + hermes_home / "skills" / ".hub", + ] + for blocked in blocked_dirs: + try: + resolved.relative_to(blocked) + except ValueError: + continue + return ( + f"Access denied: {path} is an internal Hermes cache file " + "and cannot be read directly to prevent prompt injection. " + "Use the skills_list or skill_view tools instead." + ) + return None diff --git a/agent/gemini_cloudcode_adapter.py b/agent/gemini_cloudcode_adapter.py new file mode 100644 index 000000000000..24866c3a5313 --- /dev/null +++ b/agent/gemini_cloudcode_adapter.py @@ -0,0 +1,905 @@ +"""OpenAI-compatible facade that talks to Google's Cloud Code Assist backend. + +This adapter lets Hermes use the ``google-gemini-cli`` provider as if it were +a standard OpenAI-shaped chat completion endpoint, while the underlying HTTP +traffic goes to ``cloudcode-pa.googleapis.com/v1internal:{generateContent, +streamGenerateContent}`` with a Bearer access token obtained via OAuth PKCE. + +Architecture +------------ +- ``GeminiCloudCodeClient`` exposes ``.chat.completions.create(**kwargs)`` + mirroring the subset of the OpenAI SDK that ``run_agent.py`` uses. +- Incoming OpenAI ``messages[]`` / ``tools[]`` / ``tool_choice`` are translated + to Gemini's native ``contents[]`` / ``tools[].functionDeclarations`` / + ``toolConfig`` / ``systemInstruction`` shape. +- The request body is wrapped ``{project, model, user_prompt_id, request}`` + per Code Assist API expectations. +- Responses (``candidates[].content.parts[]``) are converted back to + OpenAI ``choices[0].message`` shape with ``content`` + ``tool_calls``. +- Streaming uses SSE (``?alt=sse``) and yields OpenAI-shaped delta chunks. + +Attribution +----------- +Translation semantics follow jenslys/opencode-gemini-auth (MIT) and the public +Gemini API docs. Request envelope shape +(``{project, model, user_prompt_id, request}``) is documented nowhere; it is +reverse-engineered from the opencode-gemini-auth and clawdbot implementations. +""" + +from __future__ import annotations + +import json +import logging +import os +import time +import uuid +from types import SimpleNamespace +from typing import Any, Dict, Iterator, List, Optional + +import httpx + +from agent import google_oauth +from agent.gemini_schema import sanitize_gemini_tool_parameters +from agent.google_code_assist import ( + CODE_ASSIST_ENDPOINT, + FREE_TIER_ID, + CodeAssistError, + ProjectContext, + resolve_project_context, +) + +logger = logging.getLogger(__name__) + + +# ============================================================================= +# Request translation: OpenAI → Gemini +# ============================================================================= + +_ROLE_MAP_OPENAI_TO_GEMINI = { + "user": "user", + "assistant": "model", + "system": "user", # handled separately via systemInstruction + "tool": "user", # functionResponse is wrapped in a user-role turn + "function": "user", +} + + +def _coerce_content_to_text(content: Any) -> str: + """OpenAI content may be str or a list of parts; reduce to plain text.""" + if content is None: + return "" + if isinstance(content, str): + return content + if isinstance(content, list): + pieces: List[str] = [] + for p in content: + if isinstance(p, str): + pieces.append(p) + elif isinstance(p, dict): + if p.get("type") == "text" and isinstance(p.get("text"), str): + pieces.append(p["text"]) + # Multimodal (image_url, etc.) — stub for now; log and skip + elif p.get("type") in ("image_url", "input_audio"): + logger.debug("Dropping multimodal part (not yet supported): %s", p.get("type")) + return "\n".join(pieces) + return str(content) + + +def _translate_tool_call_to_gemini(tool_call: Dict[str, Any]) -> Dict[str, Any]: + """OpenAI tool_call -> Gemini functionCall part.""" + fn = tool_call.get("function") or {} + args_raw = fn.get("arguments", "") + try: + args = json.loads(args_raw) if isinstance(args_raw, str) and args_raw else {} + except json.JSONDecodeError: + args = {"_raw": args_raw} + if not isinstance(args, dict): + args = {"_value": args} + return { + "functionCall": { + "name": fn.get("name") or "", + "args": args, + }, + # Sentinel signature — matches opencode-gemini-auth's approach. + # Without this, Code Assist rejects function calls that originated + # outside its own chain. + "thoughtSignature": "skip_thought_signature_validator", + } + + +def _translate_tool_result_to_gemini(message: Dict[str, Any]) -> Dict[str, Any]: + """OpenAI tool-role message -> Gemini functionResponse part. + + The function name isn't in the OpenAI tool message directly; it must be + passed via the assistant message that issued the call. For simplicity we + look up ``name`` on the message (OpenAI SDK copies it there) or on the + ``tool_call_id`` cross-reference. + """ + name = str(message.get("name") or message.get("tool_call_id") or "tool") + content = _coerce_content_to_text(message.get("content")) + # Gemini expects the response as a dict under `response`. We wrap plain + # text in {"output": "..."}. + try: + parsed = json.loads(content) if content.strip().startswith(("{", "[")) else None + except json.JSONDecodeError: + parsed = None + response = parsed if isinstance(parsed, dict) else {"output": content} + return { + "functionResponse": { + "name": name, + "response": response, + }, + } + + +def _build_gemini_contents( + messages: List[Dict[str, Any]], +) -> tuple[List[Dict[str, Any]], Optional[Dict[str, Any]]]: + """Convert OpenAI messages[] to Gemini contents[] + systemInstruction.""" + system_text_parts: List[str] = [] + contents: List[Dict[str, Any]] = [] + + for msg in messages: + if not isinstance(msg, dict): + continue + role = str(msg.get("role") or "user") + + if role == "system": + system_text_parts.append(_coerce_content_to_text(msg.get("content"))) + continue + + # Tool result message — emit a user-role turn with functionResponse + if role == "tool" or role == "function": + contents.append({ + "role": "user", + "parts": [_translate_tool_result_to_gemini(msg)], + }) + continue + + gemini_role = _ROLE_MAP_OPENAI_TO_GEMINI.get(role, "user") + parts: List[Dict[str, Any]] = [] + + text = _coerce_content_to_text(msg.get("content")) + if text: + parts.append({"text": text}) + + # Assistant messages can carry tool_calls + tool_calls = msg.get("tool_calls") or [] + if isinstance(tool_calls, list): + for tc in tool_calls: + if isinstance(tc, dict): + parts.append(_translate_tool_call_to_gemini(tc)) + + if not parts: + # Gemini rejects empty parts; skip the turn entirely + continue + + contents.append({"role": gemini_role, "parts": parts}) + + system_instruction: Optional[Dict[str, Any]] = None + joined_system = "\n".join(p for p in system_text_parts if p).strip() + if joined_system: + system_instruction = { + "role": "system", + "parts": [{"text": joined_system}], + } + + return contents, system_instruction + + +def _translate_tools_to_gemini(tools: Any) -> List[Dict[str, Any]]: + """OpenAI tools[] -> Gemini tools[].functionDeclarations[].""" + if not isinstance(tools, list) or not tools: + return [] + declarations: List[Dict[str, Any]] = [] + for t in tools: + if not isinstance(t, dict): + continue + fn = t.get("function") or {} + if not isinstance(fn, dict): + continue + name = fn.get("name") + if not name: + continue + decl = {"name": str(name)} + if fn.get("description"): + decl["description"] = str(fn["description"]) + params = fn.get("parameters") + if isinstance(params, dict): + decl["parameters"] = sanitize_gemini_tool_parameters(params) + declarations.append(decl) + if not declarations: + return [] + return [{"functionDeclarations": declarations}] + + +def _translate_tool_choice_to_gemini(tool_choice: Any) -> Optional[Dict[str, Any]]: + """OpenAI tool_choice -> Gemini toolConfig.functionCallingConfig.""" + if tool_choice is None: + return None + if isinstance(tool_choice, str): + if tool_choice == "auto": + return {"functionCallingConfig": {"mode": "AUTO"}} + if tool_choice == "required": + return {"functionCallingConfig": {"mode": "ANY"}} + if tool_choice == "none": + return {"functionCallingConfig": {"mode": "NONE"}} + if isinstance(tool_choice, dict): + fn = tool_choice.get("function") or {} + name = fn.get("name") + if name: + return { + "functionCallingConfig": { + "mode": "ANY", + "allowedFunctionNames": [str(name)], + }, + } + return None + + +def _normalize_thinking_config(config: Any) -> Optional[Dict[str, Any]]: + """Accept thinkingBudget / thinkingLevel / includeThoughts (+ snake_case).""" + if not isinstance(config, dict) or not config: + return None + budget = config.get("thinkingBudget", config.get("thinking_budget")) + level = config.get("thinkingLevel", config.get("thinking_level")) + include = config.get("includeThoughts", config.get("include_thoughts")) + normalized: Dict[str, Any] = {} + if isinstance(budget, (int, float)): + normalized["thinkingBudget"] = int(budget) + if isinstance(level, str) and level.strip(): + normalized["thinkingLevel"] = level.strip().lower() + if isinstance(include, bool): + normalized["includeThoughts"] = include + return normalized or None + + +def build_gemini_request( + *, + messages: List[Dict[str, Any]], + tools: Any = None, + tool_choice: Any = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, + top_p: Optional[float] = None, + stop: Any = None, + thinking_config: Any = None, +) -> Dict[str, Any]: + """Build the inner Gemini request body (goes inside ``request`` wrapper).""" + contents, system_instruction = _build_gemini_contents(messages) + + body: Dict[str, Any] = {"contents": contents} + if system_instruction is not None: + body["systemInstruction"] = system_instruction + + gemini_tools = _translate_tools_to_gemini(tools) + if gemini_tools: + body["tools"] = gemini_tools + tool_cfg = _translate_tool_choice_to_gemini(tool_choice) + if tool_cfg is not None: + body["toolConfig"] = tool_cfg + + generation_config: Dict[str, Any] = {} + if isinstance(temperature, (int, float)): + generation_config["temperature"] = float(temperature) + if isinstance(max_tokens, int) and max_tokens > 0: + generation_config["maxOutputTokens"] = max_tokens + if isinstance(top_p, (int, float)): + generation_config["topP"] = float(top_p) + if isinstance(stop, str) and stop: + generation_config["stopSequences"] = [stop] + elif isinstance(stop, list) and stop: + generation_config["stopSequences"] = [str(s) for s in stop if s] + normalized_thinking = _normalize_thinking_config(thinking_config) + if normalized_thinking: + generation_config["thinkingConfig"] = normalized_thinking + if generation_config: + body["generationConfig"] = generation_config + + return body + + +def wrap_code_assist_request( + *, + project_id: str, + model: str, + inner_request: Dict[str, Any], + user_prompt_id: Optional[str] = None, +) -> Dict[str, Any]: + """Wrap the inner Gemini request in the Code Assist envelope.""" + return { + "project": project_id, + "model": model, + "user_prompt_id": user_prompt_id or str(uuid.uuid4()), + "request": inner_request, + } + + +# ============================================================================= +# Response translation: Gemini → OpenAI +# ============================================================================= + +def _translate_gemini_response( + resp: Dict[str, Any], + model: str, +) -> SimpleNamespace: + """Non-streaming Gemini response -> OpenAI-shaped SimpleNamespace. + + Code Assist wraps the actual Gemini response inside ``response``, so we + unwrap it first if present. + """ + inner = resp.get("response") if isinstance(resp.get("response"), dict) else resp + + candidates = inner.get("candidates") or [] + if not isinstance(candidates, list) or not candidates: + return _empty_response(model) + + cand = candidates[0] + content_obj = cand.get("content") if isinstance(cand, dict) else {} + parts = content_obj.get("parts") if isinstance(content_obj, dict) else [] + + text_pieces: List[str] = [] + reasoning_pieces: List[str] = [] + tool_calls: List[SimpleNamespace] = [] + + for i, part in enumerate(parts or []): + if not isinstance(part, dict): + continue + # Thought parts are model's internal reasoning — surface as reasoning, + # don't mix into content. + if part.get("thought") is True: + if isinstance(part.get("text"), str): + reasoning_pieces.append(part["text"]) + continue + if isinstance(part.get("text"), str): + text_pieces.append(part["text"]) + continue + fc = part.get("functionCall") + if isinstance(fc, dict) and fc.get("name"): + try: + args_str = json.dumps(fc.get("args") or {}, ensure_ascii=False) + except (TypeError, ValueError): + args_str = "{}" + tool_calls.append(SimpleNamespace( + id=f"call_{uuid.uuid4().hex[:12]}", + type="function", + index=i, + function=SimpleNamespace(name=str(fc["name"]), arguments=args_str), + )) + + finish_reason = "tool_calls" if tool_calls else _map_gemini_finish_reason( + str(cand.get("finishReason") or "") + ) + + usage_meta = inner.get("usageMetadata") or {} + usage = SimpleNamespace( + prompt_tokens=int(usage_meta.get("promptTokenCount") or 0), + completion_tokens=int(usage_meta.get("candidatesTokenCount") or 0), + total_tokens=int(usage_meta.get("totalTokenCount") or 0), + prompt_tokens_details=SimpleNamespace( + cached_tokens=int(usage_meta.get("cachedContentTokenCount") or 0), + ), + ) + + message = SimpleNamespace( + role="assistant", + content="".join(text_pieces) if text_pieces else None, + tool_calls=tool_calls or None, + reasoning="".join(reasoning_pieces) or None, + reasoning_content="".join(reasoning_pieces) or None, + reasoning_details=None, + ) + choice = SimpleNamespace( + index=0, + message=message, + finish_reason=finish_reason, + ) + return SimpleNamespace( + id=f"chatcmpl-{uuid.uuid4().hex[:12]}", + object="chat.completion", + created=int(time.time()), + model=model, + choices=[choice], + usage=usage, + ) + + +def _empty_response(model: str) -> SimpleNamespace: + message = SimpleNamespace( + role="assistant", content="", tool_calls=None, + reasoning=None, reasoning_content=None, reasoning_details=None, + ) + choice = SimpleNamespace(index=0, message=message, finish_reason="stop") + usage = SimpleNamespace( + prompt_tokens=0, completion_tokens=0, total_tokens=0, + prompt_tokens_details=SimpleNamespace(cached_tokens=0), + ) + return SimpleNamespace( + id=f"chatcmpl-{uuid.uuid4().hex[:12]}", + object="chat.completion", + created=int(time.time()), + model=model, + choices=[choice], + usage=usage, + ) + + +def _map_gemini_finish_reason(reason: str) -> str: + mapping = { + "STOP": "stop", + "MAX_TOKENS": "length", + "SAFETY": "content_filter", + "RECITATION": "content_filter", + "OTHER": "stop", + } + return mapping.get(reason.upper(), "stop") + + +# ============================================================================= +# Streaming SSE iterator +# ============================================================================= + +class _GeminiStreamChunk(SimpleNamespace): + """Mimics an OpenAI ChatCompletionChunk with .choices[0].delta.""" + pass + + +def _make_stream_chunk( + *, + model: str, + content: str = "", + tool_call_delta: Optional[Dict[str, Any]] = None, + finish_reason: Optional[str] = None, + reasoning: str = "", +) -> _GeminiStreamChunk: + delta_kwargs: Dict[str, Any] = {"role": "assistant"} + if content: + delta_kwargs["content"] = content + if tool_call_delta is not None: + delta_kwargs["tool_calls"] = [SimpleNamespace( + index=tool_call_delta.get("index", 0), + id=tool_call_delta.get("id") or f"call_{uuid.uuid4().hex[:12]}", + type="function", + function=SimpleNamespace( + name=tool_call_delta.get("name") or "", + arguments=tool_call_delta.get("arguments") or "", + ), + )] + if reasoning: + delta_kwargs["reasoning"] = reasoning + delta_kwargs["reasoning_content"] = reasoning + delta = SimpleNamespace(**delta_kwargs) + choice = SimpleNamespace(index=0, delta=delta, finish_reason=finish_reason) + return _GeminiStreamChunk( + id=f"chatcmpl-{uuid.uuid4().hex[:12]}", + object="chat.completion.chunk", + created=int(time.time()), + model=model, + choices=[choice], + usage=None, + ) + + +def _iter_sse_events(response: httpx.Response) -> Iterator[Dict[str, Any]]: + """Parse Server-Sent Events from an httpx streaming response.""" + buffer = "" + for chunk in response.iter_text(): + if not chunk: + continue + buffer += chunk + while "\n" in buffer: + line, buffer = buffer.split("\n", 1) + line = line.rstrip("\r") + if not line: + continue + if line.startswith("data: "): + data = line[6:] + if data == "[DONE]": + return + try: + yield json.loads(data) + except json.JSONDecodeError: + logger.debug("Non-JSON SSE line: %s", data[:200]) + + +def _translate_stream_event( + event: Dict[str, Any], + model: str, + tool_call_counter: List[int], +) -> List[_GeminiStreamChunk]: + """Unwrap Code Assist envelope and emit OpenAI-shaped chunk(s). + + ``tool_call_counter`` is a single-element list used as a mutable counter + across events in the same stream. Each ``functionCall`` part gets a + fresh, unique OpenAI ``index`` — keying by function name would collide + whenever the model issues parallel calls to the same tool (e.g. reading + three files in one turn). + """ + inner = event.get("response") if isinstance(event.get("response"), dict) else event + candidates = inner.get("candidates") or [] + if not candidates: + return [] + cand = candidates[0] + if not isinstance(cand, dict): + return [] + + chunks: List[_GeminiStreamChunk] = [] + + content = cand.get("content") or {} + parts = content.get("parts") if isinstance(content, dict) else [] + for part in parts or []: + if not isinstance(part, dict): + continue + if part.get("thought") is True and isinstance(part.get("text"), str): + chunks.append(_make_stream_chunk( + model=model, reasoning=part["text"], + )) + continue + if isinstance(part.get("text"), str) and part["text"]: + chunks.append(_make_stream_chunk(model=model, content=part["text"])) + fc = part.get("functionCall") + if isinstance(fc, dict) and fc.get("name"): + name = str(fc["name"]) + idx = tool_call_counter[0] + tool_call_counter[0] += 1 + try: + args_str = json.dumps(fc.get("args") or {}, ensure_ascii=False) + except (TypeError, ValueError): + args_str = "{}" + chunks.append(_make_stream_chunk( + model=model, + tool_call_delta={ + "index": idx, + "name": name, + "arguments": args_str, + }, + )) + + finish_reason_raw = str(cand.get("finishReason") or "") + if finish_reason_raw: + mapped = _map_gemini_finish_reason(finish_reason_raw) + if tool_call_counter[0] > 0: + mapped = "tool_calls" + chunks.append(_make_stream_chunk(model=model, finish_reason=mapped)) + return chunks + + +# ============================================================================= +# GeminiCloudCodeClient — OpenAI-compatible facade +# ============================================================================= + +MARKER_BASE_URL = "cloudcode-pa://google" + + +class _GeminiChatCompletions: + def __init__(self, client: "GeminiCloudCodeClient"): + self._client = client + + def create(self, **kwargs: Any) -> Any: + return self._client._create_chat_completion(**kwargs) + + +class _GeminiChatNamespace: + def __init__(self, client: "GeminiCloudCodeClient"): + self.completions = _GeminiChatCompletions(client) + + +class GeminiCloudCodeClient: + """Minimal OpenAI-SDK-compatible facade over Code Assist v1internal.""" + + def __init__( + self, + *, + api_key: Optional[str] = None, + base_url: Optional[str] = None, + default_headers: Optional[Dict[str, str]] = None, + project_id: str = "", + **_: Any, + ): + # `api_key` here is a dummy — real auth is the OAuth access token + # fetched on every call via agent.google_oauth.get_valid_access_token(). + # We accept the kwarg for openai.OpenAI interface parity. + self.api_key = api_key or "google-oauth" + self.base_url = base_url or MARKER_BASE_URL + self._default_headers = dict(default_headers or {}) + self._configured_project_id = project_id + self._project_context: Optional[ProjectContext] = None + self._project_context_lock = False # simple single-thread guard + self.chat = _GeminiChatNamespace(self) + self.is_closed = False + self._http = httpx.Client(timeout=httpx.Timeout(connect=15.0, read=600.0, write=30.0, pool=30.0)) + + def close(self) -> None: + self.is_closed = True + try: + self._http.close() + except Exception: + pass + + # Implement the OpenAI SDK's context-manager-ish closure check + def __enter__(self): + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + self.close() + + def _ensure_project_context(self, access_token: str, model: str) -> ProjectContext: + """Lazily resolve and cache the project context for this client.""" + if self._project_context is not None: + return self._project_context + + env_project = google_oauth.resolve_project_id_from_env() + creds = google_oauth.load_credentials() + stored_project = creds.project_id if creds else "" + + # Prefer what's already baked into the creds + if stored_project: + self._project_context = ProjectContext( + project_id=stored_project, + managed_project_id=creds.managed_project_id if creds else "", + tier_id="", + source="stored", + ) + return self._project_context + + ctx = resolve_project_context( + access_token, + configured_project_id=self._configured_project_id, + env_project_id=env_project, + user_agent_model=model, + ) + # Persist discovered project back to the creds file so the next + # session doesn't re-run the discovery. + if ctx.project_id or ctx.managed_project_id: + google_oauth.update_project_ids( + project_id=ctx.project_id, + managed_project_id=ctx.managed_project_id, + ) + self._project_context = ctx + return ctx + + def _create_chat_completion( + self, + *, + model: str = "gemini-2.5-flash", + messages: Optional[List[Dict[str, Any]]] = None, + stream: bool = False, + tools: Any = None, + tool_choice: Any = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, + top_p: Optional[float] = None, + stop: Any = None, + extra_body: Optional[Dict[str, Any]] = None, + timeout: Any = None, + **_: Any, + ) -> Any: + access_token = google_oauth.get_valid_access_token() + ctx = self._ensure_project_context(access_token, model) + + thinking_config = None + if isinstance(extra_body, dict): + thinking_config = extra_body.get("thinking_config") or extra_body.get("thinkingConfig") + + inner = build_gemini_request( + messages=messages or [], + tools=tools, + tool_choice=tool_choice, + temperature=temperature, + max_tokens=max_tokens, + top_p=top_p, + stop=stop, + thinking_config=thinking_config, + ) + wrapped = wrap_code_assist_request( + project_id=ctx.project_id, + model=model, + inner_request=inner, + ) + + headers = { + "Content-Type": "application/json", + "Accept": "application/json", + "Authorization": f"Bearer {access_token}", + "User-Agent": "hermes-agent (gemini-cli-compat)", + "X-Goog-Api-Client": "gl-python/hermes", + "x-activity-request-id": str(uuid.uuid4()), + } + headers.update(self._default_headers) + + if stream: + return self._stream_completion(model=model, wrapped=wrapped, headers=headers) + + url = f"{CODE_ASSIST_ENDPOINT}/v1internal:generateContent" + response = self._http.post(url, json=wrapped, headers=headers) + if response.status_code != 200: + raise _gemini_http_error(response) + try: + payload = response.json() + except ValueError as exc: + raise CodeAssistError( + f"Invalid JSON from Code Assist: {exc}", + code="code_assist_invalid_json", + ) from exc + return _translate_gemini_response(payload, model=model) + + def _stream_completion( + self, + *, + model: str, + wrapped: Dict[str, Any], + headers: Dict[str, str], + ) -> Iterator[_GeminiStreamChunk]: + """Generator that yields OpenAI-shaped streaming chunks.""" + url = f"{CODE_ASSIST_ENDPOINT}/v1internal:streamGenerateContent?alt=sse" + stream_headers = dict(headers) + stream_headers["Accept"] = "text/event-stream" + + def _generator() -> Iterator[_GeminiStreamChunk]: + try: + with self._http.stream("POST", url, json=wrapped, headers=stream_headers) as response: + if response.status_code != 200: + # Materialize error body for better diagnostics + response.read() + raise _gemini_http_error(response) + tool_call_counter: List[int] = [0] + for event in _iter_sse_events(response): + for chunk in _translate_stream_event(event, model, tool_call_counter): + yield chunk + except httpx.HTTPError as exc: + raise CodeAssistError( + f"Streaming request failed: {exc}", + code="code_assist_stream_error", + ) from exc + + return _generator() + + +def _gemini_http_error(response: httpx.Response) -> CodeAssistError: + """Translate an httpx response into a CodeAssistError with rich metadata. + + Parses Google's error envelope (``{"error": {"code", "message", "status", + "details": [...]}}``) so the agent's error classifier can reason about + the failure — ``status_code`` enables the rate_limit / auth classification + paths, and ``response`` lets the main loop honor ``Retry-After`` just + like it does for OpenAI SDK exceptions. + + Also lifts a few recognizable Google conditions into human-readable + messages so the user sees something better than a 500-char JSON dump: + + MODEL_CAPACITY_EXHAUSTED → "Gemini model capacity exhausted for + . This is a Google-side throttle..." + RESOURCE_EXHAUSTED w/o reason → quota-style message + 404 → "Model not found at cloudcode-pa..." + """ + status = response.status_code + + # Parse the body once, surviving any weird encodings. + body_text = "" + body_json: Dict[str, Any] = {} + try: + body_text = response.text + except Exception: + body_text = "" + if body_text: + try: + parsed = json.loads(body_text) + if isinstance(parsed, dict): + body_json = parsed + except (ValueError, TypeError): + body_json = {} + + # Dig into Google's error envelope. Shape is: + # {"error": {"code": 429, "message": "...", "status": "RESOURCE_EXHAUSTED", + # "details": [{"@type": ".../ErrorInfo", "reason": "MODEL_CAPACITY_EXHAUSTED", + # "metadata": {...}}, + # {"@type": ".../RetryInfo", "retryDelay": "30s"}]}} + err_obj = body_json.get("error") if isinstance(body_json, dict) else None + if not isinstance(err_obj, dict): + err_obj = {} + err_status = str(err_obj.get("status") or "").strip() + err_message = str(err_obj.get("message") or "").strip() + _raw_details = err_obj.get("details") + err_details_list = _raw_details if isinstance(_raw_details, list) else [] + + # Extract google.rpc.ErrorInfo reason + metadata. There may be more + # than one ErrorInfo (rare), so we pick the first one with a reason. + error_reason = "" + error_metadata: Dict[str, Any] = {} + retry_delay_seconds: Optional[float] = None + for detail in err_details_list: + if not isinstance(detail, dict): + continue + type_url = str(detail.get("@type") or "") + if not error_reason and type_url.endswith("/google.rpc.ErrorInfo"): + reason = detail.get("reason") + if isinstance(reason, str) and reason: + error_reason = reason + md = detail.get("metadata") + if isinstance(md, dict): + error_metadata = md + elif retry_delay_seconds is None and type_url.endswith("/google.rpc.RetryInfo"): + # retryDelay is a google.protobuf.Duration string like "30s" or "1.5s". + delay_raw = detail.get("retryDelay") + if isinstance(delay_raw, str) and delay_raw.endswith("s"): + try: + retry_delay_seconds = float(delay_raw[:-1]) + except ValueError: + pass + elif isinstance(delay_raw, (int, float)): + retry_delay_seconds = float(delay_raw) + + # Fall back to the Retry-After header if the body didn't include RetryInfo. + if retry_delay_seconds is None: + try: + header_val = response.headers.get("Retry-After") or response.headers.get("retry-after") + except Exception: + header_val = None + if header_val: + try: + retry_delay_seconds = float(header_val) + except (TypeError, ValueError): + retry_delay_seconds = None + + # Classify the error code. ``code_assist_rate_limited`` stays the default + # for 429s; a more specific reason tag helps downstream callers (e.g. tests, + # logs) without changing the rate_limit classification path. + code = f"code_assist_http_{status}" + if status == 401: + code = "code_assist_unauthorized" + elif status == 429: + code = "code_assist_rate_limited" + if error_reason == "MODEL_CAPACITY_EXHAUSTED": + code = "code_assist_capacity_exhausted" + + # Build a human-readable message. Keep the status + a raw-body tail for + # debugging, but lead with a friendlier summary when we recognize the + # Google signal. + model_hint = "" + if isinstance(error_metadata, dict): + model_hint = str(error_metadata.get("model") or error_metadata.get("modelId") or "").strip() + + if status == 429 and error_reason == "MODEL_CAPACITY_EXHAUSTED": + target = model_hint or "this Gemini model" + message = ( + f"Gemini capacity exhausted for {target} (Google-side throttle, " + f"not a Hermes issue). Try a different Gemini model or set a " + f"fallback_providers entry to a non-Gemini provider." + ) + if retry_delay_seconds is not None: + message += f" Google suggests retrying in {retry_delay_seconds:g}s." + elif status == 429 and err_status == "RESOURCE_EXHAUSTED": + message = ( + f"Gemini quota exhausted ({err_message or 'RESOURCE_EXHAUSTED'}). " + f"Check /gquota for remaining daily requests." + ) + if retry_delay_seconds is not None: + message += f" Retry suggested in {retry_delay_seconds:g}s." + elif status == 404: + # Google returns 404 when a model has been retired or renamed. + target = model_hint or (err_message or "model") + message = ( + f"Code Assist 404: {target} is not available at " + f"cloudcode-pa.googleapis.com. It may have been renamed or " + f"retired. Check hermes_cli/models.py for the current list." + ) + elif err_message: + # Generic fallback with the parsed message. + message = f"Code Assist HTTP {status} ({err_status or 'error'}): {err_message}" + else: + # Last-ditch fallback — raw body snippet. + message = f"Code Assist returned HTTP {status}: {body_text[:500]}" + + return CodeAssistError( + message, + code=code, + status_code=status, + response=response, + retry_after=retry_delay_seconds, + details={ + "status": err_status, + "reason": error_reason, + "metadata": error_metadata, + "message": err_message, + }, + ) diff --git a/agent/gemini_native_adapter.py b/agent/gemini_native_adapter.py new file mode 100644 index 000000000000..406e4a19b763 --- /dev/null +++ b/agent/gemini_native_adapter.py @@ -0,0 +1,847 @@ +"""OpenAI-compatible facade over Google AI Studio's native Gemini API. + +Hermes keeps ``api_mode='chat_completions'`` for the ``gemini`` provider so the +main agent loop can keep using its existing OpenAI-shaped message flow. +This adapter is the transport shim that converts those OpenAI-style +``messages[]`` / ``tools[]`` requests into Gemini's native +``models/{model}:generateContent`` schema and converts the responses back. + +Why this exists +--------------- +Google's OpenAI-compatible endpoint has been brittle for Hermes's multi-turn +agent/tool loop (auth churn, tool-call replay quirks, thought-signature +requirements). The native Gemini API is the canonical path and avoids the +OpenAI-compat layer entirely. +""" + +from __future__ import annotations + +import asyncio +import base64 +import json +import logging +import time +import uuid +from types import SimpleNamespace +from typing import Any, Dict, Iterator, List, Optional + +import httpx + +from agent.gemini_schema import sanitize_gemini_tool_parameters + +logger = logging.getLogger(__name__) + +DEFAULT_GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta" + + +def is_native_gemini_base_url(base_url: str) -> bool: + """Return True when the endpoint speaks Gemini's native REST API.""" + normalized = str(base_url or "").strip().rstrip("/").lower() + if not normalized: + return False + if "generativelanguage.googleapis.com" not in normalized: + return False + return not normalized.endswith("/openai") + + +class GeminiAPIError(Exception): + """Error shape compatible with Hermes retry/error classification.""" + + def __init__( + self, + message: str, + *, + code: str = "gemini_api_error", + status_code: Optional[int] = None, + response: Optional[httpx.Response] = None, + retry_after: Optional[float] = None, + details: Optional[Dict[str, Any]] = None, + ) -> None: + super().__init__(message) + self.code = code + self.status_code = status_code + self.response = response + self.retry_after = retry_after + self.details = details or {} + + +def _coerce_content_to_text(content: Any) -> str: + if content is None: + return "" + if isinstance(content, str): + return content + if isinstance(content, list): + pieces: List[str] = [] + for part in content: + if isinstance(part, str): + pieces.append(part) + elif isinstance(part, dict) and part.get("type") == "text": + text = part.get("text") + if isinstance(text, str): + pieces.append(text) + return "\n".join(pieces) + return str(content) + + +def _extract_multimodal_parts(content: Any) -> List[Dict[str, Any]]: + if not isinstance(content, list): + text = _coerce_content_to_text(content) + return [{"text": text}] if text else [] + + parts: List[Dict[str, Any]] = [] + for item in content: + if isinstance(item, str): + parts.append({"text": item}) + continue + if not isinstance(item, dict): + continue + ptype = item.get("type") + if ptype == "text": + text = item.get("text") + if isinstance(text, str) and text: + parts.append({"text": text}) + elif ptype == "image_url": + url = ((item.get("image_url") or {}).get("url") or "") + if not isinstance(url, str) or not url.startswith("data:"): + continue + try: + header, encoded = url.split(",", 1) + mime = header.split(":", 1)[1].split(";", 1)[0] + raw = base64.b64decode(encoded) + except Exception: + continue + parts.append( + { + "inlineData": { + "mimeType": mime, + "data": base64.b64encode(raw).decode("ascii"), + } + } + ) + return parts + + +def _tool_call_extra_signature(tool_call: Dict[str, Any]) -> Optional[str]: + extra = tool_call.get("extra_content") or {} + if not isinstance(extra, dict): + return None + google = extra.get("google") or extra.get("thought_signature") + if isinstance(google, dict): + sig = google.get("thought_signature") or google.get("thoughtSignature") + return str(sig) if isinstance(sig, str) and sig else None + if isinstance(google, str) and google: + return google + return None + + +def _translate_tool_call_to_gemini(tool_call: Dict[str, Any]) -> Dict[str, Any]: + fn = tool_call.get("function") or {} + args_raw = fn.get("arguments", "") + try: + args = json.loads(args_raw) if isinstance(args_raw, str) and args_raw else {} + except json.JSONDecodeError: + args = {"_raw": args_raw} + if not isinstance(args, dict): + args = {"_value": args} + + part: Dict[str, Any] = { + "functionCall": { + "name": str(fn.get("name") or ""), + "args": args, + } + } + thought_signature = _tool_call_extra_signature(tool_call) + if thought_signature: + part["thoughtSignature"] = thought_signature + return part + + +def _translate_tool_result_to_gemini( + message: Dict[str, Any], + tool_name_by_call_id: Optional[Dict[str, str]] = None, +) -> Dict[str, Any]: + tool_name_by_call_id = tool_name_by_call_id or {} + tool_call_id = str(message.get("tool_call_id") or "") + name = str( + message.get("name") + or tool_name_by_call_id.get(tool_call_id) + or tool_call_id + or "tool" + ) + content = _coerce_content_to_text(message.get("content")) + try: + parsed = json.loads(content) if content.strip().startswith(("{", "[")) else None + except json.JSONDecodeError: + parsed = None + response = parsed if isinstance(parsed, dict) else {"output": content} + return { + "functionResponse": { + "name": name, + "response": response, + } + } + + +def _build_gemini_contents(messages: List[Dict[str, Any]]) -> tuple[List[Dict[str, Any]], Optional[Dict[str, Any]]]: + system_text_parts: List[str] = [] + contents: List[Dict[str, Any]] = [] + tool_name_by_call_id: Dict[str, str] = {} + + for msg in messages: + if not isinstance(msg, dict): + continue + role = str(msg.get("role") or "user") + + if role == "system": + system_text_parts.append(_coerce_content_to_text(msg.get("content"))) + continue + + if role in {"tool", "function"}: + contents.append( + { + "role": "user", + "parts": [ + _translate_tool_result_to_gemini( + msg, + tool_name_by_call_id=tool_name_by_call_id, + ) + ], + } + ) + continue + + gemini_role = "model" if role == "assistant" else "user" + parts: List[Dict[str, Any]] = [] + + content_parts = _extract_multimodal_parts(msg.get("content")) + parts.extend(content_parts) + + tool_calls = msg.get("tool_calls") or [] + if isinstance(tool_calls, list): + for tool_call in tool_calls: + if isinstance(tool_call, dict): + tool_call_id = str(tool_call.get("id") or tool_call.get("call_id") or "") + tool_name = str(((tool_call.get("function") or {}).get("name") or "")) + if tool_call_id and tool_name: + tool_name_by_call_id[tool_call_id] = tool_name + parts.append(_translate_tool_call_to_gemini(tool_call)) + + if parts: + contents.append({"role": gemini_role, "parts": parts}) + + system_instruction = None + joined_system = "\n".join(part for part in system_text_parts if part).strip() + if joined_system: + system_instruction = {"parts": [{"text": joined_system}]} + return contents, system_instruction + + +def _translate_tools_to_gemini(tools: Any) -> List[Dict[str, Any]]: + if not isinstance(tools, list): + return [] + declarations: List[Dict[str, Any]] = [] + for tool in tools: + if not isinstance(tool, dict): + continue + fn = tool.get("function") or {} + if not isinstance(fn, dict): + continue + name = fn.get("name") + if not isinstance(name, str) or not name: + continue + decl: Dict[str, Any] = {"name": name} + description = fn.get("description") + if isinstance(description, str) and description: + decl["description"] = description + parameters = fn.get("parameters") + if isinstance(parameters, dict): + decl["parameters"] = sanitize_gemini_tool_parameters(parameters) + declarations.append(decl) + return [{"functionDeclarations": declarations}] if declarations else [] + + +def _translate_tool_choice_to_gemini(tool_choice: Any) -> Optional[Dict[str, Any]]: + if tool_choice is None: + return None + if isinstance(tool_choice, str): + if tool_choice == "auto": + return {"functionCallingConfig": {"mode": "AUTO"}} + if tool_choice == "required": + return {"functionCallingConfig": {"mode": "ANY"}} + if tool_choice == "none": + return {"functionCallingConfig": {"mode": "NONE"}} + if isinstance(tool_choice, dict): + fn = tool_choice.get("function") or {} + name = fn.get("name") + if isinstance(name, str) and name: + return {"functionCallingConfig": {"mode": "ANY", "allowedFunctionNames": [name]}} + return None + + +def _normalize_thinking_config(config: Any) -> Optional[Dict[str, Any]]: + if not isinstance(config, dict) or not config: + return None + budget = config.get("thinkingBudget", config.get("thinking_budget")) + include = config.get("includeThoughts", config.get("include_thoughts")) + level = config.get("thinkingLevel", config.get("thinking_level")) + normalized: Dict[str, Any] = {} + if isinstance(budget, (int, float)): + normalized["thinkingBudget"] = int(budget) + if isinstance(include, bool): + normalized["includeThoughts"] = include + if isinstance(level, str) and level.strip(): + normalized["thinkingLevel"] = level.strip().lower() + return normalized or None + + +def build_gemini_request( + *, + messages: List[Dict[str, Any]], + tools: Any = None, + tool_choice: Any = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, + top_p: Optional[float] = None, + stop: Any = None, + thinking_config: Any = None, +) -> Dict[str, Any]: + contents, system_instruction = _build_gemini_contents(messages) + request: Dict[str, Any] = {"contents": contents} + if system_instruction: + request["systemInstruction"] = system_instruction + + gemini_tools = _translate_tools_to_gemini(tools) + if gemini_tools: + request["tools"] = gemini_tools + + tool_config = _translate_tool_choice_to_gemini(tool_choice) + if tool_config: + request["toolConfig"] = tool_config + + generation_config: Dict[str, Any] = {} + if temperature is not None: + generation_config["temperature"] = temperature + if max_tokens is not None: + generation_config["maxOutputTokens"] = max_tokens + if top_p is not None: + generation_config["topP"] = top_p + if stop: + generation_config["stopSequences"] = stop if isinstance(stop, list) else [str(stop)] + normalized_thinking = _normalize_thinking_config(thinking_config) + if normalized_thinking: + generation_config["thinkingConfig"] = normalized_thinking + if generation_config: + request["generationConfig"] = generation_config + + return request + + +def _map_gemini_finish_reason(reason: str) -> str: + mapping = { + "STOP": "stop", + "MAX_TOKENS": "length", + "SAFETY": "content_filter", + "RECITATION": "content_filter", + "OTHER": "stop", + } + return mapping.get(str(reason or "").upper(), "stop") + + +def _tool_call_extra_from_part(part: Dict[str, Any]) -> Optional[Dict[str, Any]]: + sig = part.get("thoughtSignature") + if isinstance(sig, str) and sig: + return {"google": {"thought_signature": sig}} + return None + + +def _empty_response(model: str) -> SimpleNamespace: + message = SimpleNamespace( + role="assistant", + content="", + tool_calls=None, + reasoning=None, + reasoning_content=None, + reasoning_details=None, + ) + choice = SimpleNamespace(index=0, message=message, finish_reason="stop") + usage = SimpleNamespace( + prompt_tokens=0, + completion_tokens=0, + total_tokens=0, + prompt_tokens_details=SimpleNamespace(cached_tokens=0), + ) + return SimpleNamespace( + id=f"chatcmpl-{uuid.uuid4().hex[:12]}", + object="chat.completion", + created=int(time.time()), + model=model, + choices=[choice], + usage=usage, + ) + + +def translate_gemini_response(resp: Dict[str, Any], model: str) -> SimpleNamespace: + candidates = resp.get("candidates") or [] + if not isinstance(candidates, list) or not candidates: + return _empty_response(model) + + cand = candidates[0] if isinstance(candidates[0], dict) else {} + content_obj = cand.get("content") if isinstance(cand, dict) else {} + parts = content_obj.get("parts") if isinstance(content_obj, dict) else [] + + text_pieces: List[str] = [] + reasoning_pieces: List[str] = [] + tool_calls: List[SimpleNamespace] = [] + + for index, part in enumerate(parts or []): + if not isinstance(part, dict): + continue + if part.get("thought") is True and isinstance(part.get("text"), str): + reasoning_pieces.append(part["text"]) + continue + if isinstance(part.get("text"), str): + text_pieces.append(part["text"]) + continue + fc = part.get("functionCall") + if isinstance(fc, dict) and fc.get("name"): + try: + args_str = json.dumps(fc.get("args") or {}, ensure_ascii=False) + except (TypeError, ValueError): + args_str = "{}" + tool_call = SimpleNamespace( + id=f"call_{uuid.uuid4().hex[:12]}", + type="function", + index=index, + function=SimpleNamespace(name=str(fc["name"]), arguments=args_str), + ) + extra_content = _tool_call_extra_from_part(part) + if extra_content: + tool_call.extra_content = extra_content + tool_calls.append(tool_call) + + finish_reason = "tool_calls" if tool_calls else _map_gemini_finish_reason(str(cand.get("finishReason") or "")) + usage_meta = resp.get("usageMetadata") or {} + usage = SimpleNamespace( + prompt_tokens=int(usage_meta.get("promptTokenCount") or 0), + completion_tokens=int(usage_meta.get("candidatesTokenCount") or 0), + total_tokens=int(usage_meta.get("totalTokenCount") or 0), + prompt_tokens_details=SimpleNamespace( + cached_tokens=int(usage_meta.get("cachedContentTokenCount") or 0), + ), + ) + reasoning = "".join(reasoning_pieces) or None + message = SimpleNamespace( + role="assistant", + content="".join(text_pieces) if text_pieces else None, + tool_calls=tool_calls or None, + reasoning=reasoning, + reasoning_content=reasoning, + reasoning_details=None, + ) + choice = SimpleNamespace(index=0, message=message, finish_reason=finish_reason) + return SimpleNamespace( + id=f"chatcmpl-{uuid.uuid4().hex[:12]}", + object="chat.completion", + created=int(time.time()), + model=model, + choices=[choice], + usage=usage, + ) + + +class _GeminiStreamChunk(SimpleNamespace): + pass + + +def _make_stream_chunk( + *, + model: str, + content: str = "", + tool_call_delta: Optional[Dict[str, Any]] = None, + finish_reason: Optional[str] = None, + reasoning: str = "", +) -> _GeminiStreamChunk: + delta_kwargs: Dict[str, Any] = { + "role": "assistant", + "content": None, + "tool_calls": None, + "reasoning": None, + "reasoning_content": None, + } + if content: + delta_kwargs["content"] = content + if tool_call_delta is not None: + tool_delta = SimpleNamespace( + index=tool_call_delta.get("index", 0), + id=tool_call_delta.get("id") or f"call_{uuid.uuid4().hex[:12]}", + type="function", + function=SimpleNamespace( + name=tool_call_delta.get("name") or "", + arguments=tool_call_delta.get("arguments") or "", + ), + ) + extra_content = tool_call_delta.get("extra_content") + if isinstance(extra_content, dict): + tool_delta.extra_content = extra_content + delta_kwargs["tool_calls"] = [tool_delta] + if reasoning: + delta_kwargs["reasoning"] = reasoning + delta_kwargs["reasoning_content"] = reasoning + delta = SimpleNamespace(**delta_kwargs) + choice = SimpleNamespace(index=0, delta=delta, finish_reason=finish_reason) + return _GeminiStreamChunk( + id=f"chatcmpl-{uuid.uuid4().hex[:12]}", + object="chat.completion.chunk", + created=int(time.time()), + model=model, + choices=[choice], + usage=None, + ) + + +def _iter_sse_events(response: httpx.Response) -> Iterator[Dict[str, Any]]: + buffer = "" + for chunk in response.iter_text(): + if not chunk: + continue + buffer += chunk + while "\n" in buffer: + line, buffer = buffer.split("\n", 1) + line = line.rstrip("\r") + if not line: + continue + if not line.startswith("data: "): + continue + data = line[6:] + if data == "[DONE]": + return + try: + payload = json.loads(data) + except json.JSONDecodeError: + logger.debug("Non-JSON Gemini SSE line: %s", data[:200]) + continue + if isinstance(payload, dict): + yield payload + + +def translate_stream_event(event: Dict[str, Any], model: str, tool_call_indices: Dict[str, Dict[str, Any]]) -> List[_GeminiStreamChunk]: + candidates = event.get("candidates") or [] + if not candidates: + return [] + cand = candidates[0] if isinstance(candidates[0], dict) else {} + parts = ((cand.get("content") or {}).get("parts") or []) if isinstance(cand, dict) else [] + chunks: List[_GeminiStreamChunk] = [] + + for part_index, part in enumerate(parts): + if not isinstance(part, dict): + continue + if part.get("thought") is True and isinstance(part.get("text"), str): + chunks.append(_make_stream_chunk(model=model, reasoning=part["text"])) + continue + if isinstance(part.get("text"), str) and part["text"]: + chunks.append(_make_stream_chunk(model=model, content=part["text"])) + fc = part.get("functionCall") + if isinstance(fc, dict) and fc.get("name"): + name = str(fc["name"]) + try: + args_str = json.dumps(fc.get("args") or {}, ensure_ascii=False, sort_keys=True) + except (TypeError, ValueError): + args_str = "{}" + thought_signature = part.get("thoughtSignature") if isinstance(part.get("thoughtSignature"), str) else "" + call_key = json.dumps( + { + "part_index": part_index, + "name": name, + "thought_signature": thought_signature, + }, + sort_keys=True, + ) + slot = tool_call_indices.get(call_key) + if slot is None: + slot = { + "index": len(tool_call_indices), + "id": f"call_{uuid.uuid4().hex[:12]}", + "last_arguments": "", + } + tool_call_indices[call_key] = slot + emitted_arguments = args_str + last_arguments = str(slot.get("last_arguments") or "") + if last_arguments: + if args_str == last_arguments: + emitted_arguments = "" + elif args_str.startswith(last_arguments): + emitted_arguments = args_str[len(last_arguments):] + slot["last_arguments"] = args_str + chunks.append( + _make_stream_chunk( + model=model, + tool_call_delta={ + "index": slot["index"], + "id": slot["id"], + "name": name, + "arguments": emitted_arguments, + "extra_content": _tool_call_extra_from_part(part), + }, + ) + ) + + finish_reason_raw = str(cand.get("finishReason") or "") + if finish_reason_raw: + mapped = "tool_calls" if tool_call_indices else _map_gemini_finish_reason(finish_reason_raw) + chunks.append(_make_stream_chunk(model=model, finish_reason=mapped)) + return chunks + + +def gemini_http_error(response: httpx.Response) -> GeminiAPIError: + status = response.status_code + body_text = "" + body_json: Dict[str, Any] = {} + try: + body_text = response.text + except Exception: + body_text = "" + if body_text: + try: + parsed = json.loads(body_text) + if isinstance(parsed, dict): + body_json = parsed + except (ValueError, TypeError): + body_json = {} + + err_obj = body_json.get("error") if isinstance(body_json, dict) else None + if not isinstance(err_obj, dict): + err_obj = {} + err_status = str(err_obj.get("status") or "").strip() + err_message = str(err_obj.get("message") or "").strip() + _raw_details = err_obj.get("details") + details_list = _raw_details if isinstance(_raw_details, list) else [] + + reason = "" + retry_after: Optional[float] = None + metadata: Dict[str, Any] = {} + for detail in details_list: + if not isinstance(detail, dict): + continue + type_url = str(detail.get("@type") or "") + if not reason and type_url.endswith("/google.rpc.ErrorInfo"): + reason_value = detail.get("reason") + if isinstance(reason_value, str): + reason = reason_value + md = detail.get("metadata") + if isinstance(md, dict): + metadata = md + header_retry = response.headers.get("Retry-After") or response.headers.get("retry-after") + if header_retry: + try: + retry_after = float(header_retry) + except (TypeError, ValueError): + retry_after = None + + code = f"gemini_http_{status}" + if status == 401: + code = "gemini_unauthorized" + elif status == 429: + code = "gemini_rate_limited" + elif status == 404: + code = "gemini_model_not_found" + + if err_message: + message = f"Gemini HTTP {status} ({err_status or 'error'}): {err_message}" + else: + message = f"Gemini returned HTTP {status}: {body_text[:500]}" + + return GeminiAPIError( + message, + code=code, + status_code=status, + response=response, + retry_after=retry_after, + details={ + "status": err_status, + "reason": reason, + "metadata": metadata, + "message": err_message, + }, + ) + + +class _GeminiChatCompletions: + def __init__(self, client: "GeminiNativeClient"): + self._client = client + + def create(self, **kwargs: Any) -> Any: + return self._client._create_chat_completion(**kwargs) + + +class _AsyncGeminiChatCompletions: + def __init__(self, client: "AsyncGeminiNativeClient"): + self._client = client + + async def create(self, **kwargs: Any) -> Any: + return await self._client._create_chat_completion(**kwargs) + + +class _GeminiChatNamespace: + def __init__(self, client: "GeminiNativeClient"): + self.completions = _GeminiChatCompletions(client) + + +class _AsyncGeminiChatNamespace: + def __init__(self, client: "AsyncGeminiNativeClient"): + self.completions = _AsyncGeminiChatCompletions(client) + + +class GeminiNativeClient: + """Minimal OpenAI-SDK-compatible facade over Gemini's native REST API.""" + + def __init__( + self, + *, + api_key: str, + base_url: Optional[str] = None, + default_headers: Optional[Dict[str, str]] = None, + timeout: Any = None, + http_client: Optional[httpx.Client] = None, + **_: Any, + ) -> None: + self.api_key = api_key + normalized_base = (base_url or DEFAULT_GEMINI_BASE_URL).rstrip("/") + if normalized_base.endswith("/openai"): + normalized_base = normalized_base[: -len("/openai")] + self.base_url = normalized_base + self._default_headers = dict(default_headers or {}) + self.chat = _GeminiChatNamespace(self) + self.is_closed = False + self._http = http_client or httpx.Client( + timeout=timeout or httpx.Timeout(connect=15.0, read=600.0, write=30.0, pool=30.0) + ) + + def close(self) -> None: + self.is_closed = True + try: + self._http.close() + except Exception: + pass + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + self.close() + + def _headers(self) -> Dict[str, str]: + headers = { + "Content-Type": "application/json", + "Accept": "application/json", + "x-goog-api-key": self.api_key, + "User-Agent": "hermes-agent (gemini-native)", + } + headers.update(self._default_headers) + return headers + + @staticmethod + def _advance_stream_iterator(iterator: Iterator[_GeminiStreamChunk]) -> tuple[bool, Optional[_GeminiStreamChunk]]: + try: + return False, next(iterator) + except StopIteration: + return True, None + + def _create_chat_completion( + self, + *, + model: str = "gemini-2.5-flash", + messages: Optional[List[Dict[str, Any]]] = None, + stream: bool = False, + tools: Any = None, + tool_choice: Any = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, + top_p: Optional[float] = None, + stop: Any = None, + extra_body: Optional[Dict[str, Any]] = None, + timeout: Any = None, + **_: Any, + ) -> Any: + thinking_config = None + if isinstance(extra_body, dict): + thinking_config = extra_body.get("thinking_config") or extra_body.get("thinkingConfig") + + request = build_gemini_request( + messages=messages or [], + tools=tools, + tool_choice=tool_choice, + temperature=temperature, + max_tokens=max_tokens, + top_p=top_p, + stop=stop, + thinking_config=thinking_config, + ) + + if stream: + return self._stream_completion(model=model, request=request, timeout=timeout) + + url = f"{self.base_url}/models/{model}:generateContent" + response = self._http.post(url, json=request, headers=self._headers(), timeout=timeout) + if response.status_code != 200: + raise gemini_http_error(response) + try: + payload = response.json() + except ValueError as exc: + raise GeminiAPIError( + f"Invalid JSON from Gemini native API: {exc}", + code="gemini_invalid_json", + status_code=response.status_code, + response=response, + ) from exc + return translate_gemini_response(payload, model=model) + + def _stream_completion(self, *, model: str, request: Dict[str, Any], timeout: Any = None) -> Iterator[_GeminiStreamChunk]: + url = f"{self.base_url}/models/{model}:streamGenerateContent?alt=sse" + stream_headers = dict(self._headers()) + stream_headers["Accept"] = "text/event-stream" + + def _generator() -> Iterator[_GeminiStreamChunk]: + try: + with self._http.stream("POST", url, json=request, headers=stream_headers, timeout=timeout) as response: + if response.status_code != 200: + response.read() + raise gemini_http_error(response) + tool_call_indices: Dict[str, Dict[str, Any]] = {} + for event in _iter_sse_events(response): + for chunk in translate_stream_event(event, model, tool_call_indices): + yield chunk + except httpx.HTTPError as exc: + raise GeminiAPIError( + f"Gemini streaming request failed: {exc}", + code="gemini_stream_error", + ) from exc + + return _generator() + + +class AsyncGeminiNativeClient: + """Async wrapper used by auxiliary_client for native Gemini calls.""" + + def __init__(self, sync_client: GeminiNativeClient): + self._sync = sync_client + self.api_key = sync_client.api_key + self.base_url = sync_client.base_url + self.chat = _AsyncGeminiChatNamespace(self) + + async def _create_chat_completion(self, **kwargs: Any) -> Any: + stream = bool(kwargs.get("stream")) + result = await asyncio.to_thread(self._sync.chat.completions.create, **kwargs) + if not stream: + return result + + async def _async_stream() -> Any: + while True: + done, chunk = await asyncio.to_thread(self._sync._advance_stream_iterator, result) + if done: + break + yield chunk + + return _async_stream() + + async def close(self) -> None: + await asyncio.to_thread(self._sync.close) diff --git a/agent/gemini_schema.py b/agent/gemini_schema.py new file mode 100644 index 000000000000..904c99d31b8e --- /dev/null +++ b/agent/gemini_schema.py @@ -0,0 +1,85 @@ +"""Helpers for translating OpenAI-style tool schemas to Gemini's schema subset.""" + +from __future__ import annotations + +from typing import Any, Dict, List + +# Gemini's ``FunctionDeclaration.parameters`` field accepts the ``Schema`` +# object, which is only a subset of OpenAPI 3.0 / JSON Schema. Strip fields +# outside that subset before sending Hermes tool schemas to Google. +_GEMINI_SCHEMA_ALLOWED_KEYS = { + "type", + "format", + "title", + "description", + "nullable", + "enum", + "maxItems", + "minItems", + "properties", + "required", + "minProperties", + "maxProperties", + "minLength", + "maxLength", + "pattern", + "example", + "anyOf", + "propertyOrdering", + "default", + "items", + "minimum", + "maximum", +} + + +def sanitize_gemini_schema(schema: Any) -> Dict[str, Any]: + """Return a Gemini-compatible copy of a tool parameter schema. + + Hermes tool schemas are OpenAI-flavored JSON Schema and may contain keys + such as ``$schema`` or ``additionalProperties`` that Google's Gemini + ``Schema`` object rejects. This helper preserves the documented Gemini + subset and recursively sanitizes nested ``properties`` / ``items`` / + ``anyOf`` definitions. + """ + + if not isinstance(schema, dict): + return {} + + cleaned: Dict[str, Any] = {} + for key, value in schema.items(): + if key not in _GEMINI_SCHEMA_ALLOWED_KEYS: + continue + if key == "properties": + if not isinstance(value, dict): + continue + props: Dict[str, Any] = {} + for prop_name, prop_schema in value.items(): + if not isinstance(prop_name, str): + continue + props[prop_name] = sanitize_gemini_schema(prop_schema) + cleaned[key] = props + continue + if key == "items": + cleaned[key] = sanitize_gemini_schema(value) + continue + if key == "anyOf": + if not isinstance(value, list): + continue + cleaned[key] = [ + sanitize_gemini_schema(item) + for item in value + if isinstance(item, dict) + ] + continue + cleaned[key] = value + return cleaned + + +def sanitize_gemini_tool_parameters(parameters: Any) -> Dict[str, Any]: + """Normalize tool parameters to a valid Gemini object schema.""" + + cleaned = sanitize_gemini_schema(parameters) + if not cleaned: + return {"type": "object", "properties": {}} + return cleaned diff --git a/agent/google_code_assist.py b/agent/google_code_assist.py new file mode 100644 index 000000000000..eba09b8f46b5 --- /dev/null +++ b/agent/google_code_assist.py @@ -0,0 +1,453 @@ +"""Google Code Assist API client — project discovery, onboarding, quota. + +The Code Assist API powers Google's official gemini-cli. It sits at +``cloudcode-pa.googleapis.com`` and provides: + +- Free tier access (generous daily quota) for personal Google accounts +- Paid tier access via GCP projects with billing / Workspace / Standard / Enterprise + +This module handles the control-plane dance needed before inference: + +1. ``load_code_assist()`` — probe the user's account to learn what tier they're on + and whether a ``cloudaicompanionProject`` is already assigned. +2. ``onboard_user()`` — if the user hasn't been onboarded yet (new account, fresh + free tier, etc.), call this with the chosen tier + project id. Supports LRO + polling for slow provisioning. +3. ``retrieve_user_quota()`` — fetch the ``buckets[]`` array showing remaining + quota per model, used by the ``/gquota`` slash command. + +VPC-SC handling: enterprise accounts under a VPC Service Controls perimeter +will get ``SECURITY_POLICY_VIOLATED`` on ``load_code_assist``. We catch this +and force the account to ``standard-tier`` so the call chain still succeeds. + +Derived from opencode-gemini-auth (MIT) and clawdbot/extensions/google. The +request/response shapes are specific to Google's internal Code Assist API, +documented nowhere public — we copy them from the reference implementations. +""" + +from __future__ import annotations + +import json +import logging +import os +import time +import urllib.error +import urllib.parse +import urllib.request +import uuid +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional + +logger = logging.getLogger(__name__) + + +# ============================================================================= +# Constants +# ============================================================================= + +CODE_ASSIST_ENDPOINT = "https://cloudcode-pa.googleapis.com" + +# Fallback endpoints tried when prod returns an error during project discovery +FALLBACK_ENDPOINTS = [ + "https://daily-cloudcode-pa.sandbox.googleapis.com", + "https://autopush-cloudcode-pa.sandbox.googleapis.com", +] + +# Tier identifiers that Google's API uses +FREE_TIER_ID = "free-tier" +LEGACY_TIER_ID = "legacy-tier" +STANDARD_TIER_ID = "standard-tier" + +# Default HTTP headers matching gemini-cli's fingerprint. +# Google may reject unrecognized User-Agents on these internal endpoints. +_GEMINI_CLI_USER_AGENT = "google-api-nodejs-client/9.15.1 (gzip)" +_X_GOOG_API_CLIENT = "gl-node/24.0.0" +_DEFAULT_REQUEST_TIMEOUT = 30.0 +_ONBOARDING_POLL_ATTEMPTS = 12 +_ONBOARDING_POLL_INTERVAL_SECONDS = 5.0 + + +class CodeAssistError(RuntimeError): + """Exception raised by the Code Assist (``cloudcode-pa``) integration. + + Carries HTTP status / response / retry-after metadata so the agent's + ``error_classifier._extract_status_code`` and the main loop's Retry-After + handling (which walks ``error.response.headers``) pick up the right + signals. Without these, 429s from the OAuth path look like opaque + ``RuntimeError`` and skip the rate-limit path. + """ + + def __init__( + self, + message: str, + *, + code: str = "code_assist_error", + status_code: Optional[int] = None, + response: Any = None, + retry_after: Optional[float] = None, + details: Optional[Dict[str, Any]] = None, + ) -> None: + super().__init__(message) + self.code = code + # ``status_code`` is picked up by ``agent.error_classifier._extract_status_code`` + # so a 429 from Code Assist classifies as FailoverReason.rate_limit and + # triggers the main loop's fallback_providers chain the same way SDK + # errors do. + self.status_code = status_code + # ``response`` is the underlying ``httpx.Response`` (or a shim with a + # ``.headers`` mapping and ``.json()`` method). The main loop reads + # ``error.response.headers["Retry-After"]`` to honor Google's retry + # hints when the backend throttles us. + self.response = response + # Parsed ``Retry-After`` seconds (kept separately for convenience — + # Google returns retry hints in both the header and the error body's + # ``google.rpc.RetryInfo`` details, and we pick whichever we found). + self.retry_after = retry_after + # Parsed structured error details from the Google error envelope + # (e.g. ``{"reason": "MODEL_CAPACITY_EXHAUSTED", "status": "RESOURCE_EXHAUSTED"}``). + # Useful for logging and for tests that want to assert on specifics. + self.details = details or {} + + +class ProjectIdRequiredError(CodeAssistError): + def __init__(self, message: str = "GCP project id required for this tier") -> None: + super().__init__(message, code="code_assist_project_id_required") + + +# ============================================================================= +# HTTP primitive (auth via Bearer token passed per-call) +# ============================================================================= + +def _build_headers(access_token: str, *, user_agent_model: str = "") -> Dict[str, str]: + ua = _GEMINI_CLI_USER_AGENT + if user_agent_model: + ua = f"{ua} model/{user_agent_model}" + return { + "Content-Type": "application/json", + "Accept": "application/json", + "Authorization": f"Bearer {access_token}", + "User-Agent": ua, + "X-Goog-Api-Client": _X_GOOG_API_CLIENT, + "x-activity-request-id": str(uuid.uuid4()), + } + + +def _client_metadata() -> Dict[str, str]: + """Match Google's gemini-cli exactly — unrecognized metadata may be rejected.""" + return { + "ideType": "IDE_UNSPECIFIED", + "platform": "PLATFORM_UNSPECIFIED", + "pluginType": "GEMINI", + } + + +def _post_json( + url: str, + body: Dict[str, Any], + access_token: str, + *, + timeout: float = _DEFAULT_REQUEST_TIMEOUT, + user_agent_model: str = "", +) -> Dict[str, Any]: + data = json.dumps(body).encode("utf-8") + request = urllib.request.Request( + url, data=data, method="POST", + headers=_build_headers(access_token, user_agent_model=user_agent_model), + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + raw = response.read().decode("utf-8", errors="replace") + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as exc: + detail = "" + try: + detail = exc.read().decode("utf-8", errors="replace") + except Exception: + pass + # Special case: VPC-SC violation should be distinguishable + if _is_vpc_sc_violation(detail): + raise CodeAssistError( + f"VPC-SC policy violation: {detail}", + code="code_assist_vpc_sc", + ) from exc + raise CodeAssistError( + f"Code Assist HTTP {exc.code}: {detail or exc.reason}", + code=f"code_assist_http_{exc.code}", + ) from exc + except urllib.error.URLError as exc: + raise CodeAssistError( + f"Code Assist request failed: {exc}", + code="code_assist_network_error", + ) from exc + + +def _is_vpc_sc_violation(body: str) -> bool: + """Detect a VPC Service Controls violation from a response body.""" + if not body: + return False + try: + parsed = json.loads(body) + except (json.JSONDecodeError, ValueError): + return "SECURITY_POLICY_VIOLATED" in body + # Walk the nested error structure Google uses + error = parsed.get("error") if isinstance(parsed, dict) else None + if not isinstance(error, dict): + return False + details = error.get("details") or [] + if isinstance(details, list): + for item in details: + if isinstance(item, dict): + reason = item.get("reason") or "" + if reason == "SECURITY_POLICY_VIOLATED": + return True + msg = str(error.get("message", "")) + return "SECURITY_POLICY_VIOLATED" in msg + + +# ============================================================================= +# load_code_assist — discovers current tier + assigned project +# ============================================================================= + +@dataclass +class CodeAssistProjectInfo: + """Result from ``load_code_assist``.""" + current_tier_id: str = "" + cloudaicompanion_project: str = "" # Google-managed project (free tier) + allowed_tiers: List[str] = field(default_factory=list) + raw: Dict[str, Any] = field(default_factory=dict) + + +def load_code_assist( + access_token: str, + *, + project_id: str = "", + user_agent_model: str = "", +) -> CodeAssistProjectInfo: + """Call ``POST /v1internal:loadCodeAssist`` with prod → sandbox fallback. + + Returns whatever tier + project info Google reports. On VPC-SC violations, + returns a synthetic ``standard-tier`` result so the chain can continue. + """ + body: Dict[str, Any] = { + "metadata": { + "duetProject": project_id, + **_client_metadata(), + }, + } + if project_id: + body["cloudaicompanionProject"] = project_id + + endpoints = [CODE_ASSIST_ENDPOINT] + FALLBACK_ENDPOINTS + last_err: Optional[Exception] = None + for endpoint in endpoints: + url = f"{endpoint}/v1internal:loadCodeAssist" + try: + resp = _post_json(url, body, access_token, user_agent_model=user_agent_model) + return _parse_load_response(resp) + except CodeAssistError as exc: + if exc.code == "code_assist_vpc_sc": + logger.info("VPC-SC violation on %s — defaulting to standard-tier", endpoint) + return CodeAssistProjectInfo( + current_tier_id=STANDARD_TIER_ID, + cloudaicompanion_project=project_id, + ) + last_err = exc + logger.warning("loadCodeAssist failed on %s: %s", endpoint, exc) + continue + if last_err: + raise last_err + return CodeAssistProjectInfo() + + +def _parse_load_response(resp: Dict[str, Any]) -> CodeAssistProjectInfo: + current_tier = resp.get("currentTier") or {} + tier_id = str(current_tier.get("id") or "") if isinstance(current_tier, dict) else "" + project = str(resp.get("cloudaicompanionProject") or "") + allowed = resp.get("allowedTiers") or [] + allowed_ids: List[str] = [] + if isinstance(allowed, list): + for t in allowed: + if isinstance(t, dict): + tid = str(t.get("id") or "") + if tid: + allowed_ids.append(tid) + return CodeAssistProjectInfo( + current_tier_id=tier_id, + cloudaicompanion_project=project, + allowed_tiers=allowed_ids, + raw=resp, + ) + + +# ============================================================================= +# onboard_user — provisions a new user on a tier (with LRO polling) +# ============================================================================= + +def onboard_user( + access_token: str, + *, + tier_id: str, + project_id: str = "", + user_agent_model: str = "", +) -> Dict[str, Any]: + """Call ``POST /v1internal:onboardUser`` to provision the user. + + For paid tiers, ``project_id`` is REQUIRED (raises ProjectIdRequiredError). + For free tiers, ``project_id`` is optional — Google will assign one. + + Returns the final operation response. Polls ``/v1internal/`` for up + to ``_ONBOARDING_POLL_ATTEMPTS`` × ``_ONBOARDING_POLL_INTERVAL_SECONDS`` + (default: 12 × 5s = 1 min). + """ + if tier_id != FREE_TIER_ID and tier_id != LEGACY_TIER_ID and not project_id: + raise ProjectIdRequiredError( + f"Tier {tier_id!r} requires a GCP project id. " + "Set HERMES_GEMINI_PROJECT_ID or GOOGLE_CLOUD_PROJECT." + ) + + body: Dict[str, Any] = { + "tierId": tier_id, + "metadata": _client_metadata(), + } + if project_id: + body["cloudaicompanionProject"] = project_id + + endpoint = CODE_ASSIST_ENDPOINT + url = f"{endpoint}/v1internal:onboardUser" + resp = _post_json(url, body, access_token, user_agent_model=user_agent_model) + + # Poll if LRO (long-running operation) + if not resp.get("done"): + op_name = resp.get("name", "") + if not op_name: + return resp + for attempt in range(_ONBOARDING_POLL_ATTEMPTS): + time.sleep(_ONBOARDING_POLL_INTERVAL_SECONDS) + poll_url = f"{endpoint}/v1internal/{op_name}" + try: + poll_resp = _post_json(poll_url, {}, access_token, user_agent_model=user_agent_model) + except CodeAssistError as exc: + logger.warning("Onboarding poll attempt %d failed: %s", attempt + 1, exc) + continue + if poll_resp.get("done"): + return poll_resp + logger.warning("Onboarding did not complete within %d attempts", _ONBOARDING_POLL_ATTEMPTS) + return resp + + +# ============================================================================= +# retrieve_user_quota — for /gquota +# ============================================================================= + +@dataclass +class QuotaBucket: + model_id: str + token_type: str = "" + remaining_fraction: float = 0.0 + reset_time_iso: str = "" + raw: Dict[str, Any] = field(default_factory=dict) + + +def retrieve_user_quota( + access_token: str, + *, + project_id: str = "", + user_agent_model: str = "", +) -> List[QuotaBucket]: + """Call ``POST /v1internal:retrieveUserQuota`` and parse ``buckets[]``.""" + body: Dict[str, Any] = {} + if project_id: + body["project"] = project_id + url = f"{CODE_ASSIST_ENDPOINT}/v1internal:retrieveUserQuota" + resp = _post_json(url, body, access_token, user_agent_model=user_agent_model) + raw_buckets = resp.get("buckets") or [] + buckets: List[QuotaBucket] = [] + if not isinstance(raw_buckets, list): + return buckets + for b in raw_buckets: + if not isinstance(b, dict): + continue + buckets.append(QuotaBucket( + model_id=str(b.get("modelId") or ""), + token_type=str(b.get("tokenType") or ""), + remaining_fraction=float(b.get("remainingFraction") or 0.0), + reset_time_iso=str(b.get("resetTime") or ""), + raw=b, + )) + return buckets + + +# ============================================================================= +# Project context resolution +# ============================================================================= + +@dataclass +class ProjectContext: + """Resolved state for a given OAuth session.""" + project_id: str = "" # effective project id sent on requests + managed_project_id: str = "" # Google-assigned project (free tier) + tier_id: str = "" + source: str = "" # "env", "config", "discovered", "onboarded" + + +def resolve_project_context( + access_token: str, + *, + configured_project_id: str = "", + env_project_id: str = "", + user_agent_model: str = "", +) -> ProjectContext: + """Figure out what project id + tier to use for requests. + + Priority: + 1. If configured_project_id or env_project_id is set, use that directly + and short-circuit (no discovery needed). + 2. Otherwise call loadCodeAssist to see what Google says. + 3. If no tier assigned yet, onboard the user (free tier default). + """ + # Short-circuit: caller provided a project id + if configured_project_id: + return ProjectContext( + project_id=configured_project_id, + tier_id=STANDARD_TIER_ID, # assume paid since they specified one + source="config", + ) + if env_project_id: + return ProjectContext( + project_id=env_project_id, + tier_id=STANDARD_TIER_ID, + source="env", + ) + + # Discover via loadCodeAssist + info = load_code_assist(access_token, user_agent_model=user_agent_model) + + effective_project = info.cloudaicompanion_project + tier = info.current_tier_id + + if not tier: + # User hasn't been onboarded — provision them on free tier + onboard_resp = onboard_user( + access_token, + tier_id=FREE_TIER_ID, + project_id="", + user_agent_model=user_agent_model, + ) + # Re-parse from the onboard response + response_body = onboard_resp.get("response") or {} + if isinstance(response_body, dict): + effective_project = ( + effective_project + or str(response_body.get("cloudaicompanionProject") or "") + ) + tier = FREE_TIER_ID + source = "onboarded" + else: + source = "discovered" + + return ProjectContext( + project_id=effective_project, + managed_project_id=effective_project if tier == FREE_TIER_ID else "", + tier_id=tier, + source=source, + ) diff --git a/agent/google_oauth.py b/agent/google_oauth.py new file mode 100644 index 000000000000..4fda090fc66d --- /dev/null +++ b/agent/google_oauth.py @@ -0,0 +1,1048 @@ +"""Google OAuth PKCE flow for the Gemini (google-gemini-cli) inference provider. + +This module implements Authorization Code + PKCE (S256) OAuth against Google's +accounts.google.com endpoints. The resulting access token is used by +``agent.gemini_cloudcode_adapter`` to talk to ``cloudcode-pa.googleapis.com`` +(Google's Code Assist backend that powers the Gemini CLI's free and paid tiers). + +Synthesized from: +- jenslys/opencode-gemini-auth (MIT) — overall flow shape, public OAuth creds, request format +- clawdbot/extensions/google/ — refresh-token rotation, VPC-SC handling reference +- PRs #10176 (@sliverp) and #10779 (@newarthur) — PKCE module structure, cross-process lock + +Storage (``~/.hermes/auth/google_oauth.json``, chmod 0o600): + + { + "refresh": "refreshToken|projectId|managedProjectId", + "access": "...", + "expires": 1744848000000, // unix MILLIseconds + "email": "user@example.com" + } + +The ``refresh`` field packs the refresh_token together with the resolved GCP +project IDs so subsequent sessions don't need to re-discover the project. +This matches opencode-gemini-auth's storage contract exactly. + +The packed format stays parseable even if no project IDs are present — just +a bare refresh_token is treated as "packed with empty IDs". + +Public client credentials +------------------------- +The client_id and client_secret below are Google's PUBLIC desktop OAuth client +for their own open-source gemini-cli. They are baked into every copy of the +gemini-cli npm package and are NOT confidential — desktop OAuth clients have +no secret-keeping requirement (PKCE provides the security). Shipping them here +is consistent with opencode-gemini-auth and the official Google gemini-cli. + +Policy note: Google considers using this OAuth client with third-party software +a policy violation. Users see an upfront warning with ``confirm(default=False)`` +before authorization begins. +""" + +from __future__ import annotations + +import base64 +import contextlib +import hashlib +import http.server +import json +import logging +import os +import secrets +import socket +import stat +import threading +import time +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, Optional, Tuple + +from hermes_constants import get_hermes_home + +logger = logging.getLogger(__name__) + + +# ============================================================================= +# OAuth client credential resolution. +# +# Resolution order: +# 1. HERMES_GEMINI_CLIENT_ID / HERMES_GEMINI_CLIENT_SECRET env vars (power users) +# 2. Shipped defaults — Google's public gemini-cli desktop OAuth client +# (baked into every copy of Google's open-source gemini-cli; NOT +# confidential — desktop OAuth clients use PKCE, not client_secret, for +# security). Using these matches opencode-gemini-auth behavior. +# 3. Fallback: scrape from a locally installed gemini-cli binary (helps forks +# that deliberately wipe the shipped defaults). +# 4. Fail with a helpful error. +# ============================================================================= + +ENV_CLIENT_ID = "HERMES_GEMINI_CLIENT_ID" +ENV_CLIENT_SECRET = "HERMES_GEMINI_CLIENT_SECRET" + +# Public gemini-cli desktop OAuth client (shipped in Google's open-source +# gemini-cli MIT repo). Composed piecewise to keep the constants readable and +# to pair each piece with an explicit comment about why it is non-confidential. +# See: https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/code_assist/oauth2.ts +_PUBLIC_CLIENT_ID_PROJECT_NUM = "681255809395" +_PUBLIC_CLIENT_ID_HASH = "oo8ft2oprdrnp9e3aqf6av3hmdib135j" +_PUBLIC_CLIENT_SECRET_SUFFIX = "4uHgMPm-1o7Sk-geV6Cu5clXFsxl" + +_DEFAULT_CLIENT_ID = ( + f"{_PUBLIC_CLIENT_ID_PROJECT_NUM}-{_PUBLIC_CLIENT_ID_HASH}" + ".apps.googleusercontent.com" +) +_DEFAULT_CLIENT_SECRET = f"GOCSPX-{_PUBLIC_CLIENT_SECRET_SUFFIX}" + +# Regex patterns for fallback scraping from an installed gemini-cli. +import re as _re +_CLIENT_ID_PATTERN = _re.compile( + r"OAUTH_CLIENT_ID\s*=\s*['\"]([0-9]+-[a-z0-9]+\.apps\.googleusercontent\.com)['\"]" +) +_CLIENT_SECRET_PATTERN = _re.compile( + r"OAUTH_CLIENT_SECRET\s*=\s*['\"](GOCSPX-[A-Za-z0-9_-]+)['\"]" +) +_CLIENT_ID_SHAPE = _re.compile(r"([0-9]{8,}-[a-z0-9]{20,}\.apps\.googleusercontent\.com)") +_CLIENT_SECRET_SHAPE = _re.compile(r"(GOCSPX-[A-Za-z0-9_-]{20,})") + + +# ============================================================================= +# Endpoints & constants +# ============================================================================= + +AUTH_ENDPOINT = "https://accounts.google.com/o/oauth2/v2/auth" +TOKEN_ENDPOINT = "https://oauth2.googleapis.com/token" +USERINFO_ENDPOINT = "https://www.googleapis.com/oauth2/v1/userinfo" + +OAUTH_SCOPES = ( + "https://www.googleapis.com/auth/cloud-platform " + "https://www.googleapis.com/auth/userinfo.email " + "https://www.googleapis.com/auth/userinfo.profile" +) + +DEFAULT_REDIRECT_PORT = 8085 +REDIRECT_HOST = "127.0.0.1" +CALLBACK_PATH = "/oauth2callback" + +# 60-second clock skew buffer (matches opencode-gemini-auth). +REFRESH_SKEW_SECONDS = 60 + +TOKEN_REQUEST_TIMEOUT_SECONDS = 20.0 +CALLBACK_WAIT_SECONDS = 300 +LOCK_TIMEOUT_SECONDS = 30.0 + +# Headless env detection +_HEADLESS_ENV_VARS = ("SSH_CONNECTION", "SSH_CLIENT", "SSH_TTY", "HERMES_HEADLESS") + + +# ============================================================================= +# Error type +# ============================================================================= + +class GoogleOAuthError(RuntimeError): + """Raised for any failure in the Google OAuth flow.""" + + def __init__(self, message: str, *, code: str = "google_oauth_error") -> None: + super().__init__(message) + self.code = code + + +# ============================================================================= +# File paths & cross-process locking +# ============================================================================= + +def _credentials_path() -> Path: + return get_hermes_home() / "auth" / "google_oauth.json" + + +def _lock_path() -> Path: + return _credentials_path().with_suffix(".json.lock") + + +_lock_state = threading.local() + + +@contextlib.contextmanager +def _credentials_lock(timeout_seconds: float = LOCK_TIMEOUT_SECONDS): + """Cross-process lock around the credentials file (fcntl POSIX / msvcrt Windows).""" + depth = getattr(_lock_state, "depth", 0) + if depth > 0: + _lock_state.depth = depth + 1 + try: + yield + finally: + _lock_state.depth -= 1 + return + + lock_file_path = _lock_path() + lock_file_path.parent.mkdir(parents=True, exist_ok=True) + fd = os.open(str(lock_file_path), os.O_CREAT | os.O_RDWR, 0o600) + acquired = False + try: + try: + import fcntl + except ImportError: + fcntl = None + + if fcntl is not None: + deadline = time.monotonic() + max(0.0, float(timeout_seconds)) + while True: + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + acquired = True + break + except BlockingIOError: + if time.monotonic() >= deadline: + raise TimeoutError( + f"Timed out acquiring Google OAuth credentials lock at {lock_file_path}." + ) + time.sleep(0.05) + else: + try: + import msvcrt # type: ignore[import-not-found] + + deadline = time.monotonic() + max(0.0, float(timeout_seconds)) + while True: + try: + msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) + acquired = True + break + except OSError: + if time.monotonic() >= deadline: + raise TimeoutError( + f"Timed out acquiring Google OAuth credentials lock at {lock_file_path}." + ) + time.sleep(0.05) + except ImportError: + acquired = True + + _lock_state.depth = 1 + yield + finally: + try: + if acquired: + try: + import fcntl + + fcntl.flock(fd, fcntl.LOCK_UN) + except ImportError: + try: + import msvcrt # type: ignore[import-not-found] + + try: + msvcrt.locking(fd, msvcrt.LK_UNLCK, 1) + except OSError: + pass + except ImportError: + pass + finally: + os.close(fd) + _lock_state.depth = 0 + + +# ============================================================================= +# Client ID resolution +# ============================================================================= + +_scraped_creds_cache: Dict[str, str] = {} + + +def _locate_gemini_cli_oauth_js() -> Optional[Path]: + """Walk the user's gemini binary install to find its oauth2.js. + + Returns None if gemini isn't installed. Supports both the npm install + (``node_modules/@google/gemini-cli-core/dist/**/code_assist/oauth2.js``) + and the Homebrew ``bundle/`` layout. + """ + import shutil + + gemini = shutil.which("gemini") + if not gemini: + return None + + try: + real = Path(gemini).resolve() + except OSError: + return None + + # Walk up from the binary to find npm install root + search_dirs: list[Path] = [] + cur = real.parent + for _ in range(8): # don't walk too far + search_dirs.append(cur) + if (cur / "node_modules").exists(): + search_dirs.append(cur / "node_modules" / "@google" / "gemini-cli-core") + break + if cur.parent == cur: + break + cur = cur.parent + + for root in search_dirs: + if not root.exists(): + continue + # Common known paths + candidates = [ + root / "dist" / "src" / "code_assist" / "oauth2.js", + root / "dist" / "code_assist" / "oauth2.js", + root / "src" / "code_assist" / "oauth2.js", + ] + for c in candidates: + if c.exists(): + return c + # Recursive fallback: look for oauth2.js within 10 dirs deep + try: + for path in root.rglob("oauth2.js"): + return path + except (OSError, ValueError): + continue + + return None + + +def _scrape_client_credentials() -> Tuple[str, str]: + """Extract client_id + client_secret from the local gemini-cli install.""" + if _scraped_creds_cache.get("resolved"): + return _scraped_creds_cache.get("client_id", ""), _scraped_creds_cache.get("client_secret", "") + + oauth_js = _locate_gemini_cli_oauth_js() + if oauth_js is None: + _scraped_creds_cache["resolved"] = "1" # Don't retry on every call + return "", "" + + try: + content = oauth_js.read_text(encoding="utf-8", errors="replace") + except OSError as exc: + logger.debug("Failed to read oauth2.js at %s: %s", oauth_js, exc) + _scraped_creds_cache["resolved"] = "1" + return "", "" + + # Precise pattern first, then fallback shape match + cid_match = _CLIENT_ID_PATTERN.search(content) or _CLIENT_ID_SHAPE.search(content) + cs_match = _CLIENT_SECRET_PATTERN.search(content) or _CLIENT_SECRET_SHAPE.search(content) + + client_id = cid_match.group(1) if cid_match else "" + client_secret = cs_match.group(1) if cs_match else "" + + _scraped_creds_cache["client_id"] = client_id + _scraped_creds_cache["client_secret"] = client_secret + _scraped_creds_cache["resolved"] = "1" + + if client_id: + logger.info("Scraped Gemini OAuth client from %s", oauth_js) + + return client_id, client_secret + + +def _get_client_id() -> str: + env_val = (os.getenv(ENV_CLIENT_ID) or "").strip() + if env_val: + return env_val + if _DEFAULT_CLIENT_ID: + return _DEFAULT_CLIENT_ID + scraped, _ = _scrape_client_credentials() + return scraped + + +def _get_client_secret() -> str: + env_val = (os.getenv(ENV_CLIENT_SECRET) or "").strip() + if env_val: + return env_val + if _DEFAULT_CLIENT_SECRET: + return _DEFAULT_CLIENT_SECRET + _, scraped = _scrape_client_credentials() + return scraped + + +def _require_client_id() -> str: + cid = _get_client_id() + if not cid: + raise GoogleOAuthError( + "Google OAuth client ID is not available.\n" + "Hermes looks for a locally installed gemini-cli to source the OAuth client. " + "Either:\n" + " 1. Install it: npm install -g @google/gemini-cli (or brew install gemini-cli)\n" + " 2. Set HERMES_GEMINI_CLIENT_ID and HERMES_GEMINI_CLIENT_SECRET in ~/.hermes/.env\n" + "\n" + "Register a Desktop OAuth client at:\n" + " https://console.cloud.google.com/apis/credentials\n" + "(enable the Generative Language API on the project).", + code="google_oauth_client_id_missing", + ) + return cid + + +# ============================================================================= +# PKCE +# ============================================================================= + +def _generate_pkce_pair() -> Tuple[str, str]: + """Generate a (verifier, challenge) pair using S256.""" + verifier = secrets.token_urlsafe(64) + digest = hashlib.sha256(verifier.encode("ascii")).digest() + challenge = base64.urlsafe_b64encode(digest).rstrip(b"=").decode("ascii") + return verifier, challenge + + +# ============================================================================= +# Packed refresh format: refresh_token[|project_id[|managed_project_id]] +# ============================================================================= + +@dataclass +class RefreshParts: + refresh_token: str + project_id: str = "" + managed_project_id: str = "" + + @classmethod + def parse(cls, packed: str) -> "RefreshParts": + if not packed: + return cls(refresh_token="") + parts = packed.split("|", 2) + return cls( + refresh_token=parts[0], + project_id=parts[1] if len(parts) > 1 else "", + managed_project_id=parts[2] if len(parts) > 2 else "", + ) + + def format(self) -> str: + if not self.refresh_token: + return "" + if not self.project_id and not self.managed_project_id: + return self.refresh_token + return f"{self.refresh_token}|{self.project_id}|{self.managed_project_id}" + + +# ============================================================================= +# Credentials (dataclass wrapping the on-disk format) +# ============================================================================= + +@dataclass +class GoogleCredentials: + access_token: str + refresh_token: str + expires_ms: int # unix milliseconds + email: str = "" + project_id: str = "" + managed_project_id: str = "" + + def to_dict(self) -> Dict[str, Any]: + return { + "refresh": RefreshParts( + refresh_token=self.refresh_token, + project_id=self.project_id, + managed_project_id=self.managed_project_id, + ).format(), + "access": self.access_token, + "expires": int(self.expires_ms), + "email": self.email, + } + + @classmethod + def from_dict(cls, data: Dict[str, Any]) -> "GoogleCredentials": + refresh_packed = str(data.get("refresh", "") or "") + parts = RefreshParts.parse(refresh_packed) + return cls( + access_token=str(data.get("access", "") or ""), + refresh_token=parts.refresh_token, + expires_ms=int(data.get("expires", 0) or 0), + email=str(data.get("email", "") or ""), + project_id=parts.project_id, + managed_project_id=parts.managed_project_id, + ) + + def expires_unix_seconds(self) -> float: + return self.expires_ms / 1000.0 + + def access_token_expired(self, skew_seconds: int = REFRESH_SKEW_SECONDS) -> bool: + if not self.access_token or not self.expires_ms: + return True + return (time.time() + max(0, skew_seconds)) * 1000 >= self.expires_ms + + +# ============================================================================= +# Credential I/O (atomic + locked) +# ============================================================================= + +def load_credentials() -> Optional[GoogleCredentials]: + """Load credentials from disk. Returns None if missing or corrupt.""" + path = _credentials_path() + if not path.exists(): + return None + try: + with _credentials_lock(): + raw = path.read_text(encoding="utf-8") + data = json.loads(raw) + except (json.JSONDecodeError, OSError, IOError) as exc: + logger.warning("Failed to read Google OAuth credentials at %s: %s", path, exc) + return None + if not isinstance(data, dict): + return None + creds = GoogleCredentials.from_dict(data) + if not creds.access_token: + return None + return creds + + +def save_credentials(creds: GoogleCredentials) -> Path: + """Atomically write creds to disk with 0o600 permissions.""" + path = _credentials_path() + path.parent.mkdir(parents=True, exist_ok=True) + payload = json.dumps(creds.to_dict(), indent=2, sort_keys=True) + "\n" + + with _credentials_lock(): + tmp_path = path.with_suffix(f".tmp.{os.getpid()}.{secrets.token_hex(4)}") + try: + with open(tmp_path, "w", encoding="utf-8") as fh: + fh.write(payload) + fh.flush() + os.fsync(fh.fileno()) + os.chmod(tmp_path, stat.S_IRUSR | stat.S_IWUSR) + os.replace(tmp_path, path) + finally: + try: + if tmp_path.exists(): + tmp_path.unlink() + except OSError: + pass + return path + + +def clear_credentials() -> None: + """Remove the creds file. Idempotent.""" + path = _credentials_path() + with _credentials_lock(): + try: + path.unlink() + except FileNotFoundError: + pass + except OSError as exc: + logger.warning("Failed to remove Google OAuth credentials at %s: %s", path, exc) + + +# ============================================================================= +# HTTP helpers +# ============================================================================= + +def _post_form(url: str, data: Dict[str, str], timeout: float) -> Dict[str, Any]: + """POST x-www-form-urlencoded and return parsed JSON response.""" + body = urllib.parse.urlencode(data).encode("ascii") + request = urllib.request.Request( + url, + data=body, + method="POST", + headers={ + "Content-Type": "application/x-www-form-urlencoded", + "Accept": "application/json", + }, + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + raw = response.read().decode("utf-8", errors="replace") + return json.loads(raw) + except urllib.error.HTTPError as exc: + detail = "" + try: + detail = exc.read().decode("utf-8", errors="replace") + except Exception: + pass + # Detect invalid_grant to signal credential revocation + code = "google_oauth_token_http_error" + if "invalid_grant" in detail.lower(): + code = "google_oauth_invalid_grant" + raise GoogleOAuthError( + f"Google OAuth token endpoint returned HTTP {exc.code}: {detail or exc.reason}", + code=code, + ) from exc + except urllib.error.URLError as exc: + raise GoogleOAuthError( + f"Google OAuth token request failed: {exc}", + code="google_oauth_token_network_error", + ) from exc + + +def exchange_code( + code: str, + verifier: str, + redirect_uri: str, + *, + client_id: Optional[str] = None, + client_secret: Optional[str] = None, + timeout: float = TOKEN_REQUEST_TIMEOUT_SECONDS, +) -> Dict[str, Any]: + """Exchange authorization code for access + refresh tokens.""" + cid = client_id if client_id is not None else _get_client_id() + csecret = client_secret if client_secret is not None else _get_client_secret() + data = { + "grant_type": "authorization_code", + "code": code, + "code_verifier": verifier, + "client_id": cid, + "redirect_uri": redirect_uri, + } + if csecret: + data["client_secret"] = csecret + return _post_form(TOKEN_ENDPOINT, data, timeout) + + +def refresh_access_token( + refresh_token: str, + *, + client_id: Optional[str] = None, + client_secret: Optional[str] = None, + timeout: float = TOKEN_REQUEST_TIMEOUT_SECONDS, +) -> Dict[str, Any]: + """Refresh the access token.""" + if not refresh_token: + raise GoogleOAuthError( + "Cannot refresh: refresh_token is empty. Re-run OAuth login.", + code="google_oauth_refresh_token_missing", + ) + cid = client_id if client_id is not None else _get_client_id() + csecret = client_secret if client_secret is not None else _get_client_secret() + data = { + "grant_type": "refresh_token", + "refresh_token": refresh_token, + "client_id": cid, + } + if csecret: + data["client_secret"] = csecret + return _post_form(TOKEN_ENDPOINT, data, timeout) + + +def _fetch_user_email(access_token: str, timeout: float = TOKEN_REQUEST_TIMEOUT_SECONDS) -> str: + """Best-effort userinfo fetch for display. Failures return empty string.""" + try: + request = urllib.request.Request( + USERINFO_ENDPOINT + "?alt=json", + headers={"Authorization": f"Bearer {access_token}"}, + ) + with urllib.request.urlopen(request, timeout=timeout) as response: + raw = response.read().decode("utf-8", errors="replace") + data = json.loads(raw) + return str(data.get("email", "") or "") + except Exception as exc: + logger.debug("Userinfo fetch failed (non-fatal): %s", exc) + return "" + + +# ============================================================================= +# In-flight refresh deduplication +# ============================================================================= + +_refresh_inflight: Dict[str, threading.Event] = {} +_refresh_inflight_lock = threading.Lock() + + +def get_valid_access_token(*, force_refresh: bool = False) -> str: + """Load creds, refreshing if near expiry, and return a valid bearer token. + + Dedupes concurrent refreshes by refresh_token. On ``invalid_grant``, the + credential file is wiped and a ``google_oauth_invalid_grant`` error is raised + (caller is expected to trigger a re-login flow). + """ + creds = load_credentials() + if creds is None: + raise GoogleOAuthError( + "No Google OAuth credentials found. Run `hermes login --provider google-gemini-cli` first.", + code="google_oauth_not_logged_in", + ) + + if not force_refresh and not creds.access_token_expired(): + return creds.access_token + + # Dedupe concurrent refreshes by refresh_token + rt = creds.refresh_token + with _refresh_inflight_lock: + event = _refresh_inflight.get(rt) + if event is None: + event = threading.Event() + _refresh_inflight[rt] = event + owner = True + else: + owner = False + + if not owner: + # Another thread is refreshing — wait, then re-read from disk. + event.wait(timeout=LOCK_TIMEOUT_SECONDS) + fresh = load_credentials() + if fresh is not None and not fresh.access_token_expired(): + return fresh.access_token + # Fall through to do our own refresh if the other attempt failed + + try: + try: + resp = refresh_access_token(rt) + except GoogleOAuthError as exc: + if exc.code == "google_oauth_invalid_grant": + logger.warning( + "Google OAuth refresh token invalid (revoked/expired). " + "Clearing credentials at %s — user must re-login.", + _credentials_path(), + ) + clear_credentials() + raise + + new_access = str(resp.get("access_token", "") or "").strip() + if not new_access: + raise GoogleOAuthError( + "Refresh response did not include an access_token.", + code="google_oauth_refresh_empty", + ) + # Google sometimes rotates refresh_token; preserve existing if omitted. + new_refresh = str(resp.get("refresh_token", "") or "").strip() or creds.refresh_token + expires_in = int(resp.get("expires_in", 0) or 0) + + creds.access_token = new_access + creds.refresh_token = new_refresh + creds.expires_ms = int((time.time() + max(60, expires_in)) * 1000) + save_credentials(creds) + return creds.access_token + finally: + if owner: + with _refresh_inflight_lock: + _refresh_inflight.pop(rt, None) + event.set() + + +# ============================================================================= +# Update project IDs on stored creds +# ============================================================================= + +def update_project_ids(project_id: str = "", managed_project_id: str = "") -> None: + """Persist resolved/discovered project IDs back into the credential file.""" + creds = load_credentials() + if creds is None: + return + if project_id: + creds.project_id = project_id + if managed_project_id: + creds.managed_project_id = managed_project_id + save_credentials(creds) + + +# ============================================================================= +# Callback server +# ============================================================================= + +class _OAuthCallbackHandler(http.server.BaseHTTPRequestHandler): + expected_state: str = "" + captured_code: Optional[str] = None + captured_error: Optional[str] = None + ready: Optional[threading.Event] = None + + def log_message(self, format: str, *args: Any) -> None: # noqa: A002, N802 + logger.debug("OAuth callback: " + format, *args) + + def do_GET(self) -> None: # noqa: N802 + parsed = urllib.parse.urlparse(self.path) + if parsed.path != CALLBACK_PATH: + self.send_response(404) + self.end_headers() + return + + params = urllib.parse.parse_qs(parsed.query) + state = (params.get("state") or [""])[0] + error = (params.get("error") or [""])[0] + code = (params.get("code") or [""])[0] + + if state != type(self).expected_state: + type(self).captured_error = "state_mismatch" + self._respond_html(400, _ERROR_PAGE.format(message="State mismatch — aborting for safety.")) + elif error: + type(self).captured_error = error + # Simple HTML-escape of the error value + safe_err = ( + str(error) + .replace("&", "&") + .replace("<", "<") + .replace(">", ">") + ) + self._respond_html(400, _ERROR_PAGE.format(message=f"Authorization denied: {safe_err}")) + elif code: + type(self).captured_code = code + self._respond_html(200, _SUCCESS_PAGE) + else: + type(self).captured_error = "no_code" + self._respond_html(400, _ERROR_PAGE.format(message="Callback received no authorization code.")) + + if type(self).ready is not None: + type(self).ready.set() + + def _respond_html(self, status: int, body: str) -> None: + payload = body.encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + +_SUCCESS_PAGE = """ +Hermes — signed in + +

Signed in to Google.

+

You can close this tab and return to your terminal.

+""" + +_ERROR_PAGE = """ +Hermes — sign-in failed + +

Sign-in failed

{message}

+

Return to your terminal — Hermes will walk you through a manual paste fallback.

+""" + + +def _bind_callback_server(preferred_port: int = DEFAULT_REDIRECT_PORT) -> Tuple[http.server.HTTPServer, int]: + try: + server = http.server.HTTPServer((REDIRECT_HOST, preferred_port), _OAuthCallbackHandler) + return server, preferred_port + except OSError as exc: + logger.info( + "Preferred OAuth callback port %d unavailable (%s); requesting ephemeral port", + preferred_port, exc, + ) + server = http.server.HTTPServer((REDIRECT_HOST, 0), _OAuthCallbackHandler) + return server, server.server_address[1] + + +def _is_headless() -> bool: + return any(os.getenv(k) for k in _HEADLESS_ENV_VARS) + + +# ============================================================================= +# Main login flow +# ============================================================================= + +def start_oauth_flow( + *, + force_relogin: bool = False, + open_browser: bool = True, + callback_wait_seconds: float = CALLBACK_WAIT_SECONDS, + project_id: str = "", +) -> GoogleCredentials: + """Run the interactive browser OAuth flow and persist credentials. + + Args: + force_relogin: If False and valid creds already exist, return them. + open_browser: If False, skip webbrowser.open and print the URL only. + callback_wait_seconds: Max seconds to wait for the browser callback. + project_id: Initial GCP project ID to bake into the stored creds. + Can be discovered/updated later via update_project_ids(). + """ + if not force_relogin: + existing = load_credentials() + if existing and existing.access_token: + logger.info("Google OAuth credentials already present; skipping login.") + return existing + + client_id = _require_client_id() # raises GoogleOAuthError with install hints + client_secret = _get_client_secret() + + verifier, challenge = _generate_pkce_pair() + state = secrets.token_urlsafe(16) + + # If headless, skip the listener and go straight to paste mode + if _is_headless() and open_browser: + logger.info("Headless environment detected; using paste-mode OAuth fallback.") + return _paste_mode_login(verifier, challenge, state, client_id, client_secret, project_id) + + server, port = _bind_callback_server(DEFAULT_REDIRECT_PORT) + redirect_uri = f"http://{REDIRECT_HOST}:{port}{CALLBACK_PATH}" + + _OAuthCallbackHandler.expected_state = state + _OAuthCallbackHandler.captured_code = None + _OAuthCallbackHandler.captured_error = None + ready = threading.Event() + _OAuthCallbackHandler.ready = ready + + params = { + "client_id": client_id, + "redirect_uri": redirect_uri, + "response_type": "code", + "scope": OAUTH_SCOPES, + "state": state, + "code_challenge": challenge, + "code_challenge_method": "S256", + "access_type": "offline", + "prompt": "consent", + } + auth_url = AUTH_ENDPOINT + "?" + urllib.parse.urlencode(params) + "#hermes" + + server_thread = threading.Thread(target=server.serve_forever, daemon=True) + server_thread.start() + + print() + print("Opening your browser to sign in to Google…") + print(f"If it does not open automatically, visit:\n {auth_url}") + print() + + if open_browser: + try: + import webbrowser + + webbrowser.open(auth_url, new=1, autoraise=True) + except Exception as exc: + logger.debug("webbrowser.open failed: %s", exc) + + code: Optional[str] = None + try: + if ready.wait(timeout=callback_wait_seconds): + code = _OAuthCallbackHandler.captured_code + error = _OAuthCallbackHandler.captured_error + if error: + raise GoogleOAuthError( + f"Authorization failed: {error}", + code="google_oauth_authorization_failed", + ) + else: + logger.info("Callback server timed out — offering manual paste fallback.") + code = _prompt_paste_fallback() + finally: + try: + server.shutdown() + except Exception: + pass + try: + server.server_close() + except Exception: + pass + server_thread.join(timeout=2.0) + + if not code: + raise GoogleOAuthError( + "No authorization code received. Aborting.", + code="google_oauth_no_code", + ) + + token_resp = exchange_code( + code, verifier, redirect_uri, + client_id=client_id, client_secret=client_secret, + ) + return _persist_token_response(token_resp, project_id=project_id) + + +def _paste_mode_login( + verifier: str, + challenge: str, + state: str, + client_id: str, + client_secret: str, + project_id: str, +) -> GoogleCredentials: + """Run OAuth flow without a local callback server.""" + # Use a placeholder redirect URI; user will paste the full URL back + redirect_uri = f"http://{REDIRECT_HOST}:{DEFAULT_REDIRECT_PORT}{CALLBACK_PATH}" + params = { + "client_id": client_id, + "redirect_uri": redirect_uri, + "response_type": "code", + "scope": OAUTH_SCOPES, + "state": state, + "code_challenge": challenge, + "code_challenge_method": "S256", + "access_type": "offline", + "prompt": "consent", + } + auth_url = AUTH_ENDPOINT + "?" + urllib.parse.urlencode(params) + "#hermes" + + print() + print("Open this URL in a browser on any device:") + print(f" {auth_url}") + print() + print("After signing in, Google will redirect to localhost (which won't load).") + print("Copy the full URL from your browser and paste it below.") + print() + + code = _prompt_paste_fallback() + if not code: + raise GoogleOAuthError("No authorization code provided.", code="google_oauth_no_code") + + token_resp = exchange_code( + code, verifier, redirect_uri, + client_id=client_id, client_secret=client_secret, + ) + return _persist_token_response(token_resp, project_id=project_id) + + +def _prompt_paste_fallback() -> Optional[str]: + print() + print("Paste the full redirect URL Google showed you, OR just the 'code=' parameter value.") + raw = input("Callback URL or code: ").strip() + if not raw: + return None + if raw.startswith("http://") or raw.startswith("https://"): + parsed = urllib.parse.urlparse(raw) + params = urllib.parse.parse_qs(parsed.query) + return (params.get("code") or [""])[0] or None + # Accept a bare query string as well + if raw.startswith("?"): + params = urllib.parse.parse_qs(raw[1:]) + return (params.get("code") or [""])[0] or None + return raw + + +def _persist_token_response( + token_resp: Dict[str, Any], + *, + project_id: str = "", +) -> GoogleCredentials: + access_token = str(token_resp.get("access_token", "") or "").strip() + refresh_token = str(token_resp.get("refresh_token", "") or "").strip() + expires_in = int(token_resp.get("expires_in", 0) or 0) + if not access_token or not refresh_token: + raise GoogleOAuthError( + "Google token response missing access_token or refresh_token.", + code="google_oauth_incomplete_token_response", + ) + creds = GoogleCredentials( + access_token=access_token, + refresh_token=refresh_token, + expires_ms=int((time.time() + max(60, expires_in)) * 1000), + email=_fetch_user_email(access_token), + project_id=project_id, + managed_project_id="", + ) + save_credentials(creds) + logger.info("Google OAuth credentials saved to %s", _credentials_path()) + return creds + + +# ============================================================================= +# Pool-compatible variant +# ============================================================================= + +def run_gemini_oauth_login_pure() -> Dict[str, Any]: + """Run the login flow and return a dict matching the credential pool shape.""" + creds = start_oauth_flow(force_relogin=True) + return { + "access_token": creds.access_token, + "refresh_token": creds.refresh_token, + "expires_at_ms": creds.expires_ms, + "email": creds.email, + "project_id": creds.project_id, + } + + +# ============================================================================= +# Project ID resolution +# ============================================================================= + +def resolve_project_id_from_env() -> str: + """Return a GCP project ID from env vars, in priority order.""" + for var in ( + "HERMES_GEMINI_PROJECT_ID", + "GOOGLE_CLOUD_PROJECT", + "GOOGLE_CLOUD_PROJECT_ID", + ): + val = (os.getenv(var) or "").strip() + if val: + return val + return "" diff --git a/agent/image_gen_provider.py b/agent/image_gen_provider.py new file mode 100644 index 000000000000..47f65c1b3435 --- /dev/null +++ b/agent/image_gen_provider.py @@ -0,0 +1,242 @@ +""" +Image Generation Provider ABC +============================= + +Defines the pluggable-backend interface for image generation. Providers register +instances via ``PluginContext.register_image_gen_provider()``; the active one +(selected via ``image_gen.provider`` in ``config.yaml``) services every +``image_generate`` tool call. + +Providers live in ``/plugins/image_gen//`` (built-in, auto-loaded +as ``kind: backend``) or ``~/.hermes/plugins/image_gen//`` (user, opt-in +via ``plugins.enabled``). + +Response shape +-------------- +All providers return a dict that :func:`success_response` / :func:`error_response` +produce. The tool wrapper JSON-serializes it. Keys: + + success bool + image str | None URL or absolute file path + model str provider-specific model identifier + prompt str echoed prompt + aspect_ratio str "landscape" | "square" | "portrait" + provider str provider name (for diagnostics) + error str only when success=False + error_type str only when success=False +""" + +from __future__ import annotations + +import abc +import base64 +import datetime +import logging +import uuid +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +logger = logging.getLogger(__name__) + + +VALID_ASPECT_RATIOS: Tuple[str, ...] = ("landscape", "square", "portrait") +DEFAULT_ASPECT_RATIO = "landscape" + + +# --------------------------------------------------------------------------- +# ABC +# --------------------------------------------------------------------------- + + +class ImageGenProvider(abc.ABC): + """Abstract base class for an image generation backend. + + Subclasses must implement :meth:`generate`. Everything else has sane + defaults — override only what your provider needs. + """ + + @property + @abc.abstractmethod + def name(self) -> str: + """Stable short identifier used in ``image_gen.provider`` config. + + Lowercase, no spaces. Examples: ``fal``, ``openai``, ``replicate``. + """ + + @property + def display_name(self) -> str: + """Human-readable label shown in ``hermes tools``. Defaults to ``name.title()``.""" + return self.name.title() + + def is_available(self) -> bool: + """Return True when this provider can service calls. + + Typically checks for a required API key. Default: True + (providers with no external dependencies are always available). + """ + return True + + def list_models(self) -> List[Dict[str, Any]]: + """Return catalog entries for ``hermes tools`` model picker. + + Each entry:: + + { + "id": "gpt-image-1.5", # required + "display": "GPT Image 1.5", # optional; defaults to id + "speed": "~10s", # optional + "strengths": "...", # optional + "price": "$...", # optional + } + + Default: empty list (provider has no user-selectable models). + """ + return [] + + def get_setup_schema(self) -> Dict[str, Any]: + """Return provider metadata for the ``hermes tools`` picker. + + Used by ``tools_config.py`` to inject this provider as a row in + the Image Generation provider list. Shape:: + + { + "name": "OpenAI", # picker label + "badge": "paid", # optional short tag + "tag": "One-line description...", # optional subtitle + "env_vars": [ # keys to prompt for + {"key": "OPENAI_API_KEY", + "prompt": "OpenAI API key", + "url": "https://platform.openai.com/api-keys"}, + ], + } + + Default: minimal entry derived from ``display_name``. Override to + expose API key prompts and custom badges. + """ + return { + "name": self.display_name, + "badge": "", + "tag": "", + "env_vars": [], + } + + def default_model(self) -> Optional[str]: + """Return the default model id, or None if not applicable.""" + models = self.list_models() + if models: + return models[0].get("id") + return None + + @abc.abstractmethod + def generate( + self, + prompt: str, + aspect_ratio: str = DEFAULT_ASPECT_RATIO, + **kwargs: Any, + ) -> Dict[str, Any]: + """Generate an image. + + Implementations should return the dict from :func:`success_response` + or :func:`error_response`. ``kwargs`` may contain forward-compat + parameters future versions of the schema will expose — implementations + should ignore unknown keys. + """ + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def resolve_aspect_ratio(value: Optional[str]) -> str: + """Clamp an aspect_ratio value to the valid set, defaulting to landscape. + + Invalid values are coerced rather than rejected so the tool surface is + forgiving of agent mistakes. + """ + if not isinstance(value, str): + return DEFAULT_ASPECT_RATIO + v = value.strip().lower() + if v in VALID_ASPECT_RATIOS: + return v + return DEFAULT_ASPECT_RATIO + + +def _images_cache_dir() -> Path: + """Return ``$HERMES_HOME/cache/images/``, creating parents as needed.""" + from hermes_constants import get_hermes_home + + path = get_hermes_home() / "cache" / "images" + path.mkdir(parents=True, exist_ok=True) + return path + + +def save_b64_image( + b64_data: str, + *, + prefix: str = "image", + extension: str = "png", +) -> Path: + """Decode base64 image data and write it under ``$HERMES_HOME/cache/images/``. + + Returns the absolute :class:`Path` to the saved file. + + Filename format: ``__.``. + """ + raw = base64.b64decode(b64_data) + ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + short = uuid.uuid4().hex[:8] + path = _images_cache_dir() / f"{prefix}_{ts}_{short}.{extension}" + path.write_bytes(raw) + return path + + +def success_response( + *, + image: str, + model: str, + prompt: str, + aspect_ratio: str, + provider: str, + extra: Optional[Dict[str, Any]] = None, +) -> Dict[str, Any]: + """Build a uniform success response dict. + + ``image`` may be an HTTP URL or an absolute filesystem path (for b64 + providers like OpenAI). Callers that need to pass through additional + backend-specific fields can supply ``extra``. + """ + payload: Dict[str, Any] = { + "success": True, + "image": image, + "model": model, + "prompt": prompt, + "aspect_ratio": aspect_ratio, + "provider": provider, + } + if extra: + for k, v in extra.items(): + payload.setdefault(k, v) + return payload + + +def error_response( + *, + error: str, + error_type: str = "provider_error", + provider: str = "", + model: str = "", + prompt: str = "", + aspect_ratio: str = DEFAULT_ASPECT_RATIO, +) -> Dict[str, Any]: + """Build a uniform error response dict.""" + return { + "success": False, + "image": None, + "error": error, + "error_type": error_type, + "model": model, + "prompt": prompt, + "aspect_ratio": aspect_ratio, + "provider": provider, + } diff --git a/agent/image_gen_registry.py b/agent/image_gen_registry.py new file mode 100644 index 000000000000..715133231cb8 --- /dev/null +++ b/agent/image_gen_registry.py @@ -0,0 +1,120 @@ +""" +Image Generation Provider Registry +================================== + +Central map of registered providers. Populated by plugins at import-time via +``PluginContext.register_image_gen_provider()``; consumed by the +``image_generate`` tool to dispatch each call to the active backend. + +Active selection +---------------- +The active provider is chosen by ``image_gen.provider`` in ``config.yaml``. +If unset, :func:`get_active_provider` applies fallback logic: + +1. If exactly one provider is registered, use it. +2. Otherwise if a provider named ``fal`` is registered, use it (legacy + default — matches pre-plugin behavior). +3. Otherwise return ``None`` (the tool surfaces a helpful error pointing + the user at ``hermes tools``). +""" + +from __future__ import annotations + +import logging +import threading +from typing import Dict, List, Optional + +from agent.image_gen_provider import ImageGenProvider + +logger = logging.getLogger(__name__) + + +_providers: Dict[str, ImageGenProvider] = {} +_lock = threading.Lock() + + +def register_provider(provider: ImageGenProvider) -> None: + """Register an image generation provider. + + Re-registration (same ``name``) overwrites the previous entry and logs + a debug message — this makes hot-reload scenarios (tests, dev loops) + behave predictably. + """ + if not isinstance(provider, ImageGenProvider): + raise TypeError( + f"register_provider() expects an ImageGenProvider instance, " + f"got {type(provider).__name__}" + ) + name = provider.name + if not isinstance(name, str) or not name.strip(): + raise ValueError("Image gen provider .name must be a non-empty string") + with _lock: + existing = _providers.get(name) + _providers[name] = provider + if existing is not None: + logger.debug("Image gen provider '%s' re-registered (was %r)", name, type(existing).__name__) + else: + logger.debug("Registered image gen provider '%s' (%s)", name, type(provider).__name__) + + +def list_providers() -> List[ImageGenProvider]: + """Return all registered providers, sorted by name.""" + with _lock: + items = list(_providers.values()) + return sorted(items, key=lambda p: p.name) + + +def get_provider(name: str) -> Optional[ImageGenProvider]: + """Return the provider registered under *name*, or None.""" + if not isinstance(name, str): + return None + with _lock: + return _providers.get(name.strip()) + + +def get_active_provider() -> Optional[ImageGenProvider]: + """Resolve the currently-active provider. + + Reads ``image_gen.provider`` from config.yaml; falls back per the + module docstring. + """ + configured: Optional[str] = None + try: + from hermes_cli.config import load_config + + cfg = load_config() + section = cfg.get("image_gen") if isinstance(cfg, dict) else None + if isinstance(section, dict): + raw = section.get("provider") + if isinstance(raw, str) and raw.strip(): + configured = raw.strip() + except Exception as exc: + logger.debug("Could not read image_gen.provider from config: %s", exc) + + with _lock: + snapshot = dict(_providers) + + if configured: + provider = snapshot.get(configured) + if provider is not None: + return provider + logger.debug( + "image_gen.provider='%s' configured but not registered; falling back", + configured, + ) + + # Fallback: single-provider case + if len(snapshot) == 1: + return next(iter(snapshot.values())) + + # Fallback: prefer legacy FAL for backward compat + if "fal" in snapshot: + return snapshot["fal"] + + return None + + +def _reset_for_tests() -> None: + """Clear the registry. **Test-only.**""" + with _lock: + _providers.clear() diff --git a/agent/insights.py b/agent/insights.py index b15327c825a0..70907b4f3d57 100644 --- a/agent/insights.py +++ b/agent/insights.py @@ -27,7 +27,6 @@ DEFAULT_PRICING, estimate_usage_cost, format_duration_compact, - get_pricing, has_known_pricing, ) @@ -125,6 +124,7 @@ def generate(self, days: int = 30, source: str = None) -> Dict[str, Any]: # Gather raw data sessions = self._get_sessions(cutoff, source) tool_usage = self._get_tool_usage(cutoff, source) + skill_usage = self._get_skill_usage(cutoff, source) message_stats = self._get_message_stats(cutoff, source) if not sessions: @@ -136,6 +136,15 @@ def generate(self, days: int = 30, source: str = None) -> Dict[str, Any]: "models": [], "platforms": [], "tools": [], + "skills": { + "summary": { + "total_skill_loads": 0, + "total_skill_edits": 0, + "total_skill_actions": 0, + "distinct_skills_used": 0, + }, + "top_skills": [], + }, "activity": {}, "top_sessions": [], } @@ -145,6 +154,7 @@ def generate(self, days: int = 30, source: str = None) -> Dict[str, Any]: models = self._compute_model_breakdown(sessions) platforms = self._compute_platform_breakdown(sessions) tools = self._compute_tool_breakdown(tool_usage) + skills = self._compute_skill_breakdown(skill_usage) activity = self._compute_activity_patterns(sessions) top_sessions = self._compute_top_sessions(sessions) @@ -157,6 +167,7 @@ def generate(self, days: int = 30, source: str = None) -> Dict[str, Any]: "models": models, "platforms": platforms, "tools": tools, + "skills": skills, "activity": activity, "top_sessions": top_sessions, } @@ -285,6 +296,82 @@ def _get_tool_usage(self, cutoff: float, source: str = None) -> List[Dict]: for name, count in tool_counts.most_common() ] + def _get_skill_usage(self, cutoff: float, source: str = None) -> List[Dict]: + """Extract per-skill usage from assistant tool calls.""" + skill_counts: Dict[str, Dict[str, Any]] = {} + + if source: + cursor = self._conn.execute( + """SELECT m.tool_calls, m.timestamp + FROM messages m + JOIN sessions s ON s.id = m.session_id + WHERE s.started_at >= ? AND s.source = ? + AND m.role = 'assistant' AND m.tool_calls IS NOT NULL""", + (cutoff, source), + ) + else: + cursor = self._conn.execute( + """SELECT m.tool_calls, m.timestamp + FROM messages m + JOIN sessions s ON s.id = m.session_id + WHERE s.started_at >= ? + AND m.role = 'assistant' AND m.tool_calls IS NOT NULL""", + (cutoff,), + ) + + for row in cursor.fetchall(): + try: + calls = row["tool_calls"] + if isinstance(calls, str): + calls = json.loads(calls) + if not isinstance(calls, list): + continue + except (json.JSONDecodeError, TypeError): + continue + + timestamp = row["timestamp"] + for call in calls: + if not isinstance(call, dict): + continue + func = call.get("function", {}) + tool_name = func.get("name") + if tool_name not in {"skill_view", "skill_manage"}: + continue + + args = func.get("arguments") + if isinstance(args, str): + try: + args = json.loads(args) + except (json.JSONDecodeError, TypeError): + continue + if not isinstance(args, dict): + continue + + skill_name = args.get("name") + if not isinstance(skill_name, str) or not skill_name.strip(): + continue + + entry = skill_counts.setdefault( + skill_name, + { + "skill": skill_name, + "view_count": 0, + "manage_count": 0, + "last_used_at": None, + }, + ) + if tool_name == "skill_view": + entry["view_count"] += 1 + else: + entry["manage_count"] += 1 + + if timestamp is not None and ( + entry["last_used_at"] is None or timestamp > entry["last_used_at"] + ): + entry["last_used_at"] = timestamp + + return list(skill_counts.values()) + def _get_message_stats(self, cutoff: float, source: str = None) -> Dict: """Get aggregate message statistics.""" if source: @@ -476,6 +563,46 @@ def _compute_tool_breakdown(self, tool_usage: List[Dict]) -> List[Dict]: }) return result + def _compute_skill_breakdown(self, skill_usage: List[Dict]) -> Dict[str, Any]: + """Process per-skill usage into summary + ranked list.""" + total_skill_loads = sum(s["view_count"] for s in skill_usage) if skill_usage else 0 + total_skill_edits = sum(s["manage_count"] for s in skill_usage) if skill_usage else 0 + total_skill_actions = total_skill_loads + total_skill_edits + + top_skills = [] + for skill in skill_usage: + total_count = skill["view_count"] + skill["manage_count"] + percentage = (total_count / total_skill_actions * 100) if total_skill_actions else 0 + top_skills.append({ + "skill": skill["skill"], + "view_count": skill["view_count"], + "manage_count": skill["manage_count"], + "total_count": total_count, + "percentage": percentage, + "last_used_at": skill.get("last_used_at"), + }) + + top_skills.sort( + key=lambda s: ( + s["total_count"], + s["view_count"], + s["manage_count"], + s["last_used_at"] or 0, + s["skill"], + ), + reverse=True, + ) + + return { + "summary": { + "total_skill_loads": total_skill_loads, + "total_skill_edits": total_skill_edits, + "total_skill_actions": total_skill_actions, + "distinct_skills_used": len(skill_usage), + }, + "top_skills": top_skills, + } + def _compute_activity_patterns(self, sessions: List[Dict]) -> Dict: """Analyze activity patterns by day of week and hour.""" day_counts = Counter() # 0=Monday ... 6=Sunday @@ -635,13 +762,7 @@ def format_terminal(self, report: Dict) -> str: lines.append(f" Sessions: {o['total_sessions']:<12} Messages: {o['total_messages']:,}") lines.append(f" Tool calls: {o['total_tool_calls']:<12,} User messages: {o['user_messages']:,}") lines.append(f" Input tokens: {o['total_input_tokens']:<12,} Output tokens: {o['total_output_tokens']:,}") - cache_total = o.get("total_cache_read_tokens", 0) + o.get("total_cache_write_tokens", 0) - if cache_total > 0: - lines.append(f" Cache read: {o['total_cache_read_tokens']:<12,} Cache write: {o['total_cache_write_tokens']:,}") - cost_str = f"${o['estimated_cost']:.2f}" - if o.get("models_without_pricing"): - cost_str += " *" - lines.append(f" Total tokens: {o['total_tokens']:<12,} Est. cost: {cost_str}") + lines.append(f" Total tokens: {o['total_tokens']:,}") if o["total_hours"] > 0: lines.append(f" Active time: ~{_format_duration(o['total_hours'] * 3600):<11} Avg session: ~{_format_duration(o['avg_session_duration'])}") lines.append(f" Avg msgs/session: {o['avg_messages_per_session']:.1f}") @@ -651,16 +772,10 @@ def format_terminal(self, report: Dict) -> str: if report["models"]: lines.append(" 🤖 Models Used") lines.append(" " + "─" * 56) - lines.append(f" {'Model':<30} {'Sessions':>8} {'Tokens':>12} {'Cost':>8}") + lines.append(f" {'Model':<30} {'Sessions':>8} {'Tokens':>12}") for m in report["models"]: model_name = m["model"][:28] - if m.get("has_pricing"): - cost_cell = f"${m['cost']:>6.2f}" - else: - cost_cell = " N/A" - lines.append(f" {model_name:<30} {m['sessions']:>8} {m['total_tokens']:>12,} {cost_cell}") - if o.get("models_without_pricing"): - lines.append(" * Cost N/A for custom/self-hosted models") + lines.append(f" {model_name:<30} {m['sessions']:>8} {m['total_tokens']:>12,}") lines.append("") # Platform breakdown @@ -683,6 +798,28 @@ def format_terminal(self, report: Dict) -> str: lines.append(f" ... and {len(report['tools']) - 15} more tools") lines.append("") + # Skill usage + skills = report.get("skills", {}) + top_skills = skills.get("top_skills", []) + if top_skills: + lines.append(" 🧠 Top Skills") + lines.append(" " + "─" * 56) + lines.append(f" {'Skill':<28} {'Loads':>7} {'Edits':>7} {'Last used':>11}") + for skill in top_skills[:10]: + last_used = "—" + if skill.get("last_used_at"): + last_used = datetime.fromtimestamp(skill["last_used_at"]).strftime("%b %d") + lines.append( + f" {skill['skill'][:28]:<28} {skill['view_count']:>7,} {skill['manage_count']:>7,} {last_used:>11}" + ) + summary = skills.get("summary", {}) + lines.append( + f" Distinct skills: {summary.get('distinct_skills_used', 0)} " + f"Loads: {summary.get('total_skill_loads', 0):,} " + f"Edits: {summary.get('total_skill_edits', 0):,}" + ) + lines.append("") + # Activity patterns act = report.get("activity", {}) if act.get("by_day"): @@ -740,15 +877,7 @@ def format_gateway(self, report: Dict) -> str: # Overview lines.append(f"**Sessions:** {o['total_sessions']} | **Messages:** {o['total_messages']:,} | **Tool calls:** {o['total_tool_calls']:,}") - cache_total = o.get("total_cache_read_tokens", 0) + o.get("total_cache_write_tokens", 0) - if cache_total > 0: - lines.append(f"**Tokens:** {o['total_tokens']:,} (in: {o['total_input_tokens']:,} / out: {o['total_output_tokens']:,} / cache: {cache_total:,})") - else: - lines.append(f"**Tokens:** {o['total_tokens']:,} (in: {o['total_input_tokens']:,} / out: {o['total_output_tokens']:,})") - cost_note = "" - if o.get("models_without_pricing"): - cost_note = " _(excludes custom/self-hosted models)_" - lines.append(f"**Est. cost:** ${o['estimated_cost']:.2f}{cost_note}") + lines.append(f"**Tokens:** {o['total_tokens']:,} (in: {o['total_input_tokens']:,} / out: {o['total_output_tokens']:,})") if o["total_hours"] > 0: lines.append(f"**Active time:** ~{_format_duration(o['total_hours'] * 3600)} | **Avg session:** ~{_format_duration(o['avg_session_duration'])}") lines.append("") @@ -757,8 +886,7 @@ def format_gateway(self, report: Dict) -> str: if report["models"]: lines.append("**🤖 Models:**") for m in report["models"][:5]: - cost_str = f"${m['cost']:.2f}" if m.get("has_pricing") else "N/A" - lines.append(f" {m['model'][:25]} — {m['sessions']} sessions, {m['total_tokens']:,} tokens, {cost_str}") + lines.append(f" {m['model'][:25]} — {m['sessions']} sessions, {m['total_tokens']:,} tokens") lines.append("") # Platforms (if multi-platform) @@ -775,6 +903,18 @@ def format_gateway(self, report: Dict) -> str: lines.append(f" {t['tool']} — {t['count']:,} calls ({t['percentage']:.1f}%)") lines.append("") + skills = report.get("skills", {}) + if skills.get("top_skills"): + lines.append("**🧠 Top Skills:**") + for skill in skills["top_skills"][:5]: + suffix = "" + if skill.get("last_used_at"): + suffix = f", last used {datetime.fromtimestamp(skill['last_used_at']).strftime('%b %d')}" + lines.append( + f" {skill['skill']} — {skill['view_count']:,} loads, {skill['manage_count']:,} edits{suffix}" + ) + lines.append("") + # Activity summary act = report.get("activity", {}) if act.get("busiest_day") and act.get("busiest_hour"): diff --git a/agent/memory_manager.py b/agent/memory_manager.py index e6e057048004..2435c3f24839 100644 --- a/agent/memory_manager.py +++ b/agent/memory_manager.py @@ -44,11 +44,22 @@ # --------------------------------------------------------------------------- _FENCE_TAG_RE = re.compile(r'', re.IGNORECASE) +_INTERNAL_CONTEXT_RE = re.compile( + r'<\s*memory-context\s*>[\s\S]*?', + re.IGNORECASE, +) +_INTERNAL_NOTE_RE = re.compile( + r'\[System note:\s*The following is recalled memory context,\s*NOT new user input\.\s*Treat as informational background data\.\]\s*', + re.IGNORECASE, +) def sanitize_context(text: str) -> str: - """Strip fence-escape sequences from provider output.""" - return _FENCE_TAG_RE.sub('', text) + """Strip fence tags, injected context blocks, and system notes from provider output.""" + text = _INTERNAL_CONTEXT_RE.sub('', text) + text = _INTERNAL_NOTE_RE.sub('', text) + text = _FENCE_TAG_RE.sub('', text) + return text def build_memory_context_block(raw_context: str) -> str: diff --git a/agent/model_metadata.py b/agent/model_metadata.py index 8a7e4d6a8b90..0843b62e62bf 100644 --- a/agent/model_metadata.py +++ b/agent/model_metadata.py @@ -4,8 +4,8 @@ and run_agent.py for pre-flight context checks. """ +import ipaddress import logging -import os import re import time from pathlib import Path @@ -15,6 +15,8 @@ import requests import yaml +from utils import base_url_host_matches, base_url_hostname + from hermes_constants import OPENROUTER_MODELS_URL logger = logging.getLogger(__name__) @@ -24,17 +26,22 @@ # are preserved so the full model name reaches cache lookups and server queries. _PROVIDER_PREFIXES: frozenset[str] = frozenset({ "openrouter", "nous", "openai-codex", "copilot", "copilot-acp", - "gemini", "zai", "kimi-coding", "kimi-coding-cn", "minimax", "minimax-cn", "anthropic", "deepseek", + "gemini", "ollama-cloud", "zai", "kimi-coding", "kimi-coding-cn", "stepfun", "minimax", "minimax-cn", "anthropic", "deepseek", "opencode-zen", "opencode-go", "ai-gateway", "kilocode", "alibaba", "qwen-oauth", "xiaomi", + "arcee", "custom", "local", # Common aliases "google", "google-gemini", "google-ai-studio", "glm", "z-ai", "z.ai", "zhipu", "github", "github-copilot", "github-models", "kimi", "moonshot", "kimi-cn", "moonshot-cn", "claude", "deep-seek", - "opencode", "zen", "go", "vercel", "kilo", "dashscope", "aliyun", "qwen", + "ollama", + "stepfun", "opencode", "zen", "go", "vercel", "kilo", "dashscope", "aliyun", "qwen", "mimo", "xiaomi-mimo", + "arcee-ai", "arceeai", + "xai", "x-ai", "x.ai", "grok", + "nvidia", "nim", "nvidia-nim", "nemotron", "qwen-portal", }) @@ -45,6 +52,13 @@ ) +# Tailscale's CGNAT range (RFC 6598). `ipaddress.is_private` excludes this +# block, so without an explicit check Ollama reached over Tailscale (e.g. +# `http://100.77.243.5:11434`) wouldn't be treated as local and its stream +# read / stale timeouts wouldn't get auto-bumped. Built once at import time. +_TAILSCALE_CGNAT = ipaddress.IPv4Network("100.64.0.0/10") + + def _strip_provider_prefix(model: str) -> str: """Strip a recognised provider prefix from a model string. @@ -99,21 +113,32 @@ def _strip_provider_prefix(model: str) -> str: # fuzzy-match collisions (e.g. "anthropic/claude-sonnet-4" is a # substring of "anthropic/claude-sonnet-4.6"). # OpenRouter-prefixed models resolve via OpenRouter live API or models.dev. + "claude-opus-4-7": 1000000, + "claude-opus-4.7": 1000000, "claude-opus-4-6": 1000000, "claude-sonnet-4-6": 1000000, "claude-opus-4.6": 1000000, "claude-sonnet-4.6": 1000000, # Catch-all for older Claude models (must sort after specific entries) "claude": 200000, - # OpenAI + # OpenAI — GPT-5 family (most have 400k; specific overrides first) + # Source: https://developers.openai.com/api/docs/models + # GPT-5.5 (launched Apr 23 2026). Verified via live ChatGPT codex/models + # endpoint: bare slug `gpt-5.5`, no -pro/-mini variants. 400k context on Codex. + "gpt-5.5": 400000, + "gpt-5.4-nano": 400000, # 400k (not 1.05M like full 5.4) + "gpt-5.4-mini": 400000, # 400k (not 1.05M like full 5.4) + "gpt-5.4": 1050000, # GPT-5.4, GPT-5.4 Pro (1.05M context) + "gpt-5.1-chat": 128000, # Chat variant has 128k context + "gpt-5": 400000, # GPT-5.x base, mini, codex variants (400k) "gpt-4.1": 1047576, - "gpt-5": 128000, "gpt-4": 128000, # Google "gemini": 1048576, # Gemma (open models served via AI Studio) + "gemma-4": 256000, # Gemma 4 family + "gemma4": 256000, # Ollama-style naming (e.g. gemma4:31b-cloud) "gemma-4-31b": 256000, - "gemma-4-26b": 256000, "gemma-3": 131072, "gemma": 8192, # fallback for older gemma models # DeepSeek @@ -147,8 +172,12 @@ def _strip_provider_prefix(model: str) -> str: "grok": 131072, # catch-all (grok-beta, unknown grok-*) # Kimi "kimi": 262144, + # Nemotron — NVIDIA's open-weights series (128K context across all sizes) + "nemotron": 131072, # Arcee "trinity": 262144, + # OpenRouter + "elephant": 262144, # Hugging Face Inference Providers — model IDs use org/name format "Qwen/Qwen3.5-397B-A17B": 131072, "Qwen/Qwen3.5-35B-A3B": 131072, @@ -157,12 +186,15 @@ def _strip_provider_prefix(model: str) -> str: "unsloth/Qwen3.6-35B-A3B-GGUF": 262144, "deepseek-ai/DeepSeek-V3.2": 65536, "moonshotai/Kimi-K2.5": 262144, + "moonshotai/Kimi-K2.6": 262144, "moonshotai/Kimi-K2-Thinking": 262144, "MiniMaxAI/MiniMax-M2.5": 204800, - "XiaomiMiMo/MiMo-V2-Flash": 256000, - "mimo-v2-pro": 1000000, - "mimo-v2-omni": 256000, - "mimo-v2-flash": 256000, + "XiaomiMiMo/MiMo-V2-Flash": 262144, + "mimo-v2-pro": 1048576, + "mimo-v2.5-pro": 1048576, + "mimo-v2.5": 1048576, + "mimo-v2-omni": 262144, + "mimo-v2-flash": 262144, "zai-org/GLM-5": 202752, } @@ -177,6 +209,7 @@ def _strip_provider_prefix(model: str) -> str: "max_seq_len", "n_ctx_train", "n_ctx", + "ctx_size", ) _MAX_COMPLETION_KEYS = ( @@ -199,8 +232,15 @@ def _normalize_base_url(base_url: str) -> str: return (base_url or "").strip().rstrip("/") +def _auth_headers(api_key: str = "") -> Dict[str, str]: + token = str(api_key or "").strip() + if not token: + return {} + return {"Authorization": f"Bearer {token}"} + + def _is_openrouter_base_url(base_url: str) -> bool: - return "openrouter.ai" in _normalize_base_url(base_url).lower() + return base_url_host_matches(base_url, "openrouter.ai") def _is_custom_endpoint(base_url: str) -> bool: @@ -213,9 +253,13 @@ def _is_custom_endpoint(base_url: str) -> bool: "chatgpt.com": "openai", "api.anthropic.com": "anthropic", "api.z.ai": "zai", + "open.bigmodel.cn": "zai", "api.moonshot.ai": "kimi-coding", "api.moonshot.cn": "kimi-coding-cn", "api.kimi.com": "kimi-coding", + "api.stepfun.ai": "stepfun", + "api.stepfun.com": "stepfun", + "api.arcee.ai": "arcee", "api.minimax": "minimax", "dashscope.aliyuncs.com": "alibaba", "dashscope-intl.aliyuncs.com": "alibaba", @@ -229,8 +273,10 @@ def _is_custom_endpoint(base_url: str) -> bool: "api.fireworks.ai": "fireworks", "opencode.ai": "opencode-go", "api.x.ai": "xai", + "integrate.api.nvidia.com": "nvidia", "api.xiaomimimo.com": "xiaomi", "xiaomimimo.com": "xiaomi", + "ollama.com": "ollama-cloud", } @@ -257,7 +303,15 @@ def _is_known_provider_base_url(base_url: str) -> bool: def is_local_endpoint(base_url: str) -> bool: - """Return True if base_url points to a local machine (localhost / RFC-1918 / WSL).""" + """Return True if base_url points to a local machine. + + Recognises loopback (``localhost``, ``127.0.0.0/8``, ``::1``), + container-internal DNS names (``host.docker.internal`` et al.), + RFC-1918 private ranges (``10/8``, ``172.16/12``, ``192.168/16``), + link-local, and Tailscale CGNAT (``100.64.0.0/10``). Tailscale CGNAT + is included so remote-but-trusted Ollama boxes reached over a + Tailscale mesh get the same timeout auto-bumps as localhost Ollama. + """ normalized = _normalize_base_url(base_url) if not normalized: return False @@ -272,14 +326,17 @@ def is_local_endpoint(base_url: str) -> bool: # Docker / Podman / Lima internal DNS names (e.g. host.docker.internal) if any(host.endswith(suffix) for suffix in _CONTAINER_LOCAL_SUFFIXES): return True - # RFC-1918 private ranges and link-local - import ipaddress + # RFC-1918 private ranges, link-local, and Tailscale CGNAT try: addr = ipaddress.ip_address(host) - return addr.is_private or addr.is_loopback or addr.is_link_local + if addr.is_private or addr.is_loopback or addr.is_link_local: + return True + if isinstance(addr, ipaddress.IPv4Address) and addr in _TAILSCALE_CGNAT: + return True except ValueError: pass # Bare IP that looks like a private range (e.g. 172.26.x.x for WSL) + # or Tailscale CGNAT (100.64.x.x–100.127.x.x). parts = host.split(".") if len(parts) == 4: try: @@ -290,12 +347,14 @@ def is_local_endpoint(base_url: str) -> bool: return True if first == 192 and second == 168: return True + if first == 100 and 64 <= second <= 127: + return True except ValueError: pass return False -def detect_local_server_type(base_url: str) -> Optional[str]: +def detect_local_server_type(base_url: str, api_key: str = "") -> Optional[str]: """Detect which local server is running at base_url by probing known endpoints. Returns one of: "ollama", "lm-studio", "vllm", "llamacpp", or None. @@ -307,8 +366,10 @@ def detect_local_server_type(base_url: str) -> Optional[str]: if server_url.endswith("/v1"): server_url = server_url[:-3] + headers = _auth_headers(api_key) + try: - with httpx.Client(timeout=2.0) as client: + with httpx.Client(timeout=2.0, headers=headers) as client: # LM Studio exposes /api/v1/models — check first (most specific) try: r = client.get(f"{server_url}/api/v1/models") @@ -495,6 +556,59 @@ def fetch_endpoint_model_metadata( headers = {"Authorization": f"Bearer {api_key}"} if api_key else {} last_error: Optional[Exception] = None + if is_local_endpoint(normalized): + try: + if detect_local_server_type(normalized, api_key=api_key) == "lm-studio": + server_url = normalized[:-3].rstrip("/") if normalized.endswith("/v1") else normalized + response = requests.get( + server_url.rstrip("/") + "/api/v1/models", + headers=headers, + timeout=10, + ) + response.raise_for_status() + payload = response.json() + cache: Dict[str, Dict[str, Any]] = {} + for model in payload.get("models", []): + if not isinstance(model, dict): + continue + model_id = model.get("key") or model.get("id") + if not model_id: + continue + entry: Dict[str, Any] = {"name": model.get("name", model_id)} + + context_length = None + for inst in model.get("loaded_instances", []) or []: + if not isinstance(inst, dict): + continue + cfg = inst.get("config", {}) + ctx = cfg.get("context_length") if isinstance(cfg, dict) else None + if isinstance(ctx, int) and ctx > 0: + context_length = ctx + break + if context_length is None: + context_length = _extract_context_length(model) + if context_length is not None: + entry["context_length"] = context_length + + max_completion_tokens = _extract_max_completion_tokens(model) + if max_completion_tokens is not None: + entry["max_completion_tokens"] = max_completion_tokens + + pricing = _extract_pricing(model) + if pricing: + entry["pricing"] = pricing + + _add_model_aliases(cache, model_id, entry) + alt_id = model.get("id") + if isinstance(alt_id, str) and alt_id and alt_id != model_id: + _add_model_aliases(cache, alt_id, entry) + + _endpoint_model_metadata_cache[normalized] = cache + _endpoint_model_metadata_cache_time[normalized] = time.time() + return cache + except Exception as exc: + last_error = exc + for candidate in candidates: url = candidate.rstrip("/") + "/models" try: @@ -701,7 +815,7 @@ def _model_id_matches(candidate_id: str, lookup_model: str) -> bool: return False -def query_ollama_num_ctx(model: str, base_url: str) -> Optional[int]: +def query_ollama_num_ctx(model: str, base_url: str, api_key: str = "") -> Optional[int]: """Query an Ollama server for the model's context length. Returns the model's maximum context from GGUF metadata via ``/api/show``, @@ -719,14 +833,16 @@ def query_ollama_num_ctx(model: str, base_url: str) -> Optional[int]: server_url = server_url[:-3] try: - server_type = detect_local_server_type(base_url) + server_type = detect_local_server_type(base_url, api_key=api_key) except Exception: return None if server_type != "ollama": return None + headers = _auth_headers(api_key) + try: - with httpx.Client(timeout=3.0) as client: + with httpx.Client(timeout=3.0, headers=headers) as client: resp = client.post(f"{server_url}/api/show", json={"name": bare_model}) if resp.status_code != 200: return None @@ -754,7 +870,7 @@ def query_ollama_num_ctx(model: str, base_url: str) -> Optional[int]: return None -def _query_local_context_length(model: str, base_url: str) -> Optional[int]: +def _query_local_context_length(model: str, base_url: str, api_key: str = "") -> Optional[int]: """Query a local server for the model's context length.""" import httpx @@ -767,13 +883,15 @@ def _query_local_context_length(model: str, base_url: str) -> Optional[int]: if server_url.endswith("/v1"): server_url = server_url[:-3] + headers = _auth_headers(api_key) + try: - server_type = detect_local_server_type(base_url) + server_type = detect_local_server_type(base_url, api_key=api_key) except Exception: server_type = None try: - with httpx.Client(timeout=3.0) as client: + with httpx.Client(timeout=3.0, headers=headers) as client: # Ollama: /api/show returns model details with context info if server_type == "ollama": resp = client.post(f"{server_url}/api/show", json={"name": model}) @@ -990,7 +1108,7 @@ def get_model_context_length( if not _is_known_provider_base_url(base_url): # 3. Try querying local server directly if is_local_endpoint(base_url): - local_ctx = _query_local_context_length(model, base_url) + local_ctx = _query_local_context_length(model, base_url, api_key=api_key) if local_ctx and local_ctx > 0: save_context_length(model, base_url, local_ctx) return local_ctx @@ -1004,12 +1122,26 @@ def get_model_context_length( # 4. Anthropic /v1/models API (only for regular API keys, not OAuth) if provider == "anthropic" or ( - base_url and "api.anthropic.com" in base_url + base_url and base_url_hostname(base_url) == "api.anthropic.com" ): ctx = _query_anthropic_context_length(model, base_url or "https://api.anthropic.com", api_key) if ctx: return ctx + # 4b. AWS Bedrock — use static context length table. + # Bedrock's ListFoundationModels doesn't expose context window sizes, + # so we maintain a curated table in bedrock_adapter.py. + if provider == "bedrock" or ( + base_url + and base_url_hostname(base_url).startswith("bedrock-runtime.") + and base_url_host_matches(base_url, "amazonaws.com") + ): + try: + from agent.bedrock_adapter import get_bedrock_context_length + return get_bedrock_context_length(model) + except ImportError: + pass # boto3 not installed — fall through to generic resolution + # 5. Provider-aware lookups (before generic OpenRouter cache) # These are provider-specific and take priority over the generic OR cache, # since the same model can have different context limits per provider @@ -1050,7 +1182,7 @@ def get_model_context_length( # 9. Query local server as last resort if base_url and is_local_endpoint(base_url): - local_ctx = _query_local_context_length(model, base_url) + local_ctx = _query_local_context_length(model, base_url, api_key=api_key) if local_ctx and local_ctx > 0: save_context_length(model, base_url, local_ctx) return local_ctx diff --git a/agent/models_dev.py b/agent/models_dev.py index 1f8cf90c8be3..236dd582f923 100644 --- a/agent/models_dev.py +++ b/agent/models_dev.py @@ -18,10 +18,8 @@ rather than parsing the raw JSON themselves. """ -import difflib import json import logging -import os import time from dataclasses import dataclass from pathlib import Path @@ -148,6 +146,7 @@ class ProviderInfo: "openai-codex": "openai", "zai": "zai", "kimi-coding": "kimi-for-coding", + "stepfun": "stepfun", "kimi-coding-cn": "kimi-for-coding", "minimax": "minimax", "minimax-cn": "minimax-cn", @@ -171,19 +170,13 @@ class ProviderInfo: "togetherai": "togetherai", "perplexity": "perplexity", "cohere": "cohere", + "ollama-cloud": "ollama-cloud", } # Reverse mapping: models.dev → Hermes (built lazily) _MODELS_DEV_TO_PROVIDER: Optional[Dict[str, str]] = None -def _get_reverse_mapping() -> Dict[str, str]: - """Return models.dev ID → Hermes provider ID mapping.""" - global _MODELS_DEV_TO_PROVIDER - if _MODELS_DEV_TO_PROVIDER is None: - _MODELS_DEV_TO_PROVIDER = {v: k for k, v in PROVIDER_TO_MODELS_DEV.items()} - return _MODELS_DEV_TO_PROVIDER - def _get_cache_path() -> Path: """Return path to disk cache file.""" @@ -425,10 +418,16 @@ def list_provider_models(provider: str) -> List[str]: Returns an empty list if the provider is unknown or has no data. """ + from hermes_cli.models import normalize_provider + provider = normalize_provider(provider) or provider + models = _get_provider_models(provider) if models is None: return [] - return list(models.keys()) + return [ + mid for mid in models.keys() + if not _should_hide_from_provider_catalog(provider, mid) + ] # Patterns that indicate non-agentic or noise models (TTS, embedding, @@ -440,6 +439,43 @@ def list_provider_models(provider: str) -> List[str]: re.IGNORECASE, ) +# Google's live Gemini catalogs currently include a mix of stale slugs and +# Gemma models whose TPM quotas are too small for normal Hermes agent traffic. +# Keep capability metadata available for direct/manual use, but hide these from +# the Gemini model catalogs we surface in setup and model selection. +_GOOGLE_HIDDEN_MODELS = frozenset({ + # Low-TPM Gemma models that trip Google input-token quota walls under + # agent-style traffic despite advertising large context windows. + "gemma-4-31b-it", + "gemma-4-26b-it", + "gemma-4-26b-a4b-it", + "gemma-3-1b", + "gemma-3-1b-it", + "gemma-3-2b", + "gemma-3-2b-it", + "gemma-3-4b", + "gemma-3-4b-it", + "gemma-3-12b", + "gemma-3-12b-it", + "gemma-3-27b", + "gemma-3-27b-it", + # Stale/retired Google slugs that still surface through models.dev-backed + # Gemini selection but 404 on the current Google endpoints. + "gemini-1.5-flash", + "gemini-1.5-pro", + "gemini-1.5-flash-8b", + "gemini-2.0-flash", + "gemini-2.0-flash-lite", +}) + + +def _should_hide_from_provider_catalog(provider: str, model_id: str) -> bool: + provider_lower = (provider or "").strip().lower() + model_lower = (model_id or "").strip().lower() + if provider_lower in {"gemini", "google"} and model_lower in _GOOGLE_HIDDEN_MODELS: + return True + return False + def list_agentic_models(provider: str) -> List[str]: """Return model IDs suitable for agentic use from models.dev. @@ -456,6 +492,8 @@ def list_agentic_models(provider: str) -> List[str]: for mid, entry in models.items(): if not isinstance(entry, dict): continue + if _should_hide_from_provider_catalog(provider, mid): + continue if not entry.get("tool_call", False): continue if _NOISE_PATTERNS.search(mid): @@ -464,93 +502,6 @@ def list_agentic_models(provider: str) -> List[str]: return result -def search_models_dev( - query: str, provider: str = None, limit: int = 5 -) -> List[Dict[str, Any]]: - """Fuzzy search across models.dev catalog. Returns matching model entries. - - Args: - query: Search string to match against model IDs. - provider: Optional Hermes provider ID to restrict search scope. - If None, searches across all providers in PROVIDER_TO_MODELS_DEV. - limit: Maximum number of results to return. - - Returns: - List of dicts, each containing 'provider', 'model_id', and the full - model 'entry' from models.dev. - """ - data = fetch_models_dev() - if not data: - return [] - - # Build list of (provider_id, model_id, entry) candidates - candidates: List[tuple] = [] - - if provider is not None: - # Search only the specified provider - mdev_provider_id = PROVIDER_TO_MODELS_DEV.get(provider) - if not mdev_provider_id: - return [] - provider_data = data.get(mdev_provider_id, {}) - if isinstance(provider_data, dict): - models = provider_data.get("models", {}) - if isinstance(models, dict): - for mid, mdata in models.items(): - candidates.append((provider, mid, mdata)) - else: - # Search across all mapped providers - for hermes_prov, mdev_prov in PROVIDER_TO_MODELS_DEV.items(): - provider_data = data.get(mdev_prov, {}) - if isinstance(provider_data, dict): - models = provider_data.get("models", {}) - if isinstance(models, dict): - for mid, mdata in models.items(): - candidates.append((hermes_prov, mid, mdata)) - - if not candidates: - return [] - - # Use difflib for fuzzy matching — case-insensitive comparison - model_ids_lower = [c[1].lower() for c in candidates] - query_lower = query.lower() - - # First try exact substring matches (more intuitive than pure edit-distance) - substring_matches = [] - for prov, mid, mdata in candidates: - if query_lower in mid.lower(): - substring_matches.append({"provider": prov, "model_id": mid, "entry": mdata}) - - # Then add difflib fuzzy matches for any remaining slots - fuzzy_ids = difflib.get_close_matches( - query_lower, model_ids_lower, n=limit * 2, cutoff=0.4 - ) - - seen_ids: set = set() - results: List[Dict[str, Any]] = [] - - # Prioritize substring matches - for match in substring_matches: - key = (match["provider"], match["model_id"]) - if key not in seen_ids: - seen_ids.add(key) - results.append(match) - if len(results) >= limit: - return results - - # Add fuzzy matches - for fid in fuzzy_ids: - # Find original-case candidates matching this lowered ID - for prov, mid, mdata in candidates: - if mid.lower() == fid: - key = (prov, mid) - if key not in seen_ids: - seen_ids.add(key) - results.append({"provider": prov, "model_id": mid, "entry": mdata}) - if len(results) >= limit: - return results - - return results - # --------------------------------------------------------------------------- # Rich dataclass constructors — parse raw models.dev JSON into dataclasses @@ -677,5 +628,3 @@ def get_model_info( return _parse_model_info(mid, mdata, mdev_id) return None - - diff --git a/agent/nous_rate_guard.py b/agent/nous_rate_guard.py new file mode 100644 index 000000000000..712d8a0f1f4d --- /dev/null +++ b/agent/nous_rate_guard.py @@ -0,0 +1,182 @@ +"""Cross-session rate limit guard for Nous Portal. + +Writes rate limit state to a shared file so all sessions (CLI, gateway, +cron, auxiliary) can check whether Nous Portal is currently rate-limited +before making requests. Prevents retry amplification when RPH is tapped. + +Each 429 from Nous triggers up to 9 API calls per conversation turn +(3 SDK retries x 3 Hermes retries), and every one of those calls counts +against RPH. By recording the rate limit state on first 429 and checking +it before subsequent attempts, we eliminate the amplification effect. +""" + +from __future__ import annotations + +import json +import logging +import os +import tempfile +import time +from typing import Any, Mapping, Optional + +logger = logging.getLogger(__name__) + +_STATE_SUBDIR = "rate_limits" +_STATE_FILENAME = "nous.json" + + +def _state_path() -> str: + """Return the path to the Nous rate limit state file.""" + try: + from hermes_constants import get_hermes_home + base = get_hermes_home() + except ImportError: + base = os.path.join(os.path.expanduser("~"), ".hermes") + return os.path.join(base, _STATE_SUBDIR, _STATE_FILENAME) + + +def _parse_reset_seconds(headers: Optional[Mapping[str, str]]) -> Optional[float]: + """Extract the best available reset-time estimate from response headers. + + Priority: + 1. x-ratelimit-reset-requests-1h (hourly RPH window — most useful) + 2. x-ratelimit-reset-requests (per-minute RPM window) + 3. retry-after (generic HTTP header) + + Returns seconds-from-now, or None if no usable header found. + """ + if not headers: + return None + + lowered = {k.lower(): v for k, v in headers.items()} + + for key in ( + "x-ratelimit-reset-requests-1h", + "x-ratelimit-reset-requests", + "retry-after", + ): + raw = lowered.get(key) + if raw is not None: + try: + val = float(raw) + if val > 0: + return val + except (TypeError, ValueError): + pass + + return None + + +def record_nous_rate_limit( + *, + headers: Optional[Mapping[str, str]] = None, + error_context: Optional[dict[str, Any]] = None, + default_cooldown: float = 300.0, +) -> None: + """Record that Nous Portal is rate-limited. + + Parses the reset time from response headers or error context. + Falls back to ``default_cooldown`` (5 minutes) if no reset info + is available. Writes to a shared file that all sessions can read. + + Args: + headers: HTTP response headers from the 429 error. + error_context: Structured error context from _extract_api_error_context(). + default_cooldown: Fallback cooldown in seconds when no header data. + """ + now = time.time() + reset_at = None + + # Try headers first (most accurate) + header_seconds = _parse_reset_seconds(headers) + if header_seconds is not None: + reset_at = now + header_seconds + + # Try error_context reset_at (from body parsing) + if reset_at is None and isinstance(error_context, dict): + ctx_reset = error_context.get("reset_at") + if isinstance(ctx_reset, (int, float)) and ctx_reset > now: + reset_at = float(ctx_reset) + + # Default cooldown + if reset_at is None: + reset_at = now + default_cooldown + + path = _state_path() + try: + state_dir = os.path.dirname(path) + os.makedirs(state_dir, exist_ok=True) + + state = { + "reset_at": reset_at, + "recorded_at": now, + "reset_seconds": reset_at - now, + } + + # Atomic write: write to temp file + rename + fd, tmp_path = tempfile.mkstemp(dir=state_dir, suffix=".tmp") + try: + with os.fdopen(fd, "w") as f: + json.dump(state, f) + os.replace(tmp_path, path) + except Exception: + # Clean up temp file on failure + try: + os.unlink(tmp_path) + except OSError: + pass + raise + + logger.info( + "Nous rate limit recorded: resets in %.0fs (at %.0f)", + reset_at - now, reset_at, + ) + except Exception as exc: + logger.debug("Failed to write Nous rate limit state: %s", exc) + + +def nous_rate_limit_remaining() -> Optional[float]: + """Check if Nous Portal is currently rate-limited. + + Returns: + Seconds remaining until reset, or None if not rate-limited. + """ + path = _state_path() + try: + with open(path) as f: + state = json.load(f) + reset_at = state.get("reset_at", 0) + remaining = reset_at - time.time() + if remaining > 0: + return remaining + # Expired — clean up + try: + os.unlink(path) + except OSError: + pass + return None + except (FileNotFoundError, json.JSONDecodeError, KeyError, TypeError): + return None + + +def clear_nous_rate_limit() -> None: + """Clear the rate limit state (e.g., after a successful Nous request).""" + try: + os.unlink(_state_path()) + except FileNotFoundError: + pass + except OSError as exc: + logger.debug("Failed to clear Nous rate limit state: %s", exc) + + +def format_remaining(seconds: float) -> str: + """Format seconds remaining into human-readable duration.""" + s = max(0, int(seconds)) + if s < 60: + return f"{s}s" + if s < 3600: + m, sec = divmod(s, 60) + return f"{m}m {sec}s" if sec else f"{m}m" + h, remainder = divmod(s, 3600) + m = remainder // 60 + return f"{h}h {m}m" if m else f"{h}h" diff --git a/agent/prompt_builder.py b/agent/prompt_builder.py index 558a57888047..3a6ec2441519 100644 --- a/agent/prompt_builder.py +++ b/agent/prompt_builder.py @@ -152,7 +152,13 @@ def _strip_yaml_frontmatter(content: str) -> str: "Do NOT save task progress, session outcomes, completed-work logs, or temporary TODO " "state to memory; use session_search to recall those from past transcripts. " "If you've discovered a new way to do something, solved a problem that could be " - "necessary later, save it as a skill with the skill tool." + "necessary later, save it as a skill with the skill tool.\n" + "Write memories as declarative facts, not instructions to yourself. " + "'User prefers concise responses' ✓ — 'Always respond concisely' ✗. " + "'Project uses pytest with xdist' ✓ — 'Run tests with pytest -n 4' ✗. " + "Imperative phrasing gets re-read as a directive in later sessions and can " + "cause repeated work or override the user's current request. Procedures and " + "workflows belong in skills, not memory." ) SESSION_SEARCH_GUIDANCE = ( @@ -295,7 +301,9 @@ def _strip_yaml_frontmatter(content: str) -> str: ), "telegram": ( "You are on a text messaging communication platform, Telegram. " - "Please do not use markdown as it does not render. " + "Standard markdown is automatically converted to Telegram format. " + "Supported: **bold**, *italic*, ~~strikethrough~~, ||spoiler||, " + "`inline code`, ```code blocks```, [links](url), and ## headers. " "You can send media files natively: to deliver a file to the user, " "include MEDIA:/absolute/path/to/file in your response. Images " "(.png, .jpg, .webp) appear as photos, audio (.ogg) sends as voice " @@ -342,7 +350,13 @@ def _strip_yaml_frontmatter(content: str) -> str: ), "cli": ( "You are a CLI AI Agent. Try not to use markdown but simple text " - "renderable inside a terminal." + "renderable inside a terminal. " + "File delivery: there is no attachment channel — the user reads your " + "response directly in their terminal. Do NOT emit MEDIA:/path tags " + "(those are only intercepted on messaging platforms like Telegram, " + "Discord, Slack, etc.; on the CLI they render as literal text). " + "When referring to a file you created or changed, just state its " + "absolute path in plain text; the user can open it from there." ), "sms": ( "You are communicating via SMS. Keep responses concise and use plain text " @@ -356,6 +370,32 @@ def _strip_yaml_frontmatter(content: str) -> str: "MEDIA:/absolute/path/to/file in your response. Images (.jpg, .png, " ".heic) appear as photos and other files arrive as attachments." ), + "mattermost": ( + "You are in a Mattermost workspace communicating with your user. " + "Mattermost renders standard Markdown — headings, bold, italic, code " + "blocks, and tables all work. " + "You can send media files natively: include MEDIA:/absolute/path/to/file " + "in your response. Images (.jpg, .png, .webp) are uploaded as photo " + "attachments, audio and video as file attachments. " + "Image URLs in markdown format ![alt](url) are rendered as inline previews automatically." + ), + "matrix": ( + "You are in a Matrix room communicating with your user. " + "Matrix renders Markdown — bold, italic, code blocks, and links work; " + "the adapter converts your Markdown to HTML for rich display. " + "You can send media files natively: include MEDIA:/absolute/path/to/file " + "in your response. Images (.jpg, .png, .webp) are sent as inline photos, " + "audio (.ogg, .mp3) as voice/audio messages, video (.mp4) inline, " + "and other files as downloadable attachments." + ), + "feishu": ( + "You are in a Feishu (Lark) workspace communicating with your user. " + "Feishu renders Markdown in messages — bold, italic, code blocks, and " + "links are supported. " + "You can send media files natively: include MEDIA:/absolute/path/to/file " + "in your response. Images (.jpg, .png, .webp) are uploaded and displayed " + "inline, audio files as voice messages, and other files as attachments." + ), "weixin": ( "You are on Weixin/WeChat. Markdown formatting is supported, so you may use it when " "it improves readability, but keep the message compact and chat-friendly. You can send media files natively: " @@ -376,6 +416,12 @@ def _strip_yaml_frontmatter(content: str) -> str: "downloaded and sent as native photos. Do NOT tell the user you lack file-sending " "capability — use MEDIA: syntax whenever a file delivery is appropriate." ), + "qqbot": ( + "You are on QQ, a popular Chinese messaging platform. QQ supports markdown formatting " + "and emoji. You can send media files natively: include MEDIA:/absolute/path/to/file in " + "your response. Images are sent as native photos, and other files arrive as downloadable " + "documents." + ), } # --------------------------------------------------------------------------- @@ -605,12 +651,14 @@ def build_skills_system_prompt( or get_session_env("HERMES_SESSION_PLATFORM") or "" ) + disabled = get_disabled_skill_names() cache_key = ( str(skills_dir.resolve()), tuple(str(d) for d in external_dirs), tuple(sorted(str(t) for t in (available_tools or set()))), tuple(sorted(str(ts) for ts in (available_toolsets or set()))), _platform_hint, + tuple(sorted(disabled)), ) with _SKILLS_PROMPT_CACHE_LOCK: cached = _SKILLS_PROMPT_CACHE.get(cache_key) @@ -618,8 +666,6 @@ def build_skills_system_prompt( _SKILLS_PROMPT_CACHE.move_to_end(cache_key) return cached - disabled = get_disabled_skill_names() - # ── Layer 2: disk snapshot ──────────────────────────────────────── snapshot = _load_skills_snapshot(skills_dir) @@ -646,7 +692,7 @@ def build_skills_system_prompt( ): continue skills_by_category.setdefault(category, []).append( - (skill_name, entry.get("description", "")) + (frontmatter_name, entry.get("description", "")) ) category_descriptions = { str(k): str(v) @@ -671,7 +717,7 @@ def build_skills_system_prompt( ): continue skills_by_category.setdefault(entry["category"], []).append( - (skill_name, entry["description"]) + (entry["frontmatter_name"], entry["description"]) ) # Read category-level DESCRIPTION.md files @@ -714,9 +760,10 @@ def build_skills_system_prompt( continue entry = _build_snapshot_entry(skill_file, ext_dir, frontmatter, desc) skill_name = entry["skill_name"] - if skill_name in seen_skill_names: + frontmatter_name = entry["frontmatter_name"] + if frontmatter_name in seen_skill_names: continue - if entry["frontmatter_name"] in disabled or skill_name in disabled: + if frontmatter_name in disabled or skill_name in disabled: continue if not _skill_should_show( extract_skill_conditions(frontmatter), @@ -724,9 +771,9 @@ def build_skills_system_prompt( available_toolsets, ): continue - seen_skill_names.add(skill_name) + seen_skill_names.add(frontmatter_name) skills_by_category.setdefault(entry["category"], []).append( - (skill_name, entry["description"]) + (frontmatter_name, entry["description"]) ) except Exception as e: logger.debug("Error reading external skill %s: %s", skill_file, e) diff --git a/agent/rate_limit_tracker.py b/agent/rate_limit_tracker.py index 73e11522299f..e20c683341b4 100644 --- a/agent/rate_limit_tracker.py +++ b/agent/rate_limit_tracker.py @@ -24,7 +24,7 @@ import time from dataclasses import dataclass, field -from typing import Any, Dict, Mapping, Optional +from typing import Any, Mapping, Optional @dataclass diff --git a/agent/redact.py b/agent/redact.py index 04d35e3c9360..3679b732360c 100644 --- a/agent/redact.py +++ b/agent/redact.py @@ -13,6 +13,48 @@ logger = logging.getLogger(__name__) +# Sensitive query-string parameter names (case-insensitive exact match). +# Ported from nearai/ironclaw#2529 — catches tokens whose values don't match +# any known vendor prefix regex (e.g. opaque tokens, short OAuth codes). +_SENSITIVE_QUERY_PARAMS = frozenset({ + "access_token", + "refresh_token", + "id_token", + "token", + "api_key", + "apikey", + "client_secret", + "password", + "auth", + "jwt", + "session", + "secret", + "key", + "code", # OAuth authorization codes + "signature", # pre-signed URL signatures + "x-amz-signature", +}) + +# Sensitive form-urlencoded / JSON body key names (case-insensitive exact match). +# Exact match, NOT substring — "token_count" and "session_id" must NOT match. +# Ported from nearai/ironclaw#2529. +_SENSITIVE_BODY_KEYS = frozenset({ + "access_token", + "refresh_token", + "id_token", + "token", + "api_key", + "apikey", + "client_secret", + "password", + "auth", + "jwt", + "secret", + "private_key", + "authorization", + "key", +}) + # Snapshot at import time so runtime env mutations (e.g. LLM-generated # `export HERMES_REDACT_SECRETS=false`) cannot disable redaction mid-session. _REDACT_ENABLED = os.getenv("HERMES_REDACT_SECRETS", "").lower() not in ("0", "false", "no", "off") @@ -93,10 +135,45 @@ re.IGNORECASE, ) +# JWT tokens: header.payload[.signature] — always start with "eyJ" (base64 for "{") +# Matches 1-part (header only), 2-part (header.payload), and full 3-part JWTs. +_JWT_RE = re.compile( + r"eyJ[A-Za-z0-9_-]{10,}" # Header (always starts with eyJ) + r"(?:\.[A-Za-z0-9_=-]{4,}){0,2}" # Optional payload and/or signature +) + +# Discord user/role mentions: <@123456789012345678> or <@!123456789012345678> +# Snowflake IDs are 17-20 digit integers that resolve to specific Discord accounts. +_DISCORD_MENTION_RE = re.compile(r"<@!?(\d{17,20})>") + # E.164 phone numbers: +, 7-15 digits # Negative lookahead prevents matching hex strings or identifiers _SIGNAL_PHONE_RE = re.compile(r"(\+[1-9]\d{6,14})(?![A-Za-z0-9])") +# URLs containing query strings — matches `scheme://...?...[# or end]`. +# Used to scan text for URLs whose query params may contain secrets. +# Ported from nearai/ironclaw#2529. +_URL_WITH_QUERY_RE = re.compile( + r"(https?|wss?|ftp)://" # scheme + r"([^\s/?#]+)" # authority (may include userinfo) + r"([^\s?#]*)" # path + r"\?([^\s#]+)" # query (required) + r"(#\S*)?", # optional fragment +) + +# URLs containing userinfo — `scheme://user:password@host` for ANY scheme +# (not just DB protocols already covered by _DB_CONNSTR_RE above). +# Catches things like `https://user:token@api.example.com/v1/foo`. +_URL_USERINFO_RE = re.compile( + r"(https?|wss?|ftp)://([^/\s:@]+):([^/\s@]+)@", +) + +# Form-urlencoded body detection: conservative — only applies when the entire +# text looks like a query string (k=v&k=v pattern with no newlines). +_FORM_BODY_RE = re.compile( + r"^[A-Za-z_][A-Za-z0-9_.-]*=[^&\s]*(?:&[A-Za-z_][A-Za-z0-9_.-]*=[^&\s]*)+$" +) + # Compile known prefix patterns into one alternation _PREFIX_RE = re.compile( r"(? str: return f"{token[:6]}...{token[-4:]}" +def _redact_query_string(query: str) -> str: + """Redact sensitive parameter values in a URL query string. + + Handles `k=v&k=v` format. Sensitive keys (case-insensitive) have values + replaced with `***`. Non-sensitive keys pass through unchanged. + Empty or malformed pairs are preserved as-is. + """ + if not query: + return query + parts = [] + for pair in query.split("&"): + if "=" not in pair: + parts.append(pair) + continue + key, _, value = pair.partition("=") + if key.lower() in _SENSITIVE_QUERY_PARAMS: + parts.append(f"{key}=***") + else: + parts.append(pair) + return "&".join(parts) + + +def _redact_url_query_params(text: str) -> str: + """Scan text for URLs with query strings and redact sensitive params. + + Catches opaque tokens that don't match vendor prefix regexes, e.g. + `https://example.com/cb?code=ABC123&state=xyz` → `...?code=***&state=xyz`. + """ + def _sub(m: re.Match) -> str: + scheme = m.group(1) + authority = m.group(2) + path = m.group(3) + query = _redact_query_string(m.group(4)) + fragment = m.group(5) or "" + return f"{scheme}://{authority}{path}?{query}{fragment}" + return _URL_WITH_QUERY_RE.sub(_sub, text) + + +def _redact_url_userinfo(text: str) -> str: + """Strip `user:password@` from HTTP/WS/FTP URLs. + + DB protocols (postgres, mysql, mongodb, redis, amqp) are handled + separately by `_DB_CONNSTR_RE`. + """ + return _URL_USERINFO_RE.sub( + lambda m: f"{m.group(1)}://{m.group(2)}:***@", + text, + ) + + +def _redact_form_body(text: str) -> str: + """Redact sensitive values in a form-urlencoded body. + + Only applies when the entire input looks like a pure form body + (k=v&k=v with no newlines, no other text). Single-line non-form + text passes through unchanged. This is a conservative pass — the + `_redact_url_query_params` function handles embedded query strings. + """ + if not text or "\n" in text or "&" not in text: + return text + # The body-body form check is strict: only trigger on clean k=v&k=v. + if not _FORM_BODY_RE.match(text.strip()): + return text + return _redact_query_string(text.strip()) + + def redact_sensitive_text(text: str) -> str: """Apply all redaction patterns to a block of text. @@ -159,6 +302,22 @@ def _redact_telegram(m): # Database connection string passwords text = _DB_CONNSTR_RE.sub(lambda m: f"{m.group(1)}***{m.group(3)}", text) + # JWT tokens (eyJ... — base64-encoded JSON headers) + text = _JWT_RE.sub(lambda m: _mask_token(m.group(0)), text) + + # URL userinfo (http(s)://user:pass@host) — redact for non-DB schemes. + # DB schemes are handled above by _DB_CONNSTR_RE. + text = _redact_url_userinfo(text) + + # URL query params containing opaque tokens (?access_token=…&code=…) + text = _redact_url_query_params(text) + + # Form-urlencoded bodies (only triggers on clean k=v&k=v inputs). + text = _redact_form_body(text) + + # Discord user/role mentions (<@snowflake_id>) + text = _DISCORD_MENTION_RE.sub(lambda m: f"<@{'!' if '!' in m.group(0) else ''}***>", text) + # E.164 phone numbers (Signal, WhatsApp) def _redact_phone(m): phone = m.group(1) diff --git a/agent/shell_hooks.py b/agent/shell_hooks.py new file mode 100644 index 000000000000..b579ad5b875f --- /dev/null +++ b/agent/shell_hooks.py @@ -0,0 +1,831 @@ +""" +Shell-script hooks bridge. + +Reads the ``hooks:`` block from ``cli-config.yaml``, prompts the user for +consent on first use of each ``(event, command)`` pair, and registers +callbacks on the existing plugin hook manager so every existing +``invoke_hook()`` site dispatches to the configured shell scripts — with +zero changes to call sites. + +Design notes +------------ +* Python plugins and shell hooks compose naturally: both flow through + :func:`hermes_cli.plugins.invoke_hook` and its aggregators. Python + plugins are registered first (via ``discover_and_load()``) so their + block decisions win ties over shell-hook blocks. +* Subprocess execution uses ``shlex.split(os.path.expanduser(command))`` + with ``shell=False`` — no shell injection footguns. Users that need + pipes/redirection wrap their logic in a script. +* First-use consent is gated by the allowlist under + ``~/.hermes/shell-hooks-allowlist.json``. Non-TTY callers must pass + ``accept_hooks=True`` (resolved from ``--accept-hooks``, + ``HERMES_ACCEPT_HOOKS``, or ``hooks_auto_accept: true`` in config) + for registration to succeed without a prompt. +* Registration is idempotent — safe to invoke from both the CLI entry + point (``hermes_cli/main.py``) and the gateway entry point + (``gateway/run.py``). + +Wire protocol +------------- +**stdin** (JSON, piped to the script):: + + { + "hook_event_name": "pre_tool_call", + "tool_name": "terminal", + "tool_input": {"command": "rm -rf /"}, + "session_id": "sess_abc123", + "cwd": "/home/user/project", + "extra": {...} # event-specific kwargs + } + +**stdout** (JSON, optional — anything else is ignored):: + + # Block a pre_tool_call (either shape accepted; normalised internally): + {"decision": "block", "reason": "Forbidden command"} # Claude-Code-style + {"action": "block", "message": "Forbidden command"} # Hermes-canonical + + # Inject context for pre_llm_call: + {"context": "Today is Friday"} + + # Silent no-op: + +""" + +from __future__ import annotations + +import difflib +import json +import logging +import os +import re +import shlex +import subprocess +import sys +import tempfile +import threading +import time +from contextlib import contextmanager +from dataclasses import dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Callable, Dict, Iterator, List, Optional, Set, Tuple + +try: + import fcntl # POSIX only; Windows falls back to best-effort without flock. +except ImportError: # pragma: no cover + fcntl = None # type: ignore[assignment] + +from hermes_constants import get_hermes_home + +logger = logging.getLogger(__name__) + +DEFAULT_TIMEOUT_SECONDS = 60 +MAX_TIMEOUT_SECONDS = 300 +ALLOWLIST_FILENAME = "shell-hooks-allowlist.json" + +# (event, matcher, command) triples that have been wired to the plugin +# manager in the current process. Matcher is part of the key because +# the same script can legitimately register for different matchers under +# the same event (e.g. one entry per tool the user wants to gate). +# Second registration attempts for the exact same triple become no-ops +# so the CLI and gateway can both call register_from_config() safely. +_registered: Set[Tuple[str, Optional[str], str]] = set() +_registered_lock = threading.Lock() + +# Intra-process lock for allowlist read-modify-write on platforms that +# lack ``fcntl`` (non-POSIX). Kept separate from ``_registered_lock`` +# because ``register_from_config`` already holds ``_registered_lock`` when +# it triggers ``_record_approval`` — reusing it here would self-deadlock +# (``threading.Lock`` is non-reentrant). POSIX callers use the sibling +# ``.lock`` file via ``fcntl.flock`` and bypass this. +_allowlist_write_lock = threading.Lock() + + +@dataclass +class ShellHookSpec: + """Parsed and validated representation of a single ``hooks:`` entry.""" + + event: str + command: str + matcher: Optional[str] = None + timeout: int = DEFAULT_TIMEOUT_SECONDS + compiled_matcher: Optional[re.Pattern] = field(default=None, repr=False) + + def __post_init__(self) -> None: + # Strip whitespace introduced by YAML quirks (e.g. multi-line string + # folding) — a matcher of " terminal" would otherwise silently fail + # to match "terminal" without any diagnostic. + if isinstance(self.matcher, str): + stripped = self.matcher.strip() + self.matcher = stripped if stripped else None + if self.matcher: + try: + self.compiled_matcher = re.compile(self.matcher) + except re.error as exc: + logger.warning( + "shell hook matcher %r is invalid (%s) — treating as " + "literal equality", self.matcher, exc, + ) + self.compiled_matcher = None + + def matches_tool(self, tool_name: Optional[str]) -> bool: + if not self.matcher: + return True + if tool_name is None: + return False + if self.compiled_matcher is not None: + return self.compiled_matcher.fullmatch(tool_name) is not None + # compiled_matcher is None only when the regex failed to compile, + # in which case we already warned and fall back to literal equality. + return tool_name == self.matcher + + +# --------------------------------------------------------------------------- +# Public API +# --------------------------------------------------------------------------- + +def register_from_config( + cfg: Optional[Dict[str, Any]], + *, + accept_hooks: bool = False, +) -> List[ShellHookSpec]: + """Register every configured shell hook on the plugin manager. + + ``cfg`` is the full parsed config dict (``hermes_cli.config.load_config`` + output). The ``hooks:`` key is read out of it. Missing, empty, or + non-dict ``hooks`` is treated as zero configured hooks. + + ``accept_hooks=True`` skips the TTY consent prompt — the caller is + promising that the user has opted in via a flag, env var, or config + setting. ``HERMES_ACCEPT_HOOKS=1`` and ``hooks_auto_accept: true`` are + also honored inside this function so either CLI or gateway call sites + pick them up. + + Returns the list of :class:`ShellHookSpec` entries that ended up wired + up on the plugin manager. Skipped entries (unknown events, malformed, + not allowlisted, already registered) are logged but not returned. + """ + if not isinstance(cfg, dict): + return [] + + effective_accept = _resolve_effective_accept(cfg, accept_hooks) + + specs = _parse_hooks_block(cfg.get("hooks")) + if not specs: + return [] + + registered: List[ShellHookSpec] = [] + + # Import lazily — avoids circular imports at module-load time. + from hermes_cli.plugins import get_plugin_manager + + manager = get_plugin_manager() + + # Idempotence + allowlist read happen under the lock; the TTY + # prompt runs outside so other threads aren't parked on a blocking + # input(). Mutation re-takes the lock with a defensive idempotence + # re-check in case two callers ever race through the prompt. + for spec in specs: + key = (spec.event, spec.matcher, spec.command) + with _registered_lock: + if key in _registered: + continue + already_allowlisted = _is_allowlisted(spec.event, spec.command) + + if not already_allowlisted: + if not _prompt_and_record( + spec.event, spec.command, accept_hooks=effective_accept, + ): + logger.warning( + "shell hook for %s (%s) not allowlisted — skipped. " + "Use --accept-hooks / HERMES_ACCEPT_HOOKS=1 / " + "hooks_auto_accept: true, or approve at the TTY " + "prompt next run.", + spec.event, spec.command, + ) + continue + + with _registered_lock: + if key in _registered: + continue + manager._hooks.setdefault(spec.event, []).append(_make_callback(spec)) + _registered.add(key) + registered.append(spec) + logger.info( + "shell hook registered: %s -> %s (matcher=%s, timeout=%ds)", + spec.event, spec.command, spec.matcher, spec.timeout, + ) + + return registered + + +def iter_configured_hooks(cfg: Optional[Dict[str, Any]]) -> List[ShellHookSpec]: + """Return the parsed ``ShellHookSpec`` entries from config without + registering anything. Used by ``hermes hooks list`` and ``doctor``.""" + if not isinstance(cfg, dict): + return [] + return _parse_hooks_block(cfg.get("hooks")) + + +def reset_for_tests() -> None: + """Clear the idempotence set. Test-only helper.""" + with _registered_lock: + _registered.clear() + + +# --------------------------------------------------------------------------- +# Config parsing +# --------------------------------------------------------------------------- + +def _parse_hooks_block(hooks_cfg: Any) -> List[ShellHookSpec]: + """Normalise the ``hooks:`` dict into a flat list of ``ShellHookSpec``. + + Malformed entries warn-and-skip — we never raise from config parsing + because a broken hook must not crash the agent. + """ + from hermes_cli.plugins import VALID_HOOKS + + if not isinstance(hooks_cfg, dict): + return [] + + specs: List[ShellHookSpec] = [] + + for event_name, entries in hooks_cfg.items(): + if event_name not in VALID_HOOKS: + suggestion = difflib.get_close_matches( + str(event_name), VALID_HOOKS, n=1, cutoff=0.6, + ) + if suggestion: + logger.warning( + "unknown hook event %r in hooks: config — did you mean %r?", + event_name, suggestion[0], + ) + else: + logger.warning( + "unknown hook event %r in hooks: config (valid: %s)", + event_name, ", ".join(sorted(VALID_HOOKS)), + ) + continue + + if entries is None: + continue + + if not isinstance(entries, list): + logger.warning( + "hooks.%s must be a list of hook definitions; got %s", + event_name, type(entries).__name__, + ) + continue + + for i, raw in enumerate(entries): + spec = _parse_single_entry(event_name, i, raw) + if spec is not None: + specs.append(spec) + + return specs + + +def _parse_single_entry( + event: str, index: int, raw: Any, +) -> Optional[ShellHookSpec]: + if not isinstance(raw, dict): + logger.warning( + "hooks.%s[%d] must be a mapping with a 'command' key; got %s", + event, index, type(raw).__name__, + ) + return None + + command = raw.get("command") + if not isinstance(command, str) or not command.strip(): + logger.warning( + "hooks.%s[%d] is missing a non-empty 'command' field", + event, index, + ) + return None + + matcher = raw.get("matcher") + if matcher is not None and not isinstance(matcher, str): + logger.warning( + "hooks.%s[%d].matcher must be a string regex; ignoring", + event, index, + ) + matcher = None + + if matcher is not None and event not in ("pre_tool_call", "post_tool_call"): + logger.warning( + "hooks.%s[%d].matcher=%r will be ignored at runtime — the " + "matcher field is only honored for pre_tool_call / " + "post_tool_call. The hook will fire on every %s event.", + event, index, matcher, event, + ) + matcher = None + + timeout_raw = raw.get("timeout", DEFAULT_TIMEOUT_SECONDS) + try: + timeout = int(timeout_raw) + except (TypeError, ValueError): + logger.warning( + "hooks.%s[%d].timeout must be an int (got %r); using default %ds", + event, index, timeout_raw, DEFAULT_TIMEOUT_SECONDS, + ) + timeout = DEFAULT_TIMEOUT_SECONDS + + if timeout < 1: + logger.warning( + "hooks.%s[%d].timeout must be >=1; using default %ds", + event, index, DEFAULT_TIMEOUT_SECONDS, + ) + timeout = DEFAULT_TIMEOUT_SECONDS + + if timeout > MAX_TIMEOUT_SECONDS: + logger.warning( + "hooks.%s[%d].timeout=%ds exceeds max %ds; clamping", + event, index, timeout, MAX_TIMEOUT_SECONDS, + ) + timeout = MAX_TIMEOUT_SECONDS + + return ShellHookSpec( + event=event, + command=command.strip(), + matcher=matcher, + timeout=timeout, + ) + + +# --------------------------------------------------------------------------- +# Subprocess callback +# --------------------------------------------------------------------------- + +_TOP_LEVEL_PAYLOAD_KEYS = {"tool_name", "args", "session_id", "parent_session_id"} + + +def _spawn(spec: ShellHookSpec, stdin_json: str) -> Dict[str, Any]: + """Run ``spec.command`` as a subprocess with ``stdin_json`` on stdin. + + Returns a diagnostic dict with the same keys for every outcome + (``returncode``, ``stdout``, ``stderr``, ``timed_out``, + ``elapsed_seconds``, ``error``). This is the single place the + subprocess is actually invoked — both the live callback path + (:func:`_make_callback`) and the CLI test helper (:func:`run_once`) + go through it. + """ + result: Dict[str, Any] = { + "returncode": None, + "stdout": "", + "stderr": "", + "timed_out": False, + "elapsed_seconds": 0.0, + "error": None, + } + try: + argv = shlex.split(os.path.expanduser(spec.command)) + except ValueError as exc: + result["error"] = f"command {spec.command!r} cannot be parsed: {exc}" + return result + if not argv: + result["error"] = "empty command" + return result + + t0 = time.monotonic() + try: + proc = subprocess.run( + argv, + input=stdin_json, + capture_output=True, + timeout=spec.timeout, + text=True, + shell=False, + ) + except subprocess.TimeoutExpired: + result["timed_out"] = True + result["elapsed_seconds"] = round(time.monotonic() - t0, 3) + return result + except FileNotFoundError: + result["error"] = "command not found" + return result + except PermissionError: + result["error"] = "command not executable" + return result + except Exception as exc: # pragma: no cover — defensive + result["error"] = str(exc) + return result + + result["returncode"] = proc.returncode + result["stdout"] = proc.stdout or "" + result["stderr"] = proc.stderr or "" + result["elapsed_seconds"] = round(time.monotonic() - t0, 3) + return result + + +def _make_callback(spec: ShellHookSpec) -> Callable[..., Optional[Dict[str, Any]]]: + """Build the closure that ``invoke_hook()`` will call per firing.""" + + def _callback(**kwargs: Any) -> Optional[Dict[str, Any]]: + # Matcher gate — only meaningful for tool-scoped events. + if spec.event in ("pre_tool_call", "post_tool_call"): + if not spec.matches_tool(kwargs.get("tool_name")): + return None + + r = _spawn(spec, _serialize_payload(spec.event, kwargs)) + + if r["error"]: + logger.warning( + "shell hook failed (event=%s command=%s): %s", + spec.event, spec.command, r["error"], + ) + return None + if r["timed_out"]: + logger.warning( + "shell hook timed out after %.2fs (event=%s command=%s)", + r["elapsed_seconds"], spec.event, spec.command, + ) + return None + + stderr = r["stderr"].strip() + if stderr: + logger.debug( + "shell hook stderr (event=%s command=%s): %s", + spec.event, spec.command, stderr[:400], + ) + # Non-zero exits: log but still parse stdout so scripts that + # signal failure via exit code can also return a block directive. + if r["returncode"] != 0: + logger.warning( + "shell hook exited %d (event=%s command=%s); stderr=%s", + r["returncode"], spec.event, spec.command, stderr[:400], + ) + return _parse_response(spec.event, r["stdout"]) + + _callback.__name__ = f"shell_hook[{spec.event}:{spec.command}]" + _callback.__qualname__ = _callback.__name__ + return _callback + + +def _serialize_payload(event: str, kwargs: Dict[str, Any]) -> str: + """Render the stdin JSON payload. Unserialisable values are + stringified via ``default=str`` rather than dropped.""" + extras = {k: v for k, v in kwargs.items() if k not in _TOP_LEVEL_PAYLOAD_KEYS} + try: + cwd = str(Path.cwd()) + except OSError: + cwd = "" + payload = { + "hook_event_name": event, + "tool_name": kwargs.get("tool_name"), + "tool_input": kwargs.get("args") if isinstance(kwargs.get("args"), dict) else None, + "session_id": kwargs.get("session_id") or kwargs.get("parent_session_id") or "", + "cwd": cwd, + "extra": extras, + } + return json.dumps(payload, ensure_ascii=False, default=str) + + +def _parse_response(event: str, stdout: str) -> Optional[Dict[str, Any]]: + """Translate stdout JSON into a Hermes wire-shape dict. + + For ``pre_tool_call`` the Claude-Code-style ``{"decision": "block", + "reason": "..."}`` payload is translated into the canonical Hermes + ``{"action": "block", "message": "..."}`` shape expected by + :func:`hermes_cli.plugins.get_pre_tool_call_block_message`. This is + the single most important correctness invariant in this module — + skipping the translation silently breaks every ``pre_tool_call`` + block directive. + + For ``pre_llm_call``, ``{"context": "..."}`` is passed through + unchanged to match the existing plugin-hook contract. + + Anything else returns ``None``. + """ + stdout = (stdout or "").strip() + if not stdout: + return None + + try: + data = json.loads(stdout) + except json.JSONDecodeError: + logger.warning( + "shell hook stdout was not valid JSON (event=%s): %s", + event, stdout[:200], + ) + return None + + if not isinstance(data, dict): + return None + + if event == "pre_tool_call": + if data.get("action") == "block": + message = data.get("message") or data.get("reason") or "" + if isinstance(message, str) and message: + return {"action": "block", "message": message} + if data.get("decision") == "block": + message = data.get("reason") or data.get("message") or "" + if isinstance(message, str) and message: + return {"action": "block", "message": message} + return None + + context = data.get("context") + if isinstance(context, str) and context.strip(): + return {"context": context} + + return None + + +# --------------------------------------------------------------------------- +# Allowlist / consent +# --------------------------------------------------------------------------- + +def allowlist_path() -> Path: + """Path to the per-user shell-hook allowlist file.""" + return get_hermes_home() / ALLOWLIST_FILENAME + + +def load_allowlist() -> Dict[str, Any]: + """Return the parsed allowlist, or an empty skeleton if absent.""" + try: + raw = json.loads(allowlist_path().read_text()) + except (FileNotFoundError, json.JSONDecodeError, OSError): + return {"approvals": []} + if not isinstance(raw, dict): + return {"approvals": []} + approvals = raw.get("approvals") + if not isinstance(approvals, list): + raw["approvals"] = [] + return raw + + +def save_allowlist(data: Dict[str, Any]) -> None: + """Atomically persist the allowlist via per-process ``mkstemp`` + + ``os.replace``. Cross-process read-modify-write races are handled + by :func:`_locked_update_approvals` (``fcntl.flock``). On OSError + the failure is logged; the in-process hook still registers but + the approval won't survive across runs.""" + p = allowlist_path() + try: + p.parent.mkdir(parents=True, exist_ok=True) + fd, tmp_path = tempfile.mkstemp( + prefix=f"{p.name}.", suffix=".tmp", dir=str(p.parent), + ) + try: + with os.fdopen(fd, "w") as fh: + fh.write(json.dumps(data, indent=2, sort_keys=True)) + os.replace(tmp_path, p) + except Exception: + try: + os.unlink(tmp_path) + except OSError: + pass + raise + except OSError as exc: + logger.warning( + "Failed to persist shell hook allowlist to %s: %s. " + "The approval is in-memory for this run, but the next " + "startup will re-prompt (or skip registration on non-TTY " + "runs without --accept-hooks / HERMES_ACCEPT_HOOKS).", + p, exc, + ) + + +def _is_allowlisted(event: str, command: str) -> bool: + data = load_allowlist() + return any( + isinstance(e, dict) + and e.get("event") == event + and e.get("command") == command + for e in data.get("approvals", []) + ) + + +@contextmanager +def _locked_update_approvals() -> Iterator[Dict[str, Any]]: + """Serialise read-modify-write on the allowlist across processes. + + Holds an exclusive ``flock`` on a sibling lock file for the duration + of the update so concurrent ``_record_approval``/``revoke`` callers + cannot clobber each other's changes (the race Codex reproduced with + 20–50 simultaneous writers). Falls back to an in-process lock on + platforms without ``fcntl``. + """ + p = allowlist_path() + p.parent.mkdir(parents=True, exist_ok=True) + lock_path = p.with_suffix(p.suffix + ".lock") + + if fcntl is None: # pragma: no cover — non-POSIX fallback + with _allowlist_write_lock: + data = load_allowlist() + yield data + save_allowlist(data) + return + + with open(lock_path, "a+") as lock_fh: + fcntl.flock(lock_fh.fileno(), fcntl.LOCK_EX) + try: + data = load_allowlist() + yield data + save_allowlist(data) + finally: + fcntl.flock(lock_fh.fileno(), fcntl.LOCK_UN) + + +def _prompt_and_record( + event: str, command: str, *, accept_hooks: bool, +) -> bool: + """Decide whether to approve an unseen ``(event, command)`` pair. + Returns ``True`` iff the approval was granted and recorded. + """ + if accept_hooks: + _record_approval(event, command) + logger.info( + "shell hook auto-approved via --accept-hooks / env / config: " + "%s -> %s", event, command, + ) + return True + + if not sys.stdin.isatty(): + return False + + print( + f"\n⚠ Hermes is about to register a shell hook that will run a\n" + f" command on your behalf.\n\n" + f" Event: {event}\n" + f" Command: {command}\n\n" + f" Commands run with your full user credentials. Only approve\n" + f" commands you trust." + ) + try: + answer = input("Allow this hook to run? [y/N]: ").strip().lower() + except (EOFError, KeyboardInterrupt): + print() # keep the terminal tidy after ^C + return False + + if answer in ("y", "yes"): + _record_approval(event, command) + return True + + return False + + +def _record_approval(event: str, command: str) -> None: + entry = { + "event": event, + "command": command, + "approved_at": _utc_now_iso(), + "script_mtime_at_approval": script_mtime_iso(command), + } + with _locked_update_approvals() as data: + data["approvals"] = [ + e for e in data.get("approvals", []) + if not ( + isinstance(e, dict) + and e.get("event") == event + and e.get("command") == command + ) + ] + [entry] + + +def _utc_now_iso() -> str: + return datetime.now(tz=timezone.utc).isoformat().replace("+00:00", "Z") + + +def revoke(command: str) -> int: + """Remove every allowlist entry matching ``command``. + + Returns the number of entries removed. Does not unregister any + callbacks that are already live on the plugin manager in the current + process — restart the CLI / gateway to drop them. + """ + with _locked_update_approvals() as data: + before = len(data.get("approvals", [])) + data["approvals"] = [ + e for e in data.get("approvals", []) + if not (isinstance(e, dict) and e.get("command") == command) + ] + after = len(data["approvals"]) + return before - after + + +_SCRIPT_EXTENSIONS: Tuple[str, ...] = ( + ".sh", ".bash", ".zsh", ".fish", + ".py", ".pyw", + ".rb", ".pl", ".lua", + ".js", ".mjs", ".cjs", ".ts", +) + + +def _command_script_path(command: str) -> str: + """Return the script path from ``command`` for doctor / drift checks. + + Prefers a token ending in a known script extension, then a token + containing ``/`` or leading ``~``, then the first token. Handles + ``python3 /path/hook.py``, ``/usr/bin/env bash hook.sh``, and the + common bare-path form. + """ + try: + parts = shlex.split(command) + except ValueError: + return command + if not parts: + return command + for part in parts: + if part.lower().endswith(_SCRIPT_EXTENSIONS): + return part + for part in parts: + if "/" in part or part.startswith("~"): + return part + return parts[0] + + +# --------------------------------------------------------------------------- +# Helpers for accept-hooks resolution +# --------------------------------------------------------------------------- + +def _resolve_effective_accept( + cfg: Dict[str, Any], accept_hooks_arg: bool, +) -> bool: + """Combine all three opt-in channels into a single boolean. + + Precedence (any truthy source flips us on): + 1. ``--accept-hooks`` flag (CLI) / explicit argument + 2. ``HERMES_ACCEPT_HOOKS`` env var + 3. ``hooks_auto_accept: true`` in ``cli-config.yaml`` + """ + if accept_hooks_arg: + return True + env = os.environ.get("HERMES_ACCEPT_HOOKS", "").strip().lower() + if env in ("1", "true", "yes", "on"): + return True + cfg_val = cfg.get("hooks_auto_accept", False) + return bool(cfg_val) + + +# --------------------------------------------------------------------------- +# Introspection (used by `hermes hooks` CLI) +# --------------------------------------------------------------------------- + +def allowlist_entry_for(event: str, command: str) -> Optional[Dict[str, Any]]: + """Return the allowlist record for this pair, if any.""" + for e in load_allowlist().get("approvals", []): + if ( + isinstance(e, dict) + and e.get("event") == event + and e.get("command") == command + ): + return e + return None + + +def script_mtime_iso(command: str) -> Optional[str]: + """ISO-8601 mtime of the resolved script path, or ``None`` if the + script is missing.""" + path = _command_script_path(command) + if not path: + return None + try: + expanded = os.path.expanduser(path) + return datetime.fromtimestamp( + os.path.getmtime(expanded), tz=timezone.utc, + ).isoformat().replace("+00:00", "Z") + except OSError: + return None + + +def script_is_executable(command: str) -> bool: + """Return ``True`` iff ``command`` is runnable as configured. + + For a bare invocation (``/path/hook.sh``) the script itself must be + executable. For interpreter-prefixed commands (``python3 + /path/hook.py``, ``/usr/bin/env bash hook.sh``) the script just has + to be readable — the interpreter doesn't care about the ``X_OK`` + bit. Mirrors what ``_spawn`` would actually do at runtime.""" + path = _command_script_path(command) + if not path: + return False + expanded = os.path.expanduser(path) + if not os.path.isfile(expanded): + return False + try: + argv = shlex.split(command) + except ValueError: + return False + is_bare_invocation = bool(argv) and argv[0] == path + required = os.X_OK if is_bare_invocation else os.R_OK + return os.access(expanded, required) + + +def run_once( + spec: ShellHookSpec, kwargs: Dict[str, Any], +) -> Dict[str, Any]: + """Fire a single shell-hook invocation with a synthetic payload. + Used by ``hermes hooks test`` and ``hermes hooks doctor``. + + ``kwargs`` is the same dict that :func:`hermes_cli.plugins.invoke_hook` + would pass at runtime. It is routed through :func:`_serialize_payload` + so the synthetic stdin exactly matches what a real hook firing would + produce — otherwise scripts tested via ``hermes hooks test`` could + diverge silently from production behaviour. + + Returns the :func:`_spawn` diagnostic dict plus a ``parsed`` field + holding the canonical Hermes-wire-shape response.""" + stdin_json = _serialize_payload(spec.event, kwargs) + result = _spawn(spec, stdin_json) + result["parsed"] = _parse_response(spec.event, result["stdout"]) + return result diff --git a/agent/skill_commands.py b/agent/skill_commands.py index 1f000eefed2a..9c130ab84afc 100644 --- a/agent/skill_commands.py +++ b/agent/skill_commands.py @@ -8,10 +8,13 @@ import json import logging import re +import subprocess from datetime import datetime from pathlib import Path from typing import Any, Dict, Optional +from hermes_constants import display_hermes_home + logger = logging.getLogger(__name__) _skill_commands: Dict[str, Dict[str, Any]] = {} @@ -20,6 +23,110 @@ _SKILL_INVALID_CHARS = re.compile(r"[^a-z0-9-]") _SKILL_MULTI_HYPHEN = re.compile(r"-{2,}") +# Matches ${HERMES_SKILL_DIR} / ${HERMES_SESSION_ID} tokens in SKILL.md. +# Tokens that don't resolve (e.g. ${HERMES_SESSION_ID} with no session) are +# left as-is so the user can debug them. +_SKILL_TEMPLATE_RE = re.compile(r"\$\{(HERMES_SKILL_DIR|HERMES_SESSION_ID)\}") + +# Matches inline shell snippets like: !`date +%Y-%m-%d` +# Non-greedy, single-line only — no newlines inside the backticks. +_INLINE_SHELL_RE = re.compile(r"!`([^`\n]+)`") + +# Cap inline-shell output so a runaway command can't blow out the context. +_INLINE_SHELL_MAX_OUTPUT = 4000 + + +def _load_skills_config() -> dict: + """Load the ``skills`` section of config.yaml (best-effort).""" + try: + from hermes_cli.config import load_config + + cfg = load_config() or {} + skills_cfg = cfg.get("skills") + if isinstance(skills_cfg, dict): + return skills_cfg + except Exception: + logger.debug("Could not read skills config", exc_info=True) + return {} + + +def _substitute_template_vars( + content: str, + skill_dir: Path | None, + session_id: str | None, +) -> str: + """Replace ${HERMES_SKILL_DIR} / ${HERMES_SESSION_ID} in skill content. + + Only substitutes tokens for which a concrete value is available — + unresolved tokens are left in place so the author can spot them. + """ + if not content: + return content + + skill_dir_str = str(skill_dir) if skill_dir else None + + def _replace(match: re.Match) -> str: + token = match.group(1) + if token == "HERMES_SKILL_DIR" and skill_dir_str: + return skill_dir_str + if token == "HERMES_SESSION_ID" and session_id: + return str(session_id) + return match.group(0) + + return _SKILL_TEMPLATE_RE.sub(_replace, content) + + +def _run_inline_shell(command: str, cwd: Path | None, timeout: int) -> str: + """Execute a single inline-shell snippet and return its stdout (trimmed). + + Failures return a short ``[inline-shell error: ...]`` marker instead of + raising, so one bad snippet can't wreck the whole skill message. + """ + try: + completed = subprocess.run( + ["bash", "-c", command], + cwd=str(cwd) if cwd else None, + capture_output=True, + text=True, + timeout=max(1, int(timeout)), + check=False, + ) + except subprocess.TimeoutExpired: + return f"[inline-shell timeout after {timeout}s: {command}]" + except FileNotFoundError: + return f"[inline-shell error: bash not found]" + except Exception as exc: + return f"[inline-shell error: {exc}]" + + output = (completed.stdout or "").rstrip("\n") + if not output and completed.stderr: + output = completed.stderr.rstrip("\n") + if len(output) > _INLINE_SHELL_MAX_OUTPUT: + output = output[:_INLINE_SHELL_MAX_OUTPUT] + "…[truncated]" + return output + + +def _expand_inline_shell( + content: str, + skill_dir: Path | None, + timeout: int, +) -> str: + """Replace every !`cmd` snippet in ``content`` with its stdout. + + Runs each snippet with the skill directory as CWD so relative paths in + the snippet work the way the author expects. + """ + if "!`" not in content: + return content + + def _replace(match: re.Match) -> str: + cmd = match.group(1).strip() + if not cmd: + return "" + return _run_inline_shell(cmd, skill_dir, timeout) + + return _INLINE_SHELL_RE.sub(_replace, content) + def build_plan_path( user_instruction: str = "", @@ -70,7 +177,14 @@ def _load_skill_payload(skill_identifier: str, task_id: str | None = None) -> tu skill_name = str(loaded_skill.get("name") or normalized) skill_path = str(loaded_skill.get("path") or "") skill_dir = None - if skill_path: + # Prefer the absolute skill_dir returned by skill_view() — this is + # correct for both local and external skills. Fall back to the old + # SKILLS_DIR-relative reconstruction only when skill_dir is absent + # (e.g. legacy skill_view responses). + abs_skill_dir = loaded_skill.get("skill_dir") + if abs_skill_dir: + skill_dir = Path(abs_skill_dir) + elif skill_path: try: skill_dir = SKILLS_DIR / Path(skill_path).parent except Exception: @@ -108,7 +222,7 @@ def _inject_skill_config(loaded_skill: dict[str, Any], parts: list[str]) -> None if not resolved: return - lines = ["", "[Skill config (from ~/.hermes/config.yaml):"] + lines = ["", f"[Skill config (from {display_hermes_home()}/config.yaml):"] for key, value in resolved.items(): display_val = str(value) if value else "(not set)" lines.append(f" {key} = {display_val}") @@ -124,14 +238,36 @@ def _build_skill_message( activation_note: str, user_instruction: str = "", runtime_note: str = "", + session_id: str | None = None, ) -> str: """Format a loaded skill into a user/system message payload.""" from tools.skills_tool import SKILLS_DIR content = str(loaded_skill.get("content") or "") + # ── Template substitution and inline-shell expansion ── + # Done before anything else so downstream blocks (setup notes, + # supporting-file hints) see the expanded content. + skills_cfg = _load_skills_config() + if skills_cfg.get("template_vars", True): + content = _substitute_template_vars(content, skill_dir, session_id) + if skills_cfg.get("inline_shell", False): + timeout = int(skills_cfg.get("inline_shell_timeout", 10) or 10) + content = _expand_inline_shell(content, skill_dir, timeout) + parts = [activation_note, "", content.strip()] + # ── Inject the absolute skill directory so the agent can reference + # bundled scripts without an extra skill_view() round-trip. ── + if skill_dir: + parts.append("") + parts.append(f"[Skill directory: {skill_dir}]") + parts.append( + "Resolve any relative paths in this skill (e.g. `scripts/foo.js`, " + "`templates/config.yaml`) against that directory, then run them " + "with the terminal tool using the absolute path." + ) + # ── Inject resolved skill config values ── _inject_skill_config(loaded_skill, parts) @@ -179,11 +315,13 @@ def _build_skill_message( # Skill is from an external dir — use the skill name instead skill_view_target = skill_dir.name parts.append("") - parts.append("[This skill has supporting files you can load with the skill_view tool:]") + parts.append("[This skill has supporting files:]") for sf in supporting: - parts.append(f"- {sf}") + parts.append(f"- {sf} -> {skill_dir / sf}") parts.append( - f'\nTo view any of these, use: skill_view(name="{skill_view_target}", file_path="")' + f'\nLoad any of these with skill_view(name="{skill_view_target}", ' + f'file_path=""), or run scripts directly by absolute path ' + f"(e.g. `node {skill_dir}/scripts/foo.js`)." ) if user_instruction: @@ -207,7 +345,7 @@ def scan_skill_commands() -> Dict[str, Dict[str, Any]]: _skill_commands = {} try: from tools.skills_tool import SKILLS_DIR, _parse_frontmatter, skill_matches_platform, _get_disabled_skill_names - from agent.skill_utils import get_external_skills_dirs + from agent.skill_utils import get_external_skills_dirs, iter_skill_index_files disabled = _get_disabled_skill_names() seen_names: set = set() @@ -218,7 +356,7 @@ def scan_skill_commands() -> Dict[str, Dict[str, Any]]: dirs_to_scan.extend(get_external_skills_dirs()) for scan_dir in dirs_to_scan: - for skill_md in scan_dir.rglob("SKILL.md"): + for skill_md in iter_skill_index_files(scan_dir, "SKILL.md"): if any(part in ('.git', '.github', '.hub') for part in skill_md.parts): continue try: @@ -323,6 +461,7 @@ def build_skill_invocation_message( activation_note, user_instruction=user_instruction, runtime_note=runtime_note, + session_id=task_id, ) @@ -361,6 +500,7 @@ def build_preloaded_skills_prompt( loaded_skill, skill_dir, activation_note, + session_id=task_id, ) ) loaded_names.append(skill_name) diff --git a/agent/skill_utils.py b/agent/skill_utils.py index 97ba92b735a8..d4d94f7e280f 100644 --- a/agent/skill_utils.py +++ b/agent/skill_utils.py @@ -10,7 +10,7 @@ import re import sys from pathlib import Path -from typing import Any, Dict, List, Set, Tuple +from typing import Any, Dict, List, Optional, Set, Tuple from hermes_constants import get_config_path, get_skills_dir @@ -435,9 +435,31 @@ def iter_skill_index_files(skills_dir: Path, filename: str): Excludes ``.git``, ``.github``, ``.hub`` directories. """ matches = [] - for root, dirs, files in os.walk(skills_dir): + for root, dirs, files in os.walk(skills_dir, followlinks=True): dirs[:] = [d for d in dirs if d not in EXCLUDED_SKILL_DIRS] if filename in files: matches.append(Path(root) / filename) for path in sorted(matches, key=lambda p: str(p.relative_to(skills_dir))): yield path + + +# ── Namespace helpers for plugin-provided skills ─────────────────────────── + +_NAMESPACE_RE = re.compile(r"^[a-zA-Z0-9_-]+$") + + +def parse_qualified_name(name: str) -> Tuple[Optional[str], str]: + """Split ``'namespace:skill-name'`` into ``(namespace, bare_name)``. + + Returns ``(None, name)`` when there is no ``':'``. + """ + if ":" not in name: + return None, name + return tuple(name.split(":", 1)) # type: ignore[return-value] + + +def is_valid_namespace(candidate: Optional[str]) -> bool: + """Check whether *candidate* is a valid namespace (``[a-zA-Z0-9_-]+``).""" + if not candidate: + return False + return bool(_NAMESPACE_RE.match(candidate)) diff --git a/agent/smart_model_routing.py b/agent/smart_model_routing.py deleted file mode 100644 index 6d482be27051..000000000000 --- a/agent/smart_model_routing.py +++ /dev/null @@ -1,195 +0,0 @@ -"""Helpers for optional cheap-vs-strong model routing.""" - -from __future__ import annotations - -import os -import re -from typing import Any, Dict, Optional - -from utils import is_truthy_value - -_COMPLEX_KEYWORDS = { - "debug", - "debugging", - "implement", - "implementation", - "refactor", - "patch", - "traceback", - "stacktrace", - "exception", - "error", - "analyze", - "analysis", - "investigate", - "architecture", - "design", - "compare", - "benchmark", - "optimize", - "optimise", - "review", - "terminal", - "shell", - "tool", - "tools", - "pytest", - "test", - "tests", - "plan", - "planning", - "delegate", - "subagent", - "cron", - "docker", - "kubernetes", -} - -_URL_RE = re.compile(r"https?://|www\.", re.IGNORECASE) - - -def _coerce_bool(value: Any, default: bool = False) -> bool: - return is_truthy_value(value, default=default) - - -def _coerce_int(value: Any, default: int) -> int: - try: - return int(value) - except (TypeError, ValueError): - return default - - -def choose_cheap_model_route(user_message: str, routing_config: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]: - """Return the configured cheap-model route when a message looks simple. - - Conservative by design: if the message has signs of code/tool/debugging/ - long-form work, keep the primary model. - """ - cfg = routing_config or {} - if not _coerce_bool(cfg.get("enabled"), False): - return None - - cheap_model = cfg.get("cheap_model") or {} - if not isinstance(cheap_model, dict): - return None - provider = str(cheap_model.get("provider") or "").strip().lower() - model = str(cheap_model.get("model") or "").strip() - if not provider or not model: - return None - - text = (user_message or "").strip() - if not text: - return None - - max_chars = _coerce_int(cfg.get("max_simple_chars"), 160) - max_words = _coerce_int(cfg.get("max_simple_words"), 28) - - if len(text) > max_chars: - return None - if len(text.split()) > max_words: - return None - if text.count("\n") > 1: - return None - if "```" in text or "`" in text: - return None - if _URL_RE.search(text): - return None - - lowered = text.lower() - words = {token.strip(".,:;!?()[]{}\"'`") for token in lowered.split()} - if words & _COMPLEX_KEYWORDS: - return None - - route = dict(cheap_model) - route["provider"] = provider - route["model"] = model - route["routing_reason"] = "simple_turn" - return route - - -def resolve_turn_route(user_message: str, routing_config: Optional[Dict[str, Any]], primary: Dict[str, Any]) -> Dict[str, Any]: - """Resolve the effective model/runtime for one turn. - - Returns a dict with model/runtime/signature/label fields. - """ - route = choose_cheap_model_route(user_message, routing_config) - if not route: - return { - "model": primary.get("model"), - "runtime": { - "api_key": primary.get("api_key"), - "base_url": primary.get("base_url"), - "provider": primary.get("provider"), - "api_mode": primary.get("api_mode"), - "command": primary.get("command"), - "args": list(primary.get("args") or []), - "credential_pool": primary.get("credential_pool"), - }, - "label": None, - "signature": ( - primary.get("model"), - primary.get("provider"), - primary.get("base_url"), - primary.get("api_mode"), - primary.get("command"), - tuple(primary.get("args") or ()), - ), - } - - from hermes_cli.runtime_provider import resolve_runtime_provider - - explicit_api_key = None - api_key_env = str(route.get("api_key_env") or "").strip() - if api_key_env: - explicit_api_key = os.getenv(api_key_env) or None - - try: - runtime = resolve_runtime_provider( - requested=route.get("provider"), - explicit_api_key=explicit_api_key, - explicit_base_url=route.get("base_url"), - ) - except Exception: - return { - "model": primary.get("model"), - "runtime": { - "api_key": primary.get("api_key"), - "base_url": primary.get("base_url"), - "provider": primary.get("provider"), - "api_mode": primary.get("api_mode"), - "command": primary.get("command"), - "args": list(primary.get("args") or []), - "credential_pool": primary.get("credential_pool"), - }, - "label": None, - "signature": ( - primary.get("model"), - primary.get("provider"), - primary.get("base_url"), - primary.get("api_mode"), - primary.get("command"), - tuple(primary.get("args") or ()), - ), - } - - return { - "model": route.get("model"), - "runtime": { - "api_key": runtime.get("api_key"), - "base_url": runtime.get("base_url"), - "provider": runtime.get("provider"), - "api_mode": runtime.get("api_mode"), - "command": runtime.get("command"), - "args": list(runtime.get("args") or []), - "credential_pool": runtime.get("credential_pool"), - }, - "label": f"smart route → {route.get('model')} ({runtime.get('provider')})", - "signature": ( - route.get("model"), - runtime.get("provider"), - runtime.get("base_url"), - runtime.get("api_mode"), - runtime.get("command"), - tuple(runtime.get("args") or ()), - ), - } diff --git a/agent/title_generator.py b/agent/title_generator.py index d6ed9200a26d..99c771cb5095 100644 --- a/agent/title_generator.py +++ b/agent/title_generator.py @@ -38,7 +38,7 @@ def generate_title(user_message: str, assistant_response: str, timeout: float = response = call_llm( task="title_generation", messages=messages, - max_tokens=30, + max_tokens=500, temperature=0.3, timeout=timeout, ) diff --git a/agent/transports/__init__.py b/agent/transports/__init__.py new file mode 100644 index 000000000000..5752113325cc --- /dev/null +++ b/agent/transports/__init__.py @@ -0,0 +1,51 @@ +"""Transport layer types and registry for provider response normalization. + +Usage: + from agent.transports import get_transport + transport = get_transport("anthropic_messages") + result = transport.normalize_response(raw_response) +""" + +from agent.transports.types import NormalizedResponse, ToolCall, Usage, build_tool_call, map_finish_reason # noqa: F401 + +_REGISTRY: dict = {} + + +def register_transport(api_mode: str, transport_cls: type) -> None: + """Register a transport class for an api_mode string.""" + _REGISTRY[api_mode] = transport_cls + + +def get_transport(api_mode: str): + """Get a transport instance for the given api_mode. + + Returns None if no transport is registered for this api_mode. + This allows gradual migration — call sites can check for None + and fall back to the legacy code path. + """ + if not _REGISTRY: + _discover_transports() + cls = _REGISTRY.get(api_mode) + if cls is None: + return None + return cls() + + +def _discover_transports() -> None: + """Import all transport modules to trigger auto-registration.""" + try: + import agent.transports.anthropic # noqa: F401 + except ImportError: + pass + try: + import agent.transports.codex # noqa: F401 + except ImportError: + pass + try: + import agent.transports.chat_completions # noqa: F401 + except ImportError: + pass + try: + import agent.transports.bedrock # noqa: F401 + except ImportError: + pass diff --git a/agent/transports/anthropic.py b/agent/transports/anthropic.py new file mode 100644 index 000000000000..66c485b523b8 --- /dev/null +++ b/agent/transports/anthropic.py @@ -0,0 +1,177 @@ +"""Anthropic Messages API transport. + +Delegates to the existing adapter functions in agent/anthropic_adapter.py. +This transport owns format conversion and normalization — NOT client lifecycle. +""" + +from typing import Any, Dict, List, Optional + +from agent.transports.base import ProviderTransport +from agent.transports.types import NormalizedResponse + + +class AnthropicTransport(ProviderTransport): + """Transport for api_mode='anthropic_messages'. + + Wraps the existing functions in anthropic_adapter.py behind the + ProviderTransport ABC. Each method delegates — no logic is duplicated. + """ + + @property + def api_mode(self) -> str: + return "anthropic_messages" + + def convert_messages(self, messages: List[Dict[str, Any]], **kwargs) -> Any: + """Convert OpenAI messages to Anthropic (system, messages) tuple. + + kwargs: + base_url: Optional[str] — affects thinking signature handling. + """ + from agent.anthropic_adapter import convert_messages_to_anthropic + + base_url = kwargs.get("base_url") + return convert_messages_to_anthropic(messages, base_url=base_url) + + def convert_tools(self, tools: List[Dict[str, Any]]) -> Any: + """Convert OpenAI tool schemas to Anthropic input_schema format.""" + from agent.anthropic_adapter import convert_tools_to_anthropic + + return convert_tools_to_anthropic(tools) + + def build_kwargs( + self, + model: str, + messages: List[Dict[str, Any]], + tools: Optional[List[Dict[str, Any]]] = None, + **params, + ) -> Dict[str, Any]: + """Build Anthropic messages.create() kwargs. + + Calls convert_messages and convert_tools internally. + + params (all optional): + max_tokens: int + reasoning_config: dict | None + tool_choice: str | None + is_oauth: bool + preserve_dots: bool + context_length: int | None + base_url: str | None + fast_mode: bool + """ + from agent.anthropic_adapter import build_anthropic_kwargs + + return build_anthropic_kwargs( + model=model, + messages=messages, + tools=tools, + max_tokens=params.get("max_tokens", 16384), + reasoning_config=params.get("reasoning_config"), + tool_choice=params.get("tool_choice"), + is_oauth=params.get("is_oauth", False), + preserve_dots=params.get("preserve_dots", False), + context_length=params.get("context_length"), + base_url=params.get("base_url"), + fast_mode=params.get("fast_mode", False), + ) + + def normalize_response(self, response: Any, **kwargs) -> NormalizedResponse: + """Normalize Anthropic response to NormalizedResponse. + + Parses content blocks (text, thinking, tool_use), maps stop_reason + to OpenAI finish_reason, and collects reasoning_details in provider_data. + """ + import json + from agent.anthropic_adapter import _to_plain_data + from agent.transports.types import ToolCall + + strip_tool_prefix = kwargs.get("strip_tool_prefix", False) + _MCP_PREFIX = "mcp_" + + text_parts = [] + reasoning_parts = [] + reasoning_details = [] + tool_calls = [] + + for block in response.content: + if block.type == "text": + text_parts.append(block.text) + elif block.type == "thinking": + reasoning_parts.append(block.thinking) + block_dict = _to_plain_data(block) + if isinstance(block_dict, dict): + reasoning_details.append(block_dict) + elif block.type == "tool_use": + name = block.name + if strip_tool_prefix and name.startswith(_MCP_PREFIX): + name = name[len(_MCP_PREFIX):] + tool_calls.append( + ToolCall( + id=block.id, + name=name, + arguments=json.dumps(block.input), + ) + ) + + finish_reason = self._STOP_REASON_MAP.get(response.stop_reason, "stop") + + provider_data = {} + if reasoning_details: + provider_data["reasoning_details"] = reasoning_details + + return NormalizedResponse( + content="\n".join(text_parts) if text_parts else None, + tool_calls=tool_calls or None, + finish_reason=finish_reason, + reasoning="\n\n".join(reasoning_parts) if reasoning_parts else None, + usage=None, + provider_data=provider_data or None, + ) + + def validate_response(self, response: Any) -> bool: + """Check Anthropic response structure is valid. + + An empty content list is legitimate when ``stop_reason == "end_turn"`` + — the model's canonical way of signalling "nothing more to add" after + a tool turn that already delivered the user-facing text. Treating it + as invalid falsely retries a completed response. + """ + if response is None: + return False + content_blocks = getattr(response, "content", None) + if not isinstance(content_blocks, list): + return False + if not content_blocks: + return getattr(response, "stop_reason", None) == "end_turn" + return True + + def extract_cache_stats(self, response: Any) -> Optional[Dict[str, int]]: + """Extract Anthropic cache_read and cache_creation token counts.""" + usage = getattr(response, "usage", None) + if usage is None: + return None + cached = getattr(usage, "cache_read_input_tokens", 0) or 0 + written = getattr(usage, "cache_creation_input_tokens", 0) or 0 + if cached or written: + return {"cached_tokens": cached, "creation_tokens": written} + return None + + # Promote the adapter's canonical mapping to module level so it's shared + _STOP_REASON_MAP = { + "end_turn": "stop", + "tool_use": "tool_calls", + "max_tokens": "length", + "stop_sequence": "stop", + "refusal": "content_filter", + "model_context_window_exceeded": "length", + } + + def map_finish_reason(self, raw_reason: str) -> str: + """Map Anthropic stop_reason to OpenAI finish_reason.""" + return self._STOP_REASON_MAP.get(raw_reason, "stop") + + +# Auto-register on import +from agent.transports import register_transport # noqa: E402 + +register_transport("anthropic_messages", AnthropicTransport) diff --git a/agent/transports/base.py b/agent/transports/base.py new file mode 100644 index 000000000000..b516967b6af0 --- /dev/null +++ b/agent/transports/base.py @@ -0,0 +1,89 @@ +"""Abstract base for provider transports. + +A transport owns the data path for one api_mode: + convert_messages → convert_tools → build_kwargs → normalize_response + +It does NOT own: client construction, streaming, credential refresh, +prompt caching, interrupt handling, or retry logic. Those stay on AIAgent. +""" + +from abc import ABC, abstractmethod +from typing import Any, Dict, List, Optional + +from agent.transports.types import NormalizedResponse + + +class ProviderTransport(ABC): + """Base class for provider-specific format conversion and normalization.""" + + @property + @abstractmethod + def api_mode(self) -> str: + """The api_mode string this transport handles (e.g. 'anthropic_messages').""" + ... + + @abstractmethod + def convert_messages(self, messages: List[Dict[str, Any]], **kwargs) -> Any: + """Convert OpenAI-format messages to provider-native format. + + Returns provider-specific structure (e.g. (system, messages) for Anthropic, + or the messages list unchanged for chat_completions). + """ + ... + + @abstractmethod + def convert_tools(self, tools: List[Dict[str, Any]]) -> Any: + """Convert OpenAI-format tool definitions to provider-native format. + + Returns provider-specific tool list (e.g. Anthropic input_schema format). + """ + ... + + @abstractmethod + def build_kwargs( + self, + model: str, + messages: List[Dict[str, Any]], + tools: Optional[List[Dict[str, Any]]] = None, + **params, + ) -> Dict[str, Any]: + """Build the complete API call kwargs dict. + + This is the primary entry point — it typically calls convert_messages() + and convert_tools() internally, then adds model-specific config. + + Returns a dict ready to be passed to the provider's SDK client. + """ + ... + + @abstractmethod + def normalize_response(self, response: Any, **kwargs) -> NormalizedResponse: + """Normalize a raw provider response to the shared NormalizedResponse type. + + This is the only method that returns a transport-layer type. + """ + ... + + def validate_response(self, response: Any) -> bool: + """Optional: check if the raw response is structurally valid. + + Returns True if valid, False if the response should be treated as invalid. + Default implementation always returns True. + """ + return True + + def extract_cache_stats(self, response: Any) -> Optional[Dict[str, int]]: + """Optional: extract provider-specific cache hit/creation stats. + + Returns dict with 'cached_tokens' and 'creation_tokens', or None. + Default returns None. + """ + return None + + def map_finish_reason(self, raw_reason: str) -> str: + """Optional: map provider-specific stop reason to OpenAI equivalent. + + Default returns the raw reason unchanged. Override for providers + with different stop reason vocabularies. + """ + return raw_reason diff --git a/agent/transports/bedrock.py b/agent/transports/bedrock.py new file mode 100644 index 000000000000..af549e7eae66 --- /dev/null +++ b/agent/transports/bedrock.py @@ -0,0 +1,154 @@ +"""AWS Bedrock Converse API transport. + +Delegates to the existing adapter functions in agent/bedrock_adapter.py. +Bedrock uses its own boto3 client (not the OpenAI SDK), so the transport +owns format conversion and normalization, while client construction and +boto3 calls stay on AIAgent. +""" + +from typing import Any, Dict, List, Optional + +from agent.transports.base import ProviderTransport +from agent.transports.types import NormalizedResponse, ToolCall, Usage + + +class BedrockTransport(ProviderTransport): + """Transport for api_mode='bedrock_converse'.""" + + @property + def api_mode(self) -> str: + return "bedrock_converse" + + def convert_messages(self, messages: List[Dict[str, Any]], **kwargs) -> Any: + """Convert OpenAI messages to Bedrock Converse format.""" + from agent.bedrock_adapter import convert_messages_to_converse + return convert_messages_to_converse(messages) + + def convert_tools(self, tools: List[Dict[str, Any]]) -> Any: + """Convert OpenAI tool schemas to Bedrock Converse toolConfig.""" + from agent.bedrock_adapter import convert_tools_to_converse + return convert_tools_to_converse(tools) + + def build_kwargs( + self, + model: str, + messages: List[Dict[str, Any]], + tools: Optional[List[Dict[str, Any]]] = None, + **params, + ) -> Dict[str, Any]: + """Build Bedrock converse() kwargs. + + Calls convert_messages and convert_tools internally. + + params: + max_tokens: int — output token limit (default 4096) + temperature: float | None + guardrail_config: dict | None — Bedrock guardrails + region: str — AWS region (default 'us-east-1') + """ + from agent.bedrock_adapter import build_converse_kwargs + + region = params.get("region", "us-east-1") + guardrail = params.get("guardrail_config") + + kwargs = build_converse_kwargs( + model=model, + messages=messages, + tools=tools, + max_tokens=params.get("max_tokens", 4096), + temperature=params.get("temperature"), + guardrail_config=guardrail, + ) + # Sentinel keys for dispatch — agent pops these before the boto3 call + kwargs["__bedrock_converse__"] = True + kwargs["__bedrock_region__"] = region + return kwargs + + def normalize_response(self, response: Any, **kwargs) -> NormalizedResponse: + """Normalize Bedrock response to NormalizedResponse. + + Handles two shapes: + 1. Raw boto3 dict (from direct converse() calls) + 2. Already-normalized SimpleNamespace with .choices (from dispatch site) + """ + from agent.bedrock_adapter import normalize_converse_response + + # Normalize to OpenAI-compatible SimpleNamespace + if hasattr(response, "choices") and response.choices: + # Already normalized at dispatch site + ns = response + else: + # Raw boto3 dict + ns = normalize_converse_response(response) + + choice = ns.choices[0] + msg = choice.message + finish_reason = choice.finish_reason or "stop" + + tool_calls = None + if msg.tool_calls: + tool_calls = [ + ToolCall( + id=tc.id, + name=tc.function.name, + arguments=tc.function.arguments, + ) + for tc in msg.tool_calls + ] + + usage = None + if hasattr(ns, "usage") and ns.usage: + u = ns.usage + usage = Usage( + prompt_tokens=getattr(u, "prompt_tokens", 0) or 0, + completion_tokens=getattr(u, "completion_tokens", 0) or 0, + total_tokens=getattr(u, "total_tokens", 0) or 0, + ) + + reasoning = getattr(msg, "reasoning", None) or getattr(msg, "reasoning_content", None) + + return NormalizedResponse( + content=msg.content, + tool_calls=tool_calls, + finish_reason=finish_reason, + reasoning=reasoning, + usage=usage, + ) + + def validate_response(self, response: Any) -> bool: + """Check Bedrock response structure. + + After normalize_converse_response, the response has OpenAI-compatible + .choices — same check as chat_completions. + """ + if response is None: + return False + # Raw Bedrock dict response — check for 'output' key + if isinstance(response, dict): + return "output" in response + # Already-normalized SimpleNamespace + if hasattr(response, "choices"): + return bool(response.choices) + return False + + def map_finish_reason(self, raw_reason: str) -> str: + """Map Bedrock stop reason to OpenAI finish_reason. + + The adapter already does this mapping inside normalize_converse_response, + so this is only used for direct access to raw responses. + """ + _MAP = { + "end_turn": "stop", + "tool_use": "tool_calls", + "max_tokens": "length", + "stop_sequence": "stop", + "guardrail_intervened": "content_filter", + "content_filtered": "content_filter", + } + return _MAP.get(raw_reason, "stop") + + +# Auto-register on import +from agent.transports import register_transport # noqa: E402 + +register_transport("bedrock_converse", BedrockTransport) diff --git a/agent/transports/chat_completions.py b/agent/transports/chat_completions.py new file mode 100644 index 000000000000..900f59dcf475 --- /dev/null +++ b/agent/transports/chat_completions.py @@ -0,0 +1,387 @@ +"""OpenAI Chat Completions transport. + +Handles the default api_mode ('chat_completions') used by ~16 OpenAI-compatible +providers (OpenRouter, Nous, NVIDIA, Qwen, Ollama, DeepSeek, xAI, Kimi, etc.). + +Messages and tools are already in OpenAI format — convert_messages and +convert_tools are near-identity. The complexity lives in build_kwargs +which has provider-specific conditionals for max_tokens defaults, +reasoning configuration, temperature handling, and extra_body assembly. +""" + +import copy +from typing import Any, Dict, List, Optional + +from agent.prompt_builder import DEVELOPER_ROLE_MODELS +from agent.transports.base import ProviderTransport +from agent.transports.types import NormalizedResponse, ToolCall, Usage + + +class ChatCompletionsTransport(ProviderTransport): + """Transport for api_mode='chat_completions'. + + The default path for OpenAI-compatible providers. + """ + + @property + def api_mode(self) -> str: + return "chat_completions" + + def convert_messages(self, messages: List[Dict[str, Any]], **kwargs) -> List[Dict[str, Any]]: + """Messages are already in OpenAI format — sanitize Codex leaks only. + + Strips Codex Responses API fields (``codex_reasoning_items`` on the + message, ``call_id``/``response_item_id`` on tool_calls) that strict + chat-completions providers reject with 400/422. + """ + needs_sanitize = False + for msg in messages: + if not isinstance(msg, dict): + continue + if "codex_reasoning_items" in msg: + needs_sanitize = True + break + tool_calls = msg.get("tool_calls") + if isinstance(tool_calls, list): + for tc in tool_calls: + if isinstance(tc, dict) and ("call_id" in tc or "response_item_id" in tc): + needs_sanitize = True + break + if needs_sanitize: + break + + if not needs_sanitize: + return messages + + sanitized = copy.deepcopy(messages) + for msg in sanitized: + if not isinstance(msg, dict): + continue + msg.pop("codex_reasoning_items", None) + tool_calls = msg.get("tool_calls") + if isinstance(tool_calls, list): + for tc in tool_calls: + if isinstance(tc, dict): + tc.pop("call_id", None) + tc.pop("response_item_id", None) + return sanitized + + def convert_tools(self, tools: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Tools are already in OpenAI format — identity.""" + return tools + + def build_kwargs( + self, + model: str, + messages: List[Dict[str, Any]], + tools: Optional[List[Dict[str, Any]]] = None, + **params, + ) -> Dict[str, Any]: + """Build chat.completions.create() kwargs. + + This is the most complex transport method — it handles ~16 providers + via params rather than subclasses. + + params: + timeout: float — API call timeout + max_tokens: int | None — user-configured max tokens + ephemeral_max_output_tokens: int | None — one-shot override (error recovery) + max_tokens_param_fn: callable — returns {max_tokens: N} or {max_completion_tokens: N} + reasoning_config: dict | None + request_overrides: dict | None + session_id: str | None + qwen_session_metadata: dict | None — {sessionId, promptId} precomputed + model_lower: str — lowercase model name for pattern matching + # Provider detection flags (all optional, default False) + is_openrouter: bool + is_nous: bool + is_qwen_portal: bool + is_github_models: bool + is_nvidia_nim: bool + is_kimi: bool + is_custom_provider: bool + ollama_num_ctx: int | None + # Provider routing + provider_preferences: dict | None + # Qwen-specific + qwen_prepare_fn: callable | None — runs AFTER codex sanitization + qwen_prepare_inplace_fn: callable | None — in-place variant for deepcopied lists + # Temperature + fixed_temperature: Any — from _fixed_temperature_for_model() + omit_temperature: bool + # Reasoning + supports_reasoning: bool + github_reasoning_extra: dict | None + # Claude on OpenRouter/Nous max output + anthropic_max_output: int | None + # Extra + extra_body_additions: dict | None — pre-built extra_body entries + """ + # Codex sanitization: drop reasoning_items / call_id / response_item_id + sanitized = self.convert_messages(messages) + + # Qwen portal prep AFTER codex sanitization. If sanitize already + # deepcopied, reuse that copy via the in-place variant to avoid a + # second deepcopy. + is_qwen = params.get("is_qwen_portal", False) + if is_qwen: + qwen_prep = params.get("qwen_prepare_fn") + qwen_prep_inplace = params.get("qwen_prepare_inplace_fn") + if sanitized is messages: + if qwen_prep is not None: + sanitized = qwen_prep(sanitized) + else: + # Already deepcopied — transform in place + if qwen_prep_inplace is not None: + qwen_prep_inplace(sanitized) + elif qwen_prep is not None: + sanitized = qwen_prep(sanitized) + + # Developer role swap for GPT-5/Codex models + model_lower = params.get("model_lower", (model or "").lower()) + if ( + sanitized + and isinstance(sanitized[0], dict) + and sanitized[0].get("role") == "system" + and any(p in model_lower for p in DEVELOPER_ROLE_MODELS) + ): + sanitized = list(sanitized) + sanitized[0] = {**sanitized[0], "role": "developer"} + + api_kwargs: Dict[str, Any] = { + "model": model, + "messages": sanitized, + } + + timeout = params.get("timeout") + if timeout is not None: + api_kwargs["timeout"] = timeout + + # Temperature + fixed_temp = params.get("fixed_temperature") + omit_temp = params.get("omit_temperature", False) + if omit_temp: + api_kwargs.pop("temperature", None) + elif fixed_temp is not None: + api_kwargs["temperature"] = fixed_temp + + # Qwen metadata (caller precomputes {sessionId, promptId}) + qwen_meta = params.get("qwen_session_metadata") + if qwen_meta and is_qwen: + api_kwargs["metadata"] = qwen_meta + + # Tools + if tools: + api_kwargs["tools"] = tools + + # max_tokens resolution — priority: ephemeral > user > provider default + max_tokens_fn = params.get("max_tokens_param_fn") + ephemeral = params.get("ephemeral_max_output_tokens") + max_tokens = params.get("max_tokens") + anthropic_max_out = params.get("anthropic_max_output") + is_nvidia_nim = params.get("is_nvidia_nim", False) + is_kimi = params.get("is_kimi", False) + reasoning_config = params.get("reasoning_config") + + if ephemeral is not None and max_tokens_fn: + api_kwargs.update(max_tokens_fn(ephemeral)) + elif max_tokens is not None and max_tokens_fn: + api_kwargs.update(max_tokens_fn(max_tokens)) + elif is_nvidia_nim and max_tokens_fn: + api_kwargs.update(max_tokens_fn(16384)) + elif is_qwen and max_tokens_fn: + api_kwargs.update(max_tokens_fn(65536)) + elif is_kimi and max_tokens_fn: + # Kimi/Moonshot: 32000 matches Kimi CLI's default + api_kwargs.update(max_tokens_fn(32000)) + elif anthropic_max_out is not None: + api_kwargs["max_tokens"] = anthropic_max_out + + # Kimi: top-level reasoning_effort (unless thinking disabled) + if is_kimi: + _kimi_thinking_off = bool( + reasoning_config + and isinstance(reasoning_config, dict) + and reasoning_config.get("enabled") is False + ) + if not _kimi_thinking_off: + _kimi_effort = "medium" + if reasoning_config and isinstance(reasoning_config, dict): + _e = (reasoning_config.get("effort") or "").strip().lower() + if _e in ("low", "medium", "high"): + _kimi_effort = _e + api_kwargs["reasoning_effort"] = _kimi_effort + + # extra_body assembly + extra_body: Dict[str, Any] = {} + + is_openrouter = params.get("is_openrouter", False) + is_nous = params.get("is_nous", False) + is_github_models = params.get("is_github_models", False) + + provider_prefs = params.get("provider_preferences") + if provider_prefs and is_openrouter: + extra_body["provider"] = provider_prefs + + # Kimi extra_body.thinking + if is_kimi: + _kimi_thinking_enabled = True + if reasoning_config and isinstance(reasoning_config, dict): + if reasoning_config.get("enabled") is False: + _kimi_thinking_enabled = False + extra_body["thinking"] = { + "type": "enabled" if _kimi_thinking_enabled else "disabled", + } + + # Reasoning + if params.get("supports_reasoning", False): + if is_github_models: + gh_reasoning = params.get("github_reasoning_extra") + if gh_reasoning is not None: + extra_body["reasoning"] = gh_reasoning + else: + if reasoning_config is not None: + rc = dict(reasoning_config) + if is_nous and rc.get("enabled") is False: + pass # omit for Nous when disabled + else: + extra_body["reasoning"] = rc + else: + extra_body["reasoning"] = {"enabled": True, "effort": "medium"} + + if is_nous: + extra_body["tags"] = ["product=hermes-agent"] + + # Ollama num_ctx + ollama_ctx = params.get("ollama_num_ctx") + if ollama_ctx: + options = extra_body.get("options", {}) + options["num_ctx"] = ollama_ctx + extra_body["options"] = options + + # Ollama/custom think=false + if params.get("is_custom_provider", False): + if reasoning_config and isinstance(reasoning_config, dict): + _effort = (reasoning_config.get("effort") or "").strip().lower() + _enabled = reasoning_config.get("enabled", True) + if _effort == "none" or _enabled is False: + extra_body["think"] = False + + if is_qwen: + extra_body["vl_high_resolution_images"] = True + + # Merge any pre-built extra_body additions + additions = params.get("extra_body_additions") + if additions: + extra_body.update(additions) + + if extra_body: + api_kwargs["extra_body"] = extra_body + + # Request overrides last (service_tier etc.) + overrides = params.get("request_overrides") + if overrides: + api_kwargs.update(overrides) + + return api_kwargs + + def normalize_response(self, response: Any, **kwargs) -> NormalizedResponse: + """Normalize OpenAI ChatCompletion to NormalizedResponse. + + For chat_completions, this is near-identity — the response is already + in OpenAI format. extra_content on tool_calls (Gemini thought_signature) + is preserved via ToolCall.provider_data. reasoning_details (OpenRouter + unified format) and reasoning_content (DeepSeek/Moonshot) are also + preserved for downstream replay. + """ + choice = response.choices[0] + msg = choice.message + finish_reason = choice.finish_reason or "stop" + + tool_calls = None + if msg.tool_calls: + tool_calls = [] + for tc in msg.tool_calls: + # Preserve provider-specific extras on the tool call. + # Gemini 3 thinking models attach extra_content with + # thought_signature — without replay on the next turn the API + # rejects the request with 400. + tc_provider_data: Dict[str, Any] = {} + extra = getattr(tc, "extra_content", None) + if extra is None and hasattr(tc, "model_extra"): + extra = (tc.model_extra or {}).get("extra_content") + if extra is not None: + if hasattr(extra, "model_dump"): + try: + extra = extra.model_dump() + except Exception: + pass + tc_provider_data["extra_content"] = extra + tool_calls.append(ToolCall( + id=tc.id, + name=tc.function.name, + arguments=tc.function.arguments, + provider_data=tc_provider_data or None, + )) + + usage = None + if hasattr(response, "usage") and response.usage: + u = response.usage + usage = Usage( + prompt_tokens=getattr(u, "prompt_tokens", 0) or 0, + completion_tokens=getattr(u, "completion_tokens", 0) or 0, + total_tokens=getattr(u, "total_tokens", 0) or 0, + ) + + # Preserve reasoning fields separately. DeepSeek/Moonshot use + # ``reasoning_content``; others use ``reasoning``. Downstream code + # (_extract_reasoning, thinking-prefill retry) reads both distinctly, + # so keep them apart in provider_data rather than merging. + reasoning = getattr(msg, "reasoning", None) + reasoning_content = getattr(msg, "reasoning_content", None) + + provider_data: Dict[str, Any] = {} + if reasoning_content: + provider_data["reasoning_content"] = reasoning_content + rd = getattr(msg, "reasoning_details", None) + if rd: + provider_data["reasoning_details"] = rd + + return NormalizedResponse( + content=msg.content, + tool_calls=tool_calls, + finish_reason=finish_reason, + reasoning=reasoning, + usage=usage, + provider_data=provider_data or None, + ) + + def validate_response(self, response: Any) -> bool: + """Check that response has valid choices.""" + if response is None: + return False + if not hasattr(response, "choices") or response.choices is None: + return False + if not response.choices: + return False + return True + + def extract_cache_stats(self, response: Any) -> Optional[Dict[str, int]]: + """Extract OpenRouter/OpenAI cache stats from prompt_tokens_details.""" + usage = getattr(response, "usage", None) + if usage is None: + return None + details = getattr(usage, "prompt_tokens_details", None) + if details is None: + return None + cached = getattr(details, "cached_tokens", 0) or 0 + written = getattr(details, "cache_write_tokens", 0) or 0 + if cached or written: + return {"cached_tokens": cached, "creation_tokens": written} + return None + + +# Auto-register on import +from agent.transports import register_transport # noqa: E402 + +register_transport("chat_completions", ChatCompletionsTransport) diff --git a/agent/transports/codex.py b/agent/transports/codex.py new file mode 100644 index 000000000000..ec48352193db --- /dev/null +++ b/agent/transports/codex.py @@ -0,0 +1,217 @@ +"""OpenAI Responses API (Codex) transport. + +Delegates to the existing adapter functions in agent/codex_responses_adapter.py. +This transport owns format conversion and normalization — NOT client lifecycle, +streaming, or the _run_codex_stream() call path. +""" + +from typing import Any, Dict, List, Optional + +from agent.transports.base import ProviderTransport +from agent.transports.types import NormalizedResponse, ToolCall, Usage + + +class ResponsesApiTransport(ProviderTransport): + """Transport for api_mode='codex_responses'. + + Wraps the functions extracted into codex_responses_adapter.py (PR 1). + """ + + @property + def api_mode(self) -> str: + return "codex_responses" + + def convert_messages(self, messages: List[Dict[str, Any]], **kwargs) -> Any: + """Convert OpenAI chat messages to Responses API input items.""" + from agent.codex_responses_adapter import _chat_messages_to_responses_input + return _chat_messages_to_responses_input(messages) + + def convert_tools(self, tools: List[Dict[str, Any]]) -> Any: + """Convert OpenAI tool schemas to Responses API function definitions.""" + from agent.codex_responses_adapter import _responses_tools + return _responses_tools(tools) + + def build_kwargs( + self, + model: str, + messages: List[Dict[str, Any]], + tools: Optional[List[Dict[str, Any]]] = None, + **params, + ) -> Dict[str, Any]: + """Build Responses API kwargs. + + Calls convert_messages and convert_tools internally. + + params: + instructions: str — system prompt (extracted from messages[0] if not given) + reasoning_config: dict | None — {effort, enabled} + session_id: str | None — used for prompt_cache_key + xAI conv header + max_tokens: int | None — max_output_tokens + request_overrides: dict | None — extra kwargs merged in + provider: str | None — provider name for backend-specific logic + base_url: str | None — endpoint URL + base_url_hostname: str | None — hostname for backend detection + is_github_responses: bool — Copilot/GitHub models backend + is_codex_backend: bool — chatgpt.com/backend-api/codex + is_xai_responses: bool — xAI/Grok backend + github_reasoning_extra: dict | None — Copilot reasoning params + """ + from agent.codex_responses_adapter import ( + _chat_messages_to_responses_input, + _responses_tools, + ) + + from run_agent import DEFAULT_AGENT_IDENTITY + + instructions = params.get("instructions", "") + payload_messages = messages + if not instructions: + if messages and messages[0].get("role") == "system": + instructions = str(messages[0].get("content") or "").strip() + payload_messages = messages[1:] + if not instructions: + instructions = DEFAULT_AGENT_IDENTITY + + is_github_responses = params.get("is_github_responses", False) + is_codex_backend = params.get("is_codex_backend", False) + is_xai_responses = params.get("is_xai_responses", False) + + # Resolve reasoning effort + reasoning_effort = "medium" + reasoning_enabled = True + reasoning_config = params.get("reasoning_config") + if reasoning_config and isinstance(reasoning_config, dict): + if reasoning_config.get("enabled") is False: + reasoning_enabled = False + elif reasoning_config.get("effort"): + reasoning_effort = reasoning_config["effort"] + + _effort_clamp = {"minimal": "low"} + reasoning_effort = _effort_clamp.get(reasoning_effort, reasoning_effort) + + kwargs = { + "model": model, + "instructions": instructions, + "input": _chat_messages_to_responses_input(payload_messages), + "tools": _responses_tools(tools), + "tool_choice": "auto", + "parallel_tool_calls": True, + "store": False, + } + + session_id = params.get("session_id") + if not is_github_responses and session_id: + kwargs["prompt_cache_key"] = session_id + + if reasoning_enabled and is_xai_responses: + kwargs["include"] = ["reasoning.encrypted_content"] + elif reasoning_enabled: + if is_github_responses: + github_reasoning = params.get("github_reasoning_extra") + if github_reasoning is not None: + kwargs["reasoning"] = github_reasoning + else: + kwargs["reasoning"] = {"effort": reasoning_effort, "summary": "auto"} + kwargs["include"] = ["reasoning.encrypted_content"] + elif not is_github_responses and not is_xai_responses: + kwargs["include"] = [] + + request_overrides = params.get("request_overrides") + if request_overrides: + kwargs.update(request_overrides) + + max_tokens = params.get("max_tokens") + if max_tokens is not None and not is_codex_backend: + kwargs["max_output_tokens"] = max_tokens + + if is_xai_responses and session_id: + kwargs["extra_headers"] = {"x-grok-conv-id": session_id} + + return kwargs + + def normalize_response(self, response: Any, **kwargs) -> NormalizedResponse: + """Normalize Codex Responses API response to NormalizedResponse.""" + from agent.codex_responses_adapter import ( + _normalize_codex_response, + _extract_responses_message_text, + _extract_responses_reasoning_text, + ) + + # _normalize_codex_response returns (SimpleNamespace, finish_reason_str) + msg, finish_reason = _normalize_codex_response(response) + + tool_calls = None + if msg and msg.tool_calls: + tool_calls = [] + for tc in msg.tool_calls: + provider_data = {} + if hasattr(tc, "call_id") and tc.call_id: + provider_data["call_id"] = tc.call_id + if hasattr(tc, "response_item_id") and tc.response_item_id: + provider_data["response_item_id"] = tc.response_item_id + tool_calls.append(ToolCall( + id=tc.id if hasattr(tc, "id") else (tc.function.name if hasattr(tc, "function") else None), + name=tc.function.name if hasattr(tc, "function") else getattr(tc, "name", ""), + arguments=tc.function.arguments if hasattr(tc, "function") else getattr(tc, "arguments", "{}"), + provider_data=provider_data or None, + )) + + # Extract reasoning items for provider_data + provider_data = {} + if msg and hasattr(msg, "codex_reasoning_items") and msg.codex_reasoning_items: + provider_data["codex_reasoning_items"] = msg.codex_reasoning_items + if msg and hasattr(msg, "reasoning_details") and msg.reasoning_details: + provider_data["reasoning_details"] = msg.reasoning_details + + return NormalizedResponse( + content=msg.content if msg else None, + tool_calls=tool_calls, + finish_reason=finish_reason or "stop", + reasoning=msg.reasoning if msg and hasattr(msg, "reasoning") else None, + usage=None, # Codex usage is extracted separately in normalize_usage() + provider_data=provider_data or None, + ) + + def validate_response(self, response: Any) -> bool: + """Check Codex Responses API response has valid output structure. + + Returns True only if response.output is a non-empty list. + Does NOT check output_text fallback — the caller handles that + with diagnostic logging for stream backfill recovery. + """ + if response is None: + return False + output = getattr(response, "output", None) + if not isinstance(output, list) or not output: + return False + return True + + def preflight_kwargs(self, api_kwargs: Any, *, allow_stream: bool = False) -> dict: + """Validate and sanitize Codex API kwargs before the call. + + Normalizes input items, strips unsupported fields, validates structure. + """ + from agent.codex_responses_adapter import _preflight_codex_api_kwargs + return _preflight_codex_api_kwargs(api_kwargs, allow_stream=allow_stream) + + def map_finish_reason(self, raw_reason: str) -> str: + """Map Codex response.status to OpenAI finish_reason. + + Codex uses response.status ('completed', 'incomplete') + + response.incomplete_details.reason for granular mapping. + This method handles the simple status string; the caller + should check incomplete_details separately for 'max_output_tokens'. + """ + _MAP = { + "completed": "stop", + "incomplete": "length", + "failed": "stop", + "cancelled": "stop", + } + return _MAP.get(raw_reason, "stop") + + +# Auto-register on import +from agent.transports import register_transport # noqa: E402 + +register_transport("codex_responses", ResponsesApiTransport) diff --git a/agent/transports/types.py b/agent/transports/types.py new file mode 100644 index 000000000000..74481f85cd8f --- /dev/null +++ b/agent/transports/types.py @@ -0,0 +1,156 @@ +"""Shared types for normalized provider responses. + +These dataclasses define the canonical shape that all provider adapters +normalize responses to. The shared surface is intentionally minimal — +only fields that every downstream consumer reads are top-level. +Protocol-specific state goes in ``provider_data`` dicts (response-level +and per-tool-call) so that protocol-aware code paths can access it +without polluting the shared type. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional + + +@dataclass +class ToolCall: + """A normalized tool call from any provider. + + ``id`` is the protocol's canonical identifier — what gets used in + ``tool_call_id`` / ``tool_use_id`` when constructing tool result + messages. May be ``None`` when the provider omits it; the agent + fills it via ``_deterministic_call_id()`` before storing in history. + + ``provider_data`` carries per-tool-call protocol metadata that only + protocol-aware code reads: + + * Codex: ``{"call_id": "call_XXX", "response_item_id": "fc_XXX"}`` + * Gemini: ``{"extra_content": {"google": {"thought_signature": "..."}}}`` + * Others: ``None`` + """ + + id: Optional[str] + name: str + arguments: str # JSON string + provider_data: Optional[Dict[str, Any]] = field(default=None, repr=False) + + # ── Backward compatibility ────────────────────────────────── + # The agent loop reads tc.function.name / tc.function.arguments + # throughout run_agent.py (45+ sites). These properties let + # NormalizedResponse pass through without the _nr_to_assistant_message + # shim, while keeping ToolCall's canonical fields flat. + @property + def type(self) -> str: + return "function" + + @property + def function(self) -> "ToolCall": + """Return self so tc.function.name / tc.function.arguments work.""" + return self + + @property + def call_id(self) -> Optional[str]: + """Codex call_id from provider_data, accessed via getattr by _build_assistant_message.""" + return (self.provider_data or {}).get("call_id") + + @property + def response_item_id(self) -> Optional[str]: + """Codex response_item_id from provider_data.""" + return (self.provider_data or {}).get("response_item_id") + + @property + def extra_content(self) -> Optional[Dict[str, Any]]: + """Gemini extra_content (thought_signature) from provider_data. + + Gemini 3 thinking models attach ``extra_content`` with a + ``thought_signature`` to each tool call. This signature must be + replayed on subsequent API calls — without it the API rejects the + request with HTTP 400. The chat_completions transport stores this + in ``provider_data["extra_content"]``; this property exposes it so + ``_build_assistant_message`` can ``getattr(tc, "extra_content")`` + uniformly. + """ + return (self.provider_data or {}).get("extra_content") + + +@dataclass +class Usage: + """Token usage from an API response.""" + + prompt_tokens: int = 0 + completion_tokens: int = 0 + total_tokens: int = 0 + cached_tokens: int = 0 + + +@dataclass +class NormalizedResponse: + """Normalized API response from any provider. + + Shared fields are truly cross-provider — every caller can rely on + them without branching on api_mode. Protocol-specific state goes in + ``provider_data`` so that only protocol-aware code paths read it. + + Response-level ``provider_data`` examples: + + * Anthropic: ``{"reasoning_details": [...]}`` + * Codex: ``{"codex_reasoning_items": [...]}`` + * Others: ``None`` + """ + + content: Optional[str] + tool_calls: Optional[List[ToolCall]] + finish_reason: str # "stop", "tool_calls", "length", "content_filter" + reasoning: Optional[str] = None + usage: Optional[Usage] = None + provider_data: Optional[Dict[str, Any]] = field(default=None, repr=False) + + # ── Backward compatibility ────────────────────────────────── + # The shim _nr_to_assistant_message() mapped these from provider_data. + # These properties let NormalizedResponse pass through directly. + @property + def reasoning_content(self) -> Optional[str]: + pd = self.provider_data or {} + return pd.get("reasoning_content") + + @property + def reasoning_details(self): + pd = self.provider_data or {} + return pd.get("reasoning_details") + + @property + def codex_reasoning_items(self): + pd = self.provider_data or {} + return pd.get("codex_reasoning_items") + + +# --------------------------------------------------------------------------- +# Factory helpers +# --------------------------------------------------------------------------- + +def build_tool_call( + id: Optional[str], + name: str, + arguments: Any, + **provider_fields: Any, +) -> ToolCall: + """Build a ``ToolCall``, auto-serialising *arguments* if it's a dict. + + Any extra keyword arguments are collected into ``provider_data``. + """ + args_str = json.dumps(arguments) if isinstance(arguments, dict) else str(arguments) + pd = dict(provider_fields) if provider_fields else None + return ToolCall(id=id, name=name, arguments=args_str, provider_data=pd) + + +def map_finish_reason(reason: Optional[str], mapping: Dict[str, str]) -> str: + """Translate a provider-specific stop reason to the normalised set. + + Falls back to ``"stop"`` for unknown or ``None`` reasons. + """ + if reason is None: + return "stop" + return mapping.get(reason, "stop") diff --git a/agent/usage_pricing.py b/agent/usage_pricing.py index 2b04eab625c4..1dfe59ea327c 100644 --- a/agent/usage_pricing.py +++ b/agent/usage_pricing.py @@ -6,6 +6,7 @@ from typing import Any, Dict, Literal, Optional from agent.model_metadata import fetch_endpoint_model_metadata, fetch_model_metadata +from utils import base_url_host_matches DEFAULT_PRICING = {"input": 0.0, "output": 0.0} @@ -284,6 +285,80 @@ class CostResult: source_url="https://ai.google.dev/pricing", pricing_version="google-pricing-2026-03-16", ), + # AWS Bedrock — pricing per the Bedrock pricing page. + # Bedrock charges the same per-token rates as the model provider but + # through AWS billing. These are the on-demand prices (no commitment). + # Source: https://aws.amazon.com/bedrock/pricing/ + ( + "bedrock", + "anthropic.claude-opus-4-6", + ): PricingEntry( + input_cost_per_million=Decimal("15.00"), + output_cost_per_million=Decimal("75.00"), + source="official_docs_snapshot", + source_url="https://aws.amazon.com/bedrock/pricing/", + pricing_version="bedrock-pricing-2026-04", + ), + ( + "bedrock", + "anthropic.claude-sonnet-4-6", + ): PricingEntry( + input_cost_per_million=Decimal("3.00"), + output_cost_per_million=Decimal("15.00"), + source="official_docs_snapshot", + source_url="https://aws.amazon.com/bedrock/pricing/", + pricing_version="bedrock-pricing-2026-04", + ), + ( + "bedrock", + "anthropic.claude-sonnet-4-5", + ): PricingEntry( + input_cost_per_million=Decimal("3.00"), + output_cost_per_million=Decimal("15.00"), + source="official_docs_snapshot", + source_url="https://aws.amazon.com/bedrock/pricing/", + pricing_version="bedrock-pricing-2026-04", + ), + ( + "bedrock", + "anthropic.claude-haiku-4-5", + ): PricingEntry( + input_cost_per_million=Decimal("0.80"), + output_cost_per_million=Decimal("4.00"), + source="official_docs_snapshot", + source_url="https://aws.amazon.com/bedrock/pricing/", + pricing_version="bedrock-pricing-2026-04", + ), + ( + "bedrock", + "amazon.nova-pro", + ): PricingEntry( + input_cost_per_million=Decimal("0.80"), + output_cost_per_million=Decimal("3.20"), + source="official_docs_snapshot", + source_url="https://aws.amazon.com/bedrock/pricing/", + pricing_version="bedrock-pricing-2026-04", + ), + ( + "bedrock", + "amazon.nova-lite", + ): PricingEntry( + input_cost_per_million=Decimal("0.06"), + output_cost_per_million=Decimal("0.24"), + source="official_docs_snapshot", + source_url="https://aws.amazon.com/bedrock/pricing/", + pricing_version="bedrock-pricing-2026-04", + ), + ( + "bedrock", + "amazon.nova-micro", + ): PricingEntry( + input_cost_per_million=Decimal("0.035"), + output_cost_per_million=Decimal("0.14"), + source="official_docs_snapshot", + source_url="https://aws.amazon.com/bedrock/pricing/", + pricing_version="bedrock-pricing-2026-04", + ), } @@ -319,7 +394,7 @@ def resolve_billing_route( if provider_name == "openai-codex": return BillingRoute(provider="openai-codex", model=model, base_url=base_url or "", billing_mode="subscription_included") - if provider_name == "openrouter" or "openrouter.ai" in base: + if provider_name == "openrouter" or base_url_host_matches(base_url or "", "openrouter.ai"): return BillingRoute(provider="openrouter", model=model, base_url=base_url or "", billing_mode="official_models_api") if provider_name == "anthropic": return BillingRoute(provider="anthropic", model=model.split("/")[-1], base_url=base_url or "", billing_mode="official_docs_snapshot") @@ -458,10 +533,22 @@ def normalize_usage( prompt_total = _to_int(getattr(response_usage, "prompt_tokens", 0)) output_tokens = _to_int(getattr(response_usage, "completion_tokens", 0)) details = getattr(response_usage, "prompt_tokens_details", None) + # Primary: OpenAI-style prompt_tokens_details. Fallback: Anthropic-style + # top-level fields that some OpenAI-compatible proxies (OpenRouter, Vercel + # AI Gateway, Cline) expose when routing Claude models — without this + # fallback, cache writes are undercounted as 0 and cache reads can be + # missed when the proxy only surfaces them at the top level. + # Port of cline/cline#10266. cache_read_tokens = _to_int(getattr(details, "cached_tokens", 0) if details else 0) + if not cache_read_tokens: + cache_read_tokens = _to_int(getattr(response_usage, "cache_read_input_tokens", 0)) cache_write_tokens = _to_int( getattr(details, "cache_write_tokens", 0) if details else 0 ) + if not cache_write_tokens: + cache_write_tokens = _to_int( + getattr(response_usage, "cache_creation_input_tokens", 0) + ) input_tokens = max(0, prompt_total - cache_read_tokens - cache_write_tokens) reasoning_tokens = 0 @@ -575,25 +662,6 @@ def has_known_pricing( return entry is not None -def get_pricing( - model_name: str, - provider: Optional[str] = None, - base_url: Optional[str] = None, - api_key: Optional[str] = None, -) -> Dict[str, float]: - """Backward-compatible thin wrapper for legacy callers. - - Returns only non-cache input/output fields when a pricing entry exists. - Unknown routes return zeroes. - """ - entry = get_pricing_entry(model_name, provider=provider, base_url=base_url, api_key=api_key) - if not entry: - return {"input": 0.0, "output": 0.0} - return { - "input": float(entry.input_cost_per_million or _ZERO), - "output": float(entry.output_cost_per_million or _ZERO), - } - def format_duration_compact(seconds: float) -> str: if seconds < 60: diff --git a/batch_runner.py b/batch_runner.py index 195452c0ae0c..7413ad59f481 100644 --- a/batch_runner.py +++ b/batch_runner.py @@ -444,6 +444,7 @@ def _process_batch_worker(args: Tuple) -> Dict[str, Any]: if not reasoning.get("has_any_reasoning", True): print(f" 🚫 Prompt {prompt_index} discarded (no reasoning in any turn)") discarded_no_reasoning += 1 + completed_in_batch.append(prompt_index) continue # Get and normalize tool stats for consistent schema across all entries @@ -561,7 +562,10 @@ def __init__( provider_sort (str): Sort providers by price/throughput/latency (optional) max_tokens (int): Maximum tokens for model responses (optional, uses model default if not set) reasoning_config (Dict): OpenRouter reasoning config override (e.g. {"effort": "none"} to disable thinking) - prefill_messages (List[Dict]): Messages to prepend as prefilled conversation context (few-shot priming) + prefill_messages (List[Dict]): Messages to prepend as prefilled conversation context (few-shot priming). + NOTE: Anthropic Sonnet 4.6+ and Opus 4.6+ reject a trailing assistant-role prefill + (400 error). For those models use output_config.format or structured-output + schemas instead. Safe here for user-role priming and for older Claude / non-Claude models. max_samples (int): Only process the first N samples from the dataset (optional, processes all if not set) """ self.dataset_file = Path(dataset_file) @@ -1186,12 +1190,12 @@ def main( """ # Handle list distributions if list_distributions: - from toolset_distributions import list_distributions as get_all_dists, print_distribution_info - + from toolset_distributions import print_distribution_info + print("📊 Available Toolset Distributions") print("=" * 70) - - all_dists = get_all_dists() + + all_dists = list_distributions() for dist_name in sorted(all_dists.keys()): print_distribution_info(dist_name) diff --git a/cli-config.yaml.example b/cli-config.yaml.example index 637e45f13fd2..bd63901e122c 100644 --- a/cli-config.yaml.example +++ b/cli-config.yaml.example @@ -16,7 +16,7 @@ model: # "nous" - Nous Portal OAuth (requires: hermes login) # "nous-api" - Nous Portal API key (requires: NOUS_API_KEY) # "anthropic" - Direct Anthropic API (requires: ANTHROPIC_API_KEY) - # "openai-codex" - OpenAI Codex (requires: hermes login --provider openai-codex) + # "openai-codex" - OpenAI Codex (requires: hermes auth) # "copilot" - GitHub Copilot / GitHub Models (requires: GITHUB_TOKEN) # "gemini" - Use Google AI Studio direct (requires: GOOGLE_API_KEY or GEMINI_API_KEY) # "zai" - Use z.ai / ZhipuAI GLM models (requires: GLM_API_KEY) @@ -24,7 +24,10 @@ model: # "minimax" - MiniMax global (requires: MINIMAX_API_KEY) # "minimax-cn" - MiniMax China (requires: MINIMAX_CN_API_KEY) # "huggingface" - Hugging Face Inference (requires: HF_TOKEN) + # "nvidia" - NVIDIA NIM / build.nvidia.com (requires: NVIDIA_API_KEY) # "xiaomi" - Xiaomi MiMo (requires: XIAOMI_API_KEY) + # "arcee" - Arcee AI Trinity models (requires: ARCEEAI_API_KEY) + # "ollama-cloud" - Ollama Cloud (requires: OLLAMA_API_KEY — https://ollama.com/settings) # "kilocode" - KiloCode gateway (requires: KILOCODE_API_KEY) # "ai-gateway" - Vercel AI Gateway (requires: AI_GATEWAY_API_KEY) # @@ -36,12 +39,6 @@ model: # base_url: "http://localhost:1234/v1" # No API key needed — local servers typically ignore auth. # - # For Ollama Cloud (https://ollama.com/pricing): - # provider: "custom" - # base_url: "https://ollama.com/v1" - # Set OLLAMA_API_KEY in .env — automatically picked up when base_url - # points to ollama.com. - # # Can also be overridden with --provider flag or HERMES_INFERENCE_PROVIDER env var. provider: "auto" @@ -66,7 +63,38 @@ model: # Leave unset to use the model's native output ceiling (recommended). # Set only if you want to deliberately limit individual response length. # - # max_tokens: 8192 +# max_tokens: 8192 + +# Named provider overrides (optional) +# Use this for per-provider request timeouts, non-stream stale timeouts, +# and per-model exceptions. +# Applies to the primary turn client on every api_mode (OpenAI-wire, native +# Anthropic, and Anthropic-compatible providers), the fallback chain, and +# client rebuilds during credential rotation. For OpenAI-wire chat +# completions (streaming and non-streaming) the configured value is also +# used as the per-request ``timeout=`` kwarg so it wins over the legacy +# HERMES_API_TIMEOUT env var (which still applies when no config is set). +# ``stale_timeout_seconds`` controls the non-streaming stale-call detector and +# wins over the legacy HERMES_API_CALL_STALE_TIMEOUT env var. Leaving these +# unset keeps the legacy defaults (HERMES_API_TIMEOUT=1800s, +# HERMES_API_CALL_STALE_TIMEOUT=300s, native Anthropic 900s). +# +# Not currently wired for AWS Bedrock (bedrock_converse + AnthropicBedrock +# SDK paths) — those use boto3 with its own timeout configuration. +# +# providers: +# ollama-local: +# request_timeout_seconds: 300 # Longer timeout for local cold-starts +# stale_timeout_seconds: 900 # Explicitly re-enable stale detection on local endpoints +# anthropic: +# request_timeout_seconds: 30 # Fast-fail cloud requests +# models: +# claude-opus-4.6: +# timeout_seconds: 600 # Longer timeout for extended-thinking Opus calls +# openai-codex: +# models: +# gpt-5.4: +# stale_timeout_seconds: 1800 # Longer non-stream stale timeout for slow large-context turns # ============================================================================= # OpenRouter Provider Routing (only applies when using OpenRouter) @@ -94,20 +122,6 @@ model: # # Data policy: "allow" (default) or "deny" to exclude providers that may store data # # data_collection: "deny" -# ============================================================================= -# Smart Model Routing (optional) -# ============================================================================= -# Use a cheaper model for short/simple turns while keeping your main model for -# more complex requests. Disabled by default. -# -# smart_model_routing: -# enabled: true -# max_simple_chars: 160 -# max_simple_words: 28 -# cheap_model: -# provider: openrouter -# model: google/gemini-2.5-flash - # ============================================================================= # Git Worktree Isolation # ============================================================================= @@ -336,6 +350,7 @@ compression: # "openrouter" - Force OpenRouter (requires OPENROUTER_API_KEY) # "nous" - Force Nous Portal (requires: hermes login) # "gemini" - Force Google AI Studio direct (requires: GOOGLE_API_KEY or GEMINI_API_KEY) +# "ollama-cloud" - Ollama Cloud (requires: OLLAMA_API_KEY) # "codex" - Force Codex OAuth (requires: hermes model → Codex). # Uses gpt-5.3-codex which supports vision. # "main" - Use your custom endpoint (OPENAI_BASE_URL + OPENAI_API_KEY). @@ -359,6 +374,18 @@ compression: # web_extract: # provider: "auto" # model: "" +# +# # Session search — summarizes matching past sessions +# session_search: +# provider: "auto" +# model: "" +# timeout: 30 +# max_concurrency: 3 # Limit parallel summaries to reduce request-burst 429s +# extra_body: {} # Provider-specific OpenAI-compatible request fields +# # Example for providers that support request-body +# # reasoning controls: +# # extra_body: +# # enable_thinking: false # ============================================================================= # Persistent Memory @@ -480,6 +507,13 @@ agent: # finish, then interrupts anything still running after this timeout. # 0 = no drain, interrupt immediately. # restart_drain_timeout: 60 + + # Max app-level retry attempts for API errors (connection drops, provider + # timeouts, 5xx, etc.) before the agent surfaces the failure. Lower this + # to 1 if you use fallback providers and want fast failover on flaky + # primaries (default 3). The OpenAI SDK does its own low-level retries + # underneath this wrapper — this is the Hermes-level loop. + # api_max_retries: 3 # Enable verbose logging verbose: false @@ -522,7 +556,7 @@ agent: # - A preset like "hermes-cli" or "hermes-telegram" (curated tool set) # - A list of individual toolsets to compose your own (see list below) # -# Supported platform keys: cli, telegram, discord, whatsapp, slack +# Supported platform keys: cli, telegram, discord, whatsapp, slack, qqbot # # Examples: # @@ -551,6 +585,7 @@ agent: # slack: hermes-slack (same as telegram) # signal: hermes-signal (same as telegram) # homeassistant: hermes-homeassistant (same as telegram) +# qqbot: hermes-qqbot (same as telegram) # platform_toolsets: cli: [hermes-cli] @@ -560,6 +595,19 @@ platform_toolsets: slack: [hermes-slack] signal: [hermes-signal] homeassistant: [hermes-homeassistant] + qqbot: [hermes-qqbot] + +# ============================================================================= +# Gateway Platform Settings +# ============================================================================= +# Optional per-platform messaging settings. +# Platform-specific knobs live under `extra`. +# +# platforms: +# telegram: +# reply_to_mode: "first" # off | first | all +# extra: +# disable_link_previews: false # Set true to suppress Telegram URL previews in bot messages # ───────────────────────────────────────────────────────────────────────────── # Available toolsets (use these names in platform_toolsets or the toolsets list) @@ -729,10 +777,13 @@ code_execution: # Subagent Delegation # ============================================================================= # The delegate_task tool spawns child agents with isolated context. -# Supports single tasks and batch mode (up to 3 parallel). +# Supports single tasks and batch mode (default 3 parallel, configurable). delegation: max_iterations: 50 # Max tool-calling turns per child (default: 50) - default_toolsets: ["terminal", "file", "web"] # Default toolsets for subagents + # max_concurrent_children: 3 # Max parallel child agents (default: 3) + # max_spawn_depth: 1 # Tree depth cap (1-3, default: 1 = flat). Raise to 2 or 3 to allow orchestrator children to spawn their own workers. + # orchestrator_enabled: true # Kill switch for role="orchestrator" children (default: true). + # inherit_mcp_toolsets: true # When explicit child toolsets are narrowed, also keep the parent's MCP toolsets (default: true). Set false for strict intersection. # model: "google/gemini-3-flash-preview" # Override model for subagents (empty = inherit parent) # provider: "openrouter" # Override provider for subagents (empty = inherit parent) # # Resolves full credentials (base_url, api_key) automatically. @@ -876,3 +927,39 @@ display: # # Names and usernames are NOT affected (user-chosen, publicly visible). # # Routing/delivery still uses the original values internally. # redact_pii: false + +# ============================================================================= +# Shell-script hooks +# ============================================================================= +# Register shell scripts as plugin-hook callbacks. Each entry is executed as +# a subprocess (shell=False, shlex.split) with a JSON payload on stdin. On +# stdout the script may return JSON that either blocks the tool call or +# injects context into the next LLM call. +# +# Valid events (mirror hermes_cli.plugins.VALID_HOOKS): +# pre_tool_call, post_tool_call, pre_llm_call, post_llm_call, +# pre_api_request, post_api_request, on_session_start, on_session_end, +# on_session_finalize, on_session_reset, subagent_stop +# +# First-use consent: each (event, command) pair prompts once on a TTY, then +# is persisted to ~/.hermes/shell-hooks-allowlist.json. Non-interactive +# runs (gateway, cron) need --accept-hooks, HERMES_ACCEPT_HOOKS=1, or the +# hooks_auto_accept key below. +# +# See website/docs/user-guide/features/hooks.md for the full JSON wire +# protocol and worked examples. +# +# hooks: +# pre_tool_call: +# - matcher: "terminal" +# command: "~/.hermes/agent-hooks/block-rm-rf.sh" +# timeout: 10 +# post_tool_call: +# - matcher: "write_file|patch" +# command: "~/.hermes/agent-hooks/auto-format.sh" +# pre_llm_call: +# - command: "~/.hermes/agent-hooks/inject-cwd-context.sh" +# subagent_stop: +# - command: "~/.hermes/agent-hooks/log-orchestration.sh" +# +# hooks_auto_accept: false diff --git a/cli.py b/cli.py index dcb5bfcc5ff4..a289e3ab2374 100644 --- a/cli.py +++ b/cli.py @@ -18,11 +18,15 @@ import shutil import sys import json +import re +import concurrent.futures +import base64 import atexit import tempfile import time import uuid import textwrap +from urllib.parse import unquote, urlparse from contextlib import contextmanager from pathlib import Path from datetime import datetime @@ -63,6 +67,7 @@ format_duration_compact, format_token_count_compact, ) +from agent.account_usage import fetch_account_usage, render_account_usage_lines from hermes_cli.banner import _format_context_length, format_banner_version_label _COMMAND_SPINNER_FRAMES = ("⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏") @@ -72,12 +77,113 @@ # User-managed env files should override stale shell exports on restart. from hermes_constants import get_hermes_home, display_hermes_home from hermes_cli.env_loader import load_hermes_dotenv +from utils import base_url_host_matches _hermes_home = get_hermes_home() _project_env = Path(__file__).parent / '.env' load_hermes_dotenv(hermes_home=_hermes_home, project_env=_project_env) +_REASONING_TAGS = ( + "REASONING_SCRATCHPAD", + "think", + "thinking", + "reasoning", + "thought", +) + + +def _strip_reasoning_tags(text: str) -> str: + """Remove reasoning/thinking blocks from displayed text. + + Handles every case: + * Closed pairs ``…`` (case-insensitive, multi-line). + * Unterminated open tags that run to end-of-text (e.g. truncated + generations on NIM/MiniMax where the close tag is dropped). + * Stray orphan close tags (``stuffanswer``) left behind by + partial-content dumps. + + Covers the variants emitted by reasoning models today: ````, + ````, ````, ````, and + ```` (Gemma 4). Must stay in sync with + ``run_agent.py::_strip_think_blocks`` and the stream consumer's + ``_OPEN_THINK_TAGS`` / ``_CLOSE_THINK_TAGS`` tuples. + + Also strips tool-call XML blocks some open models leak into visible + content (````, ````, Gemma-style + ``…``). Ported from + openclaw/openclaw#67318. + """ + cleaned = text + for tag in _REASONING_TAGS: + # Closed pair — case-insensitive so … is handled too. + cleaned = re.sub( + rf"<{tag}>.*?\s*", + "", + cleaned, + flags=re.DOTALL | re.IGNORECASE, + ) + # Unterminated open tag — strip from the tag to end of text. + cleaned = re.sub( + rf"<{tag}>.*$", + "", + cleaned, + flags=re.DOTALL | re.IGNORECASE, + ) + # Stray orphan close tag left behind by partial dumps. + cleaned = re.sub( + rf"\s*", + "", + cleaned, + flags=re.IGNORECASE, + ) + # Tool-call XML blocks (openclaw/openclaw#67318). + for tc_tag in ("tool_call", "tool_calls", "tool_result", + "function_call", "function_calls"): + cleaned = re.sub( + rf"<{tc_tag}\b[^>]*>.*?\s*", + "", + cleaned, + flags=re.DOTALL | re.IGNORECASE, + ) + # — boundary + attribute gated to avoid prose FPs. + cleaned = re.sub( + r'(?:(?<=^)|(?<=[\n\r.!?:]))[ \t]*' + r']*\bname\s*=[^>]*>' + r'(?:(?:(?!).)*)\s*', + '', + cleaned, + flags=re.DOTALL | re.IGNORECASE, + ) + # Stray tool-call close tags. + cleaned = re.sub( + r'\s*', + '', + cleaned, + flags=re.IGNORECASE, + ) + return cleaned.strip() + + +def _assistant_content_as_text(content: Any) -> str: + if content is None: + return "" + if isinstance(content, str): + return content + if isinstance(content, list): + parts = [ + str(part.get("text", "")) + for part in content + if isinstance(part, dict) and part.get("type") == "text" + ] + return "\n".join(p for p in parts if p) + return str(content) + + +def _assistant_copy_text(content: Any) -> str: + return _strip_reasoning_tags(_assistant_content_as_text(content)) + + # ============================================================================= # Configuration Loading # ============================================================================= @@ -199,13 +305,23 @@ def load_cli_config() -> Dict[str, Any]: Environment variables take precedence over config file values. Returns default values if no config file exists. + + If HERMES_IGNORE_USER_CONFIG=1 is set (via ``hermes chat --ignore-user-config``), + the user config at ``~/.hermes/config.yaml`` is skipped entirely and only the + built-in defaults plus the project-level ``cli-config.yaml`` (if any) are used. + Credentials in ``.env`` are still loaded — this flag only suppresses + behavioral/config settings. """ # Check user config first ({HERMES_HOME}/config.yaml) user_config_path = _hermes_home / 'config.yaml' project_config_path = Path(__file__).parent / 'cli-config.yaml' + # --ignore-user-config: force-skip the user config.yaml (still honor project + # config as a fallback so defaults stay sensible). + ignore_user_config = os.environ.get("HERMES_IGNORE_USER_CONFIG") == "1" + # Use user config if it exists, otherwise project config - if user_config_path.exists(): + if user_config_path.exists() and not ignore_user_config: config_path = user_config_path else: config_path = project_config_path @@ -238,12 +354,6 @@ def load_cli_config() -> Dict[str, Any]: "enabled": True, # Auto-compress when approaching context limit "threshold": 0.50, # Compress at 50% of model's context limit }, - "smart_model_routing": { - "enabled": False, - "max_simple_chars": 160, - "max_simple_words": 28, - "cheap_model": {}, - }, "agent": { "max_turns": 90, # Default max tool-calling iterations (shared with subagents) "verbose": False, @@ -301,7 +411,6 @@ def load_cli_config() -> Dict[str, Any]: }, "delegation": { "max_iterations": 45, # Max tool-calling turns per child agent - "default_toolsets": ["terminal", "file", "web"], # Default toolsets for subagents "model": "", # Subagent model override (empty = inherit parent model) "provider": "", # Subagent provider override (empty = inherit parent provider) "base_url": "", # Direct OpenAI-compatible endpoint for subagents @@ -401,14 +510,27 @@ def load_cli_config() -> Dict[str, Any]: # filesystem is directly accessible. For ALL remote/container backends # (ssh, docker, modal, singularity), the host path doesn't exist on the # target -- remove the key so terminal_tool.py uses its per-backend default. - if terminal_config.get("cwd") in (".", "auto", "cwd"): - effective_backend = terminal_config.get("env_type", "local") - if effective_backend == "local": - terminal_config["cwd"] = os.getcwd() - defaults["terminal"]["cwd"] = terminal_config["cwd"] + # + # GUARD: If TERMINAL_CWD is already set to a real absolute path (by the + # gateway's config bridge earlier in the process), don't clobber it. + # This prevents a lazy import of cli.py during gateway runtime from + # rewriting TERMINAL_CWD to the service's working directory. + # See issue #10817. + _CWD_PLACEHOLDERS = (".", "auto", "cwd") + if terminal_config.get("cwd") in _CWD_PLACEHOLDERS: + _existing_cwd = os.environ.get("TERMINAL_CWD", "") + if _existing_cwd and _existing_cwd not in _CWD_PLACEHOLDERS and os.path.isabs(_existing_cwd): + # Gateway (or earlier startup) already resolved a real path — keep it + terminal_config["cwd"] = _existing_cwd + defaults["terminal"]["cwd"] = _existing_cwd else: - # Remove so TERMINAL_CWD stays unset → tool picks backend default - terminal_config.pop("cwd", None) + effective_backend = terminal_config.get("env_type", "local") + if effective_backend == "local": + terminal_config["cwd"] = os.getcwd() + defaults["terminal"]["cwd"] = terminal_config["cwd"] + else: + # Remove so TERMINAL_CWD stays unset → tool picks backend default + terminal_config.pop("cwd", None) env_mappings = { "env_type": "TERMINAL_ENV", @@ -449,7 +571,6 @@ def load_cli_config() -> Dict[str, Any]: if _file_has_terminal_config or env_var not in os.environ: val = terminal_config[config_key] if isinstance(val, list): - import json os.environ[env_var] = json.dumps(val) else: os.environ[env_var] = str(val) @@ -833,6 +954,32 @@ def _cleanup_worktree(info: Dict[str, str] = None) -> None: print(f"\033[32m✓ Worktree cleaned up: {wt_path}\033[0m") +def _run_state_db_auto_maintenance(session_db) -> None: + """Call ``SessionDB.maybe_auto_prune_and_vacuum`` using current config. + + Reads the ``sessions:`` section from config.yaml via + :func:`hermes_cli.config.load_config` (the authoritative loader that + deep-merges DEFAULT_CONFIG, so unmigrated configs still get default + values). Honours ``auto_prune`` / ``retention_days`` / + ``vacuum_after_prune`` / ``min_interval_hours``, and delegates to the + DB. Never raises — maintenance must never block interactive startup. + """ + if session_db is None: + return + try: + from hermes_cli.config import load_config as _load_full_config + cfg = (_load_full_config().get("sessions") or {}) + if not cfg.get("auto_prune", False): + return + session_db.maybe_auto_prune_and_vacuum( + retention_days=int(cfg.get("retention_days", 90)), + min_interval_hours=int(cfg.get("min_interval_hours", 24)), + vacuum=bool(cfg.get("vacuum_after_prune", True)), + ) + except Exception as exc: + logger.debug("state.db auto-maintenance skipped: %s", exc) + + def _prune_stale_worktrees(repo_root: str, max_age_hours: int = 24) -> None: """Remove stale worktrees and orphaned branches on startup. @@ -988,19 +1135,20 @@ def _prune_orphaned_branches(repo_root: str) -> None: # ANSI building blocks for conversation display _ACCENT_ANSI_DEFAULT = "\033[1;38;2;255;215;0m" # True-color #FFD700 bold — fallback _BOLD = "\033[1m" -_DIM = "\033[2m" _RST = "\033[0m" +_STREAM_PAD = " " # 4-space indent for streamed response text (matches Panel padding) -def _hex_to_ansi_bold(hex_color: str) -> str: - """Convert a hex color like '#268bd2' to a bold true-color ANSI escape.""" +def _hex_to_ansi(hex_color: str, *, bold: bool = False) -> str: + """Convert a hex color like '#268bd2' to a true-color ANSI escape.""" try: r = int(hex_color[1:3], 16) g = int(hex_color[3:5], 16) b = int(hex_color[5:7], 16) - return f"\033[1;38;2;{r};{g};{b}m" + prefix = "1;" if bold else "" + return f"\033[{prefix}38;2;{r};{g};{b}m" except (ValueError, IndexError): - return _ACCENT_ANSI_DEFAULT + return _ACCENT_ANSI_DEFAULT if bold else "\033[38;2;184;134;11m" class _SkinAwareAnsi: @@ -1010,20 +1158,22 @@ class _SkinAwareAnsi: force re-resolution after a ``/skin`` switch. """ - def __init__(self, skin_key: str, fallback_hex: str = "#FFD700"): + def __init__(self, skin_key: str, fallback_hex: str = "#FFD700", *, bold: bool = False): self._skin_key = skin_key self._fallback_hex = fallback_hex + self._bold = bold self._cached: str | None = None def __str__(self) -> str: if self._cached is None: try: from hermes_cli.skin_engine import get_active_skin - self._cached = _hex_to_ansi_bold( - get_active_skin().get_color(self._skin_key, self._fallback_hex) + self._cached = _hex_to_ansi( + get_active_skin().get_color(self._skin_key, self._fallback_hex), + bold=self._bold, ) except Exception: - self._cached = _hex_to_ansi_bold(self._fallback_hex) + self._cached = _hex_to_ansi(self._fallback_hex, bold=self._bold) return self._cached def __add__(self, other: str) -> str: @@ -1037,7 +1187,8 @@ def reset(self) -> None: self._cached = None -_ACCENT = _SkinAwareAnsi("response_border", "#FFD700") +_ACCENT = _SkinAwareAnsi("response_border", "#FFD700", bold=True) +_DIM = _SkinAwareAnsi("banner_dim", "#B8860B") def _accent_hex() -> str: @@ -1058,6 +1209,41 @@ def _rich_text_from_ansi(text: str) -> _RichText: return _RichText.from_ansi(text or "") +def _strip_markdown_syntax(text: str) -> str: + """Best-effort markdown marker removal for plain-text display.""" + plain = _rich_text_from_ansi(text or "").plain + plain = re.sub(r"^\s{0,3}(?:[-*_]\s*){3,}$", "", plain, flags=re.MULTILINE) + plain = re.sub(r"^\s{0,3}#{1,6}\s+", "", plain, flags=re.MULTILINE) + # Preserve blockquotes, lists, and checkboxes because they carry structure. + plain = re.sub(r"(```+|~~~+)", "", plain) + plain = re.sub(r"`([^`]*)`", r"\1", plain) + plain = re.sub(r"!\[([^\]]*)\]\([^\)]*\)", r"\1", plain) + plain = re.sub(r"\[([^\]]+)\]\([^\)]*\)", r"\1", plain) + plain = re.sub(r"\*\*\*([^*]+)\*\*\*", r"\1", plain) + plain = re.sub(r"(? Path | None: if (token.startswith('"') and token.endswith('"')) or (token.startswith("'") and token.endswith("'")): token = token[1:-1].strip() + token = token.replace('\\ ', ' ') if not token: return None - expanded = os.path.expandvars(os.path.expanduser(token)) + expanded = token + if token.startswith("file://"): + try: + parsed = urlparse(token) + if parsed.scheme == "file": + expanded = unquote(parsed.path or "") + if parsed.netloc and os.name == "nt": + expanded = f"//{parsed.netloc}{expanded}" + except Exception: + expanded = token + expanded = os.path.expandvars(os.path.expanduser(expanded)) + if os.name != "nt": + normalized = expanded.replace("\\", "/") + if len(normalized) >= 3 and normalized[1] == ":" and normalized[2] == "/" and normalized[0].isalpha(): + expanded = f"/mnt/{normalized[0].lower()}/{normalized[3:]}" path = Path(expanded) if not path.is_absolute(): base_dir = Path(os.getenv("TERMINAL_CWD", os.getcwd())) @@ -1237,16 +1438,36 @@ def _detect_file_drop(user_input: str) -> "dict | None": or stripped.startswith("~") or stripped.startswith("./") or stripped.startswith("../") + or stripped.startswith("file://") + or (len(stripped) >= 3 and stripped[1] == ":" and stripped[2] in ("\\", "/") and stripped[0].isalpha()) or stripped.startswith('"/') or stripped.startswith('"~') or stripped.startswith("'/") or stripped.startswith("'~") + or (len(stripped) >= 4 and stripped[0] in ("'", '"') and stripped[2] == ":" and stripped[3] in ("\\", "/") and stripped[1].isalpha()) ) if not starts_like_path: return None + direct_path = _resolve_attachment_path(stripped) + if direct_path is not None: + return { + "path": direct_path, + "is_image": direct_path.suffix.lower() in _IMAGE_EXTENSIONS, + "remainder": "", + } + first_token, remainder = _split_path_input(stripped) drop_path = _resolve_attachment_path(first_token) + if drop_path is None and " " in stripped and stripped[0] not in {"'", '"'}: + space_positions = [idx for idx, ch in enumerate(stripped) if ch == " "] + for pos in reversed(space_positions): + candidate = stripped[:pos].rstrip() + resolved = _resolve_attachment_path(candidate) + if resolved is not None: + drop_path = resolved + remainder = stripped[pos + 1 :].strip() + break if drop_path is None: return None @@ -1591,6 +1812,7 @@ def __init__( resume: str = None, checkpoints: bool = False, pass_session_id: bool = False, + ignore_rules: bool = False, ): """ Initialize the Hermes CLI. @@ -1629,10 +1851,30 @@ def __init__( # streaming: stream tokens to the terminal as they arrive (display.streaming in config.yaml) self.streaming_enabled = CLI_CONFIG["display"].get("streaming", False) + self.final_response_markdown = str( + CLI_CONFIG["display"].get("final_response_markdown", "strip") + ).strip().lower() or "strip" + if self.final_response_markdown not in {"render", "strip", "raw"}: + self.final_response_markdown = "strip" # Inline diff previews for write actions (display.inline_diffs in config.yaml) self._inline_diffs_enabled = CLI_CONFIG["display"].get("inline_diffs", True) + # Submitted multiline user-message preview (display.user_message_preview in config.yaml) + _ump = CLI_CONFIG["display"].get("user_message_preview", {}) + if not isinstance(_ump, dict): + _ump = {} + try: + _ump_first_lines = int(_ump.get("first_lines", 2)) + except (TypeError, ValueError): + _ump_first_lines = 2 + try: + _ump_last_lines = int(_ump.get("last_lines", 2)) + except (TypeError, ValueError): + _ump_last_lines = 2 + self.user_message_preview_first_lines = max(1, _ump_first_lines) + self.user_message_preview_last_lines = max(0, _ump_last_lines) + # Streaming display state self._stream_buf = "" # Partial line buffer for line-buffered rendering self._stream_started = False # True once first delta arrives @@ -1690,7 +1932,7 @@ def __init__( # Match key to resolved base_url: OpenRouter URL → prefer OPENROUTER_API_KEY, # custom endpoint → prefer OPENAI_API_KEY (issue #560). # Note: _ensure_runtime_credentials() re-resolves this before first use. - if self.base_url and "openrouter.ai" in self.base_url: + if self.base_url and base_url_host_matches(self.base_url, "openrouter.ai"): self.api_key = api_key or os.getenv("OPENROUTER_API_KEY") or os.getenv("OPENAI_API_KEY") else: self.api_key = api_key or os.getenv("OPENAI_API_KEY") or os.getenv("OPENROUTER_API_KEY") @@ -1709,13 +1951,13 @@ def __init__( # Parse and validate toolsets self.enabled_toolsets = toolsets if toolsets and "all" not in toolsets and "*" not in toolsets: - # Validate each toolset — MCP server names are added by - # _get_platform_tools() but aren't registered in TOOLSETS yet - # (that happens later in _sync_mcp_toolsets), so exclude them. + # Validate each toolset — MCP server names are resolved via + # live registry aliases (registered during discover_mcp_tools), + # but discovery hasn't run yet at this point, so exclude them. mcp_names = set((CLI_CONFIG.get("mcp_servers") or {}).keys()) invalid = [t for t in toolsets if not validate_toolset(t) and t not in mcp_names] if invalid: - self.console.print(f"[bold red]Warning: Unknown toolsets: {', '.join(invalid)}[/]") + self._console_print(f"[bold red]Warning: Unknown toolsets: {', '.join(invalid)}[/]") # Filesystem checkpoints: CLI flag > config cp_cfg = CLI_CONFIG.get("checkpoints", {}) @@ -1724,6 +1966,11 @@ def __init__( self.checkpoints_enabled = checkpoints or cp_cfg.get("enabled", False) self.checkpoint_max_snapshots = cp_cfg.get("max_snapshots", 50) self.pass_session_id = pass_session_id + # --ignore-rules: honor either the constructor flag or the env var set + # by `hermes chat --ignore-rules` in hermes_cli/main.py. When true we + # pass skip_context_files=True and skip_memory=True to AIAgent so + # AGENTS.md/SOUL.md/.cursorrules and persistent memory are not loaded. + self.ignore_rules = ignore_rules or os.environ.get("HERMES_IGNORE_RULES") == "1" # Ephemeral system prompt: env var takes precedence, then config self.system_prompt = ( @@ -1762,8 +2009,9 @@ def __init__( fb = [fb] if fb.get("provider") and fb.get("model") else [] self._fallback_model = fb - # Optional cheap-vs-strong routing for simple turns - self._smart_model_routing = CLI_CONFIG.get("smart_model_routing", {}) or {} + # Signature of the currently-initialised agent's runtime. Used to + # rebuild the agent when provider / model / base_url changes across + # turns (e.g. after /model or credential rotation). self._active_agent_route_signature = None # Agent will be initialized on first use @@ -1774,6 +2022,10 @@ def __init__( self.conversation_history: List[Dict[str, Any]] = [] self.session_start = datetime.now() self._resumed = False + # Per-prompt elapsed timer — started at the beginning of each chat turn, + # frozen when the agent thread completes, displayed in the status bar. + self._prompt_start_time: Optional[float] = None # time.time() when turn started + self._prompt_duration: float = 0.0 # frozen duration of last completed turn # Initialize SQLite session store early so /title works before first message self._session_db = None try: @@ -1781,7 +2033,13 @@ def __init__( self._session_db = SessionDB() except Exception as e: logger.warning("Failed to initialize SessionDB — session will NOT be indexed for search: %s", e) - + + # Opportunistic state.db maintenance — runs at most once per + # min_interval_hours, tracked via state_meta in state.db itself so + # it's shared across all Hermes processes for this HERMES_HOME. + # Never blocks startup on failure. + _run_state_db_auto_maintenance(self._session_db) + # Deferred title: stored in memory until the session is created in the DB self._pending_title: Optional[str] = None @@ -1850,8 +2108,7 @@ def __init__( def _invalidate(self, min_interval: float = 0.25) -> None: """Throttled UI repaint — prevents terminal blinking on slow/SSH connections.""" - import time as _time - now = _time.monotonic() + now = time.monotonic() if hasattr(self, "_app") and self._app and (now - self._last_invalidate) >= min_interval: self._last_invalidate = now self._app.invalidate() @@ -1872,6 +2129,44 @@ def _build_context_bar(self, percent_used: Optional[int], width: int = 10) -> st filled = round((safe_percent / 100) * width) return f"[{('█' * filled) + ('░' * max(0, width - filled))}]" + @staticmethod + def _format_prompt_elapsed(prompt_start_time: Optional[float], prompt_duration: float, live: bool = False) -> str: + """Format per-prompt elapsed time for the status bar. + + Always returns a string — shows 0s on fresh start before first turn. + Keeps seconds visible at all scales so it increments smoothly: + 59s → 1m → 1m 1s → ... → 1m 59s → 2m → 2m 1s → ... + 59m 59s → 1h → 1h 0m 1s → ... + 23h 59m 59s → 1d → 1d 0h 1m → ... + + Emoji prefix: ⏱ when turn is live, ⏲ when frozen or fresh start. + Uses width-1 (no variation selector) glyphs so the status bar stays + aligned in monospace terminals. + """ + if prompt_start_time is None and prompt_duration == 0.0: + return "⏲ 0s" + elapsed = time.time() - prompt_start_time if prompt_start_time is not None else prompt_duration + elapsed = max(0.0, elapsed) + + days = int(elapsed // 86400) + remaining = elapsed % 86400 + hours = int(remaining // 3600) + remaining = remaining % 3600 + minutes = int(remaining // 60) + seconds = int(remaining % 60) + + if days > 0: + time_str = f"{days}d {hours}h {minutes}m" + elif hours > 0: + time_str = f"{hours}h {minutes}m {seconds}s" if seconds else f"{hours}h {minutes}m" + elif minutes > 0: + time_str = f"{minutes}m {seconds}s" if seconds else f"{minutes}m" + else: + time_str = f"{int(elapsed)}s" + + emoji = "⏱" if live else "⏲" + return f"{emoji} {time_str}" + def _get_status_bar_snapshot(self) -> Dict[str, Any]: # Prefer the agent's model name — it updates on fallback. # self.model reflects the originally configured model and never @@ -1890,6 +2185,11 @@ def _get_status_bar_snapshot(self) -> Dict[str, Any]: "model_name": model_name, "model_short": model_short, "duration": format_duration_compact(elapsed_seconds), + "prompt_elapsed": self._format_prompt_elapsed( + getattr(self, "_prompt_start_time", None), + getattr(self, "_prompt_duration", 0.0), + live=getattr(self, "_prompt_start_time", None) is not None, + ), "context_tokens": 0, "context_length": None, "context_percent": None, @@ -2007,9 +2307,33 @@ def _agent_spacer_height(self, width: Optional[int] = None) -> int: def _spinner_widget_height(self, width: Optional[int] = None) -> int: """Return the visible height for the spinner/status text line above the status bar.""" - if not getattr(self, "_spinner_text", ""): + spinner_line = self._render_spinner_text() + if not spinner_line: return 0 - return 0 if self._use_minimal_tui_chrome(width=width) else 1 + if self._use_minimal_tui_chrome(width=width): + return 0 + width = width or self._get_tui_terminal_width() + if width and width > 10: + import math + text_width = self._status_bar_display_width(spinner_line) + return max(1, math.ceil(text_width / width)) + return 1 + + def _render_spinner_text(self) -> str: + """Return the live spinner/status text exactly as rendered in the TUI.""" + txt = getattr(self, "_spinner_text", "") + if not txt: + return "" + t0 = getattr(self, "_tool_start_time", 0) or 0 + if t0 > 0: + elapsed = time.monotonic() - t0 + if elapsed >= 60: + _m, _s = int(elapsed // 60), int(elapsed % 60) + elapsed_str = f"{_m}m {_s}s" + else: + elapsed_str = f"{elapsed:.1f}s" + return f" {txt} ({elapsed_str})" + return f" {txt}" def _get_voice_status_fragments(self, width: Optional[int] = None): """Return the voice status bar fragments for the interactive TUI.""" @@ -2056,6 +2380,9 @@ def _build_status_bar_text(self, width: Optional[int] = None) -> str: parts = [f"⚕ {snapshot['model_short']}", context_label, percent_label] parts.append(duration_label) + prompt_elapsed = snapshot.get("prompt_elapsed") + if prompt_elapsed: + parts.append(prompt_elapsed) return self._trim_status_bar_text(" │ ".join(parts), width) except Exception: return f"⚕ {self.model if getattr(self, 'model', None) else 'Hermes'}" @@ -2114,8 +2441,13 @@ def _get_status_bar_fragments(self): (bar_style, percent_label), ("class:status-bar-dim", " │ "), ("class:status-bar-dim", duration_label), - ("class:status-bar", " "), ] + # Position 7: per-prompt elapsed timer (live or frozen) + prompt_elapsed = snapshot.get("prompt_elapsed") + if prompt_elapsed: + frags.append(("class:status-bar-dim", " │ ")) + frags.append(("class:status-bar-dim", prompt_elapsed)) + frags.append(("class:status-bar", " ")) total_width = sum(self._status_bar_display_width(text) for _, text in frags) if total_width > width: @@ -2141,7 +2473,7 @@ def _normalize_model_for_provider(self, resolved_provider: str) -> bool: normalized_model = normalize_model_for_provider(current_model, resolved_provider) if normalized_model and normalized_model != current_model: if not self._model_is_default: - self.console.print( + self._console_print( f"[yellow]⚠️ Normalized model '{current_model}' to '{normalized_model}' for {resolved_provider}.[/]" ) self.model = normalized_model @@ -2157,7 +2489,7 @@ def _normalize_model_for_provider(self, resolved_provider: str) -> bool: canonical = normalize_copilot_model_id(current_model, api_key=self.api_key) if canonical and canonical != current_model: if not self._model_is_default: - self.console.print( + self._console_print( f"[yellow]⚠️ Normalized Copilot model '{current_model}' to '{canonical}'.[/]" ) self.model = canonical @@ -2179,7 +2511,7 @@ def _normalize_model_for_provider(self, resolved_provider: str) -> bool: canonical = normalize_opencode_model_id(resolved_provider, current_model) if canonical and canonical != current_model: if not self._model_is_default: - self.console.print( + self._console_print( f"[yellow]⚠️ Stripped provider prefix from '{current_model}'; using '{canonical}' for {resolved_provider}.[/]" ) self.model = canonical @@ -2201,7 +2533,7 @@ def _normalize_model_for_provider(self, resolved_provider: str) -> bool: if "/" in current_model: slug = current_model.split("/", 1)[1] if not self._model_is_default: - self.console.print( + self._console_print( f"[yellow]⚠️ Stripped provider prefix from '{current_model}'; " f"using '{slug}' for OpenAI Codex.[/]" ) @@ -2249,9 +2581,6 @@ def _current_reasoning_callback(self): def _emit_reasoning_preview(self, reasoning_text: str) -> None: """Render a buffered reasoning preview as a single [thinking] block.""" - import re - import textwrap - preview_text = reasoning_text.strip() if not preview_text: return @@ -2334,6 +2663,59 @@ def _flush_reasoning_preview(self, *, force: bool = False) -> None: if flush_text: self._emit_reasoning_preview(flush_text) + def _format_submitted_user_message_preview(self, user_input: str) -> str: + """Format the submitted user-message scrollback preview.""" + lines = user_input.split("\n") + if len(lines) <= 1: + return f"[bold {_accent_hex()}]●[/] [bold]{_escape(user_input)}[/]" + + first_lines = int(getattr(self, "user_message_preview_first_lines", 2)) + last_lines = int(getattr(self, "user_message_preview_last_lines", 2)) + first_lines = max(1, first_lines) + last_lines = max(0, last_lines) + head = lines[:first_lines] + remaining_after_head = max(0, len(lines) - len(head)) + tail_count = min(last_lines, remaining_after_head) + tail = lines[-tail_count:] if tail_count else [] + + hidden_middle_count = len(lines) - len(head) - len(tail) + if hidden_middle_count < 0: + hidden_middle_count = 0 + tail = [] + + preview_lines = [ + f"[bold {_accent_hex()}]●[/] [bold]{_escape(head[0])}[/]" + ] + preview_lines.extend(f"[bold]{_escape(line)}[/]" for line in head[1:]) + + if hidden_middle_count > 0: + noun = "line" if hidden_middle_count == 1 else "lines" + preview_lines.append(f"[dim]... (+{hidden_middle_count} more {noun})[/]") + + preview_lines.extend(f"[bold]{_escape(line)}[/]" for line in tail) + return "\n".join(preview_lines) + + def _expand_paste_references(self, text: str | None) -> str: + """Expand [Pasted text #N -> file] placeholders into file contents.""" + if not isinstance(text, str) or "[Pasted text #" not in text: + return text or "" + paste_ref_re = re.compile(r'\[Pasted text #\d+: \d+ lines \u2192 (.+?)\]') + + def _expand_ref(match): + path = Path(match.group(1)) + return path.read_text(encoding="utf-8") if path.exists() else match.group(0) + + return paste_ref_re.sub(_expand_ref, text) + + def _print_user_message_preview(self, user_input: str) -> None: + """Render a user message using the normal chat scrollback style.""" + ChatConsole().print(f"[{_accent_hex()}]{'─' * 40}[/]") + text = str(user_input or "") + if "\n" in text: + ChatConsole().print(self._format_submitted_user_message_preview(text)) + else: + ChatConsole().print(f"[bold {_accent_hex()}]●[/] [bold]{_escape(text)}[/]") + def _stream_reasoning_delta(self, text: str) -> None: """Stream reasoning/thinking tokens into a dim box above the response. @@ -2577,7 +2959,9 @@ def _emit_stream_text(self, text: str) -> None: _tc = getattr(self, "_stream_text_ansi", "") while "\n" in self._stream_buf: line, self._stream_buf = self._stream_buf.split("\n", 1) - _cprint(f"{_tc}{line}{_RST}" if _tc else line) + if self.final_response_markdown == "strip": + line = _strip_markdown_syntax(line) + _cprint(f"{_STREAM_PAD}{_tc}{line}{_RST}" if _tc else f"{_STREAM_PAD}{line}") def _flush_stream(self) -> None: """Emit any remaining partial line from the stream buffer and close the box.""" @@ -2594,7 +2978,8 @@ def _flush_stream(self) -> None: if self._stream_buf: _tc = getattr(self, "_stream_text_ansi", "") - _cprint(f"{_tc}{self._stream_buf}{_RST}" if _tc else self._stream_buf) + line = _strip_markdown_syntax(self._stream_buf) if self.final_response_markdown == "strip" else self._stream_buf + _cprint(f"{_STREAM_PAD}{_tc}{line}{_RST}" if _tc else f"{_STREAM_PAD}{line}") self._stream_buf = "" # Close the response box @@ -2637,9 +3022,7 @@ def _slow_command_status(self, command: str) -> str: def _command_spinner_frame(self) -> str: """Return the current spinner frame for slow slash commands.""" - import time as _time - - frame_idx = int(_time.monotonic() * 10) % len(_COMMAND_SPINNER_FRAMES) + frame_idx = int(time.monotonic() * 10) % len(_COMMAND_SPINNER_FRAMES) return _COMMAND_SPINNER_FRAMES[frame_idx] @contextmanager @@ -2656,6 +3039,39 @@ def _busy_command(self, status: str): self._command_status = "" self._invalidate(min_interval=0.0) + def _open_external_editor(self, buffer=None) -> bool: + """Open the active input buffer in an external editor.""" + app = getattr(self, "_app", None) + if not app: + _cprint(f"{_DIM}External editor is only available inside the interactive CLI.{_RST}") + return False + if self._command_running: + _cprint(f"{_DIM}Wait for the current command to finish before opening the editor.{_RST}") + return False + if self._sudo_state or self._secret_state or self._approval_state or self._clarify_state: + _cprint(f"{_DIM}Finish the active prompt before opening the editor.{_RST}") + return False + target_buffer = buffer or getattr(app, "current_buffer", None) + if target_buffer is None: + _cprint(f"{_DIM}No active input buffer is available for the external editor.{_RST}") + return False + try: + existing_text = getattr(target_buffer, "text", "") + expanded_text = self._expand_paste_references(existing_text) + if expanded_text != existing_text and hasattr(target_buffer, "text"): + self._skip_paste_collapse = True + target_buffer.text = expanded_text + if hasattr(target_buffer, "cursor_position"): + target_buffer.cursor_position = len(expanded_text) + # Set skip flag (again) so the text-change event fired when the + # editor closes does not re-collapse the returned content. + self._skip_paste_collapse = True + target_buffer.open_in_editor(validate_and_handle=False) + return True + except Exception as exc: + _cprint(f"{_DIM}Failed to open external editor: {exc}{_RST}") + return False + def _ensure_runtime_credentials(self) -> bool: """ Ensure runtime credentials are resolved before agent use. @@ -2763,24 +3179,36 @@ def _ensure_runtime_credentials(self) -> bool: return True def _resolve_turn_agent_config(self, user_message: str) -> dict: - """Resolve model/runtime overrides for a single user turn.""" - from agent.smart_model_routing import resolve_turn_route + """Build the effective model/runtime config for a single user turn. + + Always uses the session's primary model/provider. If the user has + toggled `/fast` on and the current model supports Priority + Processing / Anthropic fast mode, attach `request_overrides` so the + API call is marked accordingly. + """ from hermes_cli.models import resolve_fast_mode_overrides - route = resolve_turn_route( - user_message, - self._smart_model_routing, - { - "model": self.model, - "api_key": self.api_key, - "base_url": self.base_url, - "provider": self.provider, - "api_mode": self.api_mode, - "command": self.acp_command, - "args": list(self.acp_args or []), - "credential_pool": getattr(self, "_credential_pool", None), - }, - ) + runtime = { + "api_key": self.api_key, + "base_url": self.base_url, + "provider": self.provider, + "api_mode": self.api_mode, + "command": self.acp_command, + "args": list(self.acp_args or []), + "credential_pool": getattr(self, "_credential_pool", None), + } + route = { + "model": self.model, + "runtime": runtime, + "signature": ( + self.model, + runtime["provider"], + runtime["base_url"], + runtime["api_mode"], + runtime["command"], + tuple(runtime["args"]), + ), + } service_tier = getattr(self, "service_tier", None) if not service_tier: @@ -2788,13 +3216,13 @@ def _resolve_turn_agent_config(self, user_message: str) -> dict: return route try: - overrides = resolve_fast_mode_overrides(route.get("model")) + overrides = resolve_fast_mode_overrides(route["model"]) except Exception: overrides = None route["request_overrides"] = overrides return route - def _init_agent(self, *, model_override: str = None, runtime_override: dict = None, route_label: str = None, request_overrides: dict | None = None) -> bool: + def _init_agent(self, *, model_override: str = None, runtime_override: dict = None, request_overrides: dict | None = None) -> bool: """ Initialize the agent on first use. When resuming a session, restores conversation history from SQLite. @@ -2900,6 +3328,8 @@ def _init_agent(self, *, model_override: str = None, runtime_override: dict = No checkpoints_enabled=self.checkpoints_enabled, checkpoint_max_snapshots=self.checkpoint_max_snapshots, pass_session_id=self.pass_session_id, + skip_context_files=self.ignore_rules, + skip_memory=self.ignore_rules, tool_progress_callback=self._on_tool_progress, tool_start_callback=self._on_tool_start if self._inline_diffs_enabled else None, tool_complete_callback=self._on_tool_complete if self._inline_diffs_enabled else None, @@ -2950,7 +3380,7 @@ def show_banner(self): use_compact = self.compact or term_width < 80 if use_compact: - self.console.print(_build_compact_banner()) + self._console_print(_build_compact_banner()) self._show_status() else: # Get tools for display @@ -2975,25 +3405,25 @@ def show_banner(self): # Warn about very low context lengths (common with local servers) if ctx_len and ctx_len <= 8192: - self.console.print() - self.console.print( + self._console_print() + self._console_print( f"[yellow]⚠️ Context length is only {ctx_len:,} tokens — " f"this is likely too low for agent use with tools.[/]" ) - self.console.print( + self._console_print( "[dim] Hermes needs 16k–32k minimum. Tool schemas + system prompt alone use ~4k–8k.[/]" ) base_url = getattr(self, "base_url", "") or "" if "11434" in base_url or "ollama" in base_url.lower(): - self.console.print( + self._console_print( "[dim] Ollama fix: OLLAMA_CONTEXT_LENGTH=32768 ollama serve[/]" ) elif "1234" in base_url: - self.console.print( + self._console_print( "[dim] LM Studio fix: Set context length in model settings → reload model[/]" ) else: - self.console.print( + self._console_print( "[dim] Fix: Set model.context_length in config.yaml, or increase your server's context setting[/]" ) @@ -3002,20 +3432,20 @@ def show_banner(self): model_name = getattr(self, "model", "") or "" if is_nous_hermes_non_agentic(model_name): - self.console.print() - self.console.print( + self._console_print() + self._console_print( "[bold yellow]⚠ Nous Research Hermes 3 & 4 models are NOT agentic and are not " "designed for use with Hermes Agent.[/]" ) - self.console.print( + self._console_print( "[dim] They lack tool-calling capabilities required for agent workflows. " "Consider using an agentic model (Claude, GPT, Gemini, DeepSeek, etc.).[/]" ) - self.console.print( + self._console_print( "[dim] Switch with: /model sonnet or /model gpt5[/]" ) - self.console.print() + self._console_print() def _preload_resumed_session(self) -> bool: """Load a resumed session's history from the DB early (before first chat). @@ -3033,10 +3463,10 @@ def _preload_resumed_session(self) -> bool: session_meta = self._session_db.get_session(self.session_id) if not session_meta: - self.console.print( + self._console_print( f"[bold red]Session not found: {self.session_id}[/]" ) - self.console.print( + self._console_print( "[dim]Use a session ID from a previous CLI run " "(hermes sessions list).[/]" ) @@ -3051,7 +3481,7 @@ def _preload_resumed_session(self) -> bool: if session_meta.get("title"): title_part = f' "{session_meta["title"]}"' accent_color = _accent_hex() - self.console.print( + self._console_print( f"[{accent_color}]↻ Resumed session [bold]{self.session_id}[/bold]" f"{title_part} " f"({msg_count} user message{'s' if msg_count != 1 else ''}, " @@ -3059,7 +3489,7 @@ def _preload_resumed_session(self) -> bool: ) else: accent_color = _accent_hex() - self.console.print( + self._console_print( f"[{accent_color}]Session {self.session_id} found but has no " f"messages. Starting fresh.[/]" ) @@ -3098,21 +3528,6 @@ def _display_resumed_history(self): MAX_ASST_LEN = 200 # truncate assistant text MAX_ASST_LINES = 3 # max lines of assistant text - def _strip_reasoning(text: str) -> str: - """Remove ... blocks - from displayed text (reasoning model internal thoughts).""" - import re - cleaned = re.sub( - r".*?\s*", - "", text, flags=re.DOTALL, - ) - # Also strip unclosed reasoning tags at the end - cleaned = re.sub( - r".*$", - "", cleaned, flags=re.DOTALL, - ) - return cleaned.strip() - # Collect displayable entries (skip system, tool-result messages) entries = [] # list of (role, display_text) _last_asst_idx = None # index of last assistant entry @@ -3144,7 +3559,7 @@ def _strip_reasoning(text: str) -> str: elif role == "assistant": text = "" if content is None else str(content) - text = _strip_reasoning(text) + text = _strip_reasoning_tags(text) parts = [] full_parts = [] # un-truncated version if text: @@ -3249,7 +3664,7 @@ def _strip_reasoning(text: str) -> str: padding=(0, 1), style=_history_text_c, ) - self.console.print(panel) + self._console_print(panel) def _try_attach_clipboard_image(self) -> bool: """Check clipboard for an image and attach it if found. @@ -3483,6 +3898,26 @@ def _handle_stop_command(self): killed = process_registry.kill_all() print(f" ✅ Stopped {killed} process(es).") + def _handle_agents_command(self): + """Handle /agents — show background processes and agent status.""" + from tools.process_registry import format_uptime_short, process_registry + + processes = process_registry.list_sessions() + running = [p for p in processes if p.get("status") == "running"] + finished = [p for p in processes if p.get("status") != "running"] + + _cprint(f" Running processes: {len(running)}") + for p in running: + cmd = p.get("command", "")[:80] + up = format_uptime_short(p.get("uptime_seconds", 0)) + _cprint(f" {p.get('session_id', '?')} · {up} · {cmd}") + + if finished: + _cprint(f" Recently finished: {len(finished)}") + + agent_running = getattr(self, "_agent_running", False) + _cprint(f" Agent: {'running' if agent_running else 'idle'}") + def _handle_paste_command(self): """Handle /paste — explicitly check clipboard for an image. @@ -3508,6 +3943,61 @@ def _handle_paste_command(self): else: _cprint(f" {_DIM}(._.) No image found in clipboard{_RST}") + def _write_osc52_clipboard(self, text: str) -> None: + """Copy *text* to terminal clipboard via OSC 52.""" + payload = base64.b64encode(text.encode("utf-8")).decode("ascii") + seq = f"\x1b]52;c;{payload}\x07" + out = getattr(self, "_app", None) + output = getattr(out, "output", None) if out else None + if output and hasattr(output, "write_raw"): + output.write_raw(seq) + output.flush() + return + if output and hasattr(output, "write"): + output.write(seq) + output.flush() + return + sys.stdout.write(seq) + sys.stdout.flush() + + def _handle_copy_command(self, cmd_original: str) -> None: + """Handle /copy [number] — copy assistant output to clipboard.""" + parts = cmd_original.split(maxsplit=1) + arg = parts[1].strip() if len(parts) > 1 else "" + + assistant = [m for m in self.conversation_history if m.get("role") == "assistant"] + if not assistant: + _cprint(" Nothing to copy yet.") + return + + if arg: + try: + idx = int(arg) - 1 + except ValueError: + _cprint(" Usage: /copy [number]") + return + if idx < 0 or idx >= len(assistant): + _cprint(f" Invalid response number. Use 1-{len(assistant)}.") + return + else: + idx = len(assistant) - 1 + while idx >= 0 and not _assistant_copy_text(assistant[idx].get("content")): + idx -= 1 + if idx < 0: + _cprint(" Nothing to copy in assistant responses yet.") + return + + text = _assistant_copy_text(assistant[idx].get("content")) + if not text: + _cprint(" Nothing to copy in that assistant response.") + return + + try: + self._write_osc52_clipboard(text) + _cprint(f" Copied assistant response #{idx + 1} to clipboard") + except Exception as e: + _cprint(f" Clipboard copy failed: {e}") + def _handle_image_command(self, cmd_original: str): """Handle /image — attach a local image file for the next prompt.""" raw_args = (cmd_original.split(None, 1)[1].strip() if " " in cmd_original else "") @@ -3545,7 +4035,6 @@ def _preprocess_images_with_vision(self, text: str, images: list, *, announce: b image later with ``vision_analyze`` if needed. """ import asyncio as _asyncio - import json as _json from tools.vision_tools import vision_analyze_tool analysis_prompt = ( @@ -3565,7 +4054,7 @@ def _preprocess_images_with_vision(self, text: str, images: list, *, announce: b result_json = _asyncio.run( vision_analyze_tool(image_url=str(img_path), user_prompt=analysis_prompt) ) - result = _json.loads(result_json) + result = json.loads(result_json) if result.get("success"): description = result.get("analysis", "") enriched_parts.append( @@ -3610,14 +4099,14 @@ def _show_tool_availability_warnings(self): api_key_missing = [u for u in unavailable if u["missing_vars"]] if api_key_missing: - self.console.print() - self.console.print("[yellow]⚠️ Some tools disabled (missing API keys):[/]") + self._console_print() + self._console_print("[yellow]⚠️ Some tools disabled (missing API keys):[/]") for item in api_key_missing: tools_str = ", ".join(item["tools"][:2]) # Show first 2 tools if len(item["tools"]) > 2: tools_str += f", +{len(item['tools'])-2} more" - self.console.print(f" [dim]• {item['name']}[/] [dim italic]({', '.join(item['missing_vars'])})[/]") - self.console.print("[dim] Run 'hermes setup' to configure[/]") + self._console_print(f" [dim]• {item['name']}[/] [dim italic]({', '.join(item['missing_vars'])})[/]") + self._console_print("[dim] Run 'hermes setup' to configure[/]") except Exception: pass # Don't crash on import errors @@ -3644,7 +4133,7 @@ def _show_status(self): skin = get_active_skin() separator_color = skin.get_color("banner_dim", "#B8860B") accent_color = skin.get_color("ui_accent", "#FFBF00") - label_color = skin.get_color("ui_label", "#4dd0e1") + label_color = skin.get_color("ui_label", "#DAA520") except Exception: separator_color, accent_color, label_color = "#B8860B", "#FFBF00", "cyan" toolsets_info = "" @@ -3655,7 +4144,7 @@ def _show_status(self): if self._provider_source: provider_info += f" [dim {separator_color}]·[/] [dim]auth: {self._provider_source}[/]" - self.console.print( + self._console_print( f" {api_indicator} [{accent_color}]{model_short}[/] " f"[dim {separator_color}]·[/] [bold {label_color}]{tool_count} tools[/]" f"{toolsets_info}{provider_info}" @@ -3712,7 +4201,7 @@ def _show_session_status(self): f"Tokens: {total_tokens:,}", f"Agent Running: {'Yes' if is_running else 'No'}", ]) - self.console.print("\n".join(lines), highlight=False, markup=False) + self._console_print("\n".join(lines), highlight=False, markup=False) def _fast_command_available(self) -> bool: try: @@ -3761,6 +4250,7 @@ def show_help(self): _cprint(f"\n {_DIM}Tip: Just type your message to chat with Hermes!{_RST}") _cprint(f" {_DIM}Multi-line: Alt+Enter for a new line{_RST}") + _cprint(f" {_DIM}Draft editor: Ctrl+G{_RST}") if _is_termux_environment(): _cprint(f" {_DIM}Attach image: /image {_termux_example_image_path()} or start your prompt with a local image path{_RST}\n") else: @@ -3819,8 +4309,37 @@ def _handle_tools_command(self, cmd: str): """ import shlex from argparse import Namespace + from contextlib import redirect_stdout + from io import StringIO from hermes_cli.tools_config import tools_disable_enable_command + def _run_capture(ns: Namespace) -> None: + """Run tools_disable_enable_command, routing its ANSI-colored + print() output through _cprint when inside the interactive TUI + so escapes aren't mangled by patch_stdout's StdoutProxy into + garbled '?[32m...?[0m' text. + + Outside the TUI (standalone mode, tests), call straight through + so real stdout / pytest capture works as expected. + """ + # Standalone/tests, run as usual + if getattr(self, "_app", None) is None: + tools_disable_enable_command(ns) + return + + # Buffer reports isatty()=True so color() in hermes_cli/colors.py + # still emits ANSI escapes. StringIO.isatty() is False, which + # would otherwise strip all colors before we re-render them. + class _TTYBuf(StringIO): + def isatty(self) -> bool: + return True + + buf = _TTYBuf() + with redirect_stdout(buf): + tools_disable_enable_command(ns) + for line in buf.getvalue().splitlines(): + _cprint(line) + try: parts = shlex.split(cmd) except ValueError: @@ -3832,8 +4351,7 @@ def _handle_tools_command(self, cmd: str): return if subcommand == "list": - tools_disable_enable_command( - Namespace(tools_action="list", platform="cli")) + _run_capture(Namespace(tools_action="list", platform="cli")) return names = parts[2:] @@ -3850,8 +4368,7 @@ def _handle_tools_command(self, cmd: str): label = ", ".join(names) _cprint(f"{_ACCENT}{verb} {label}...{_RST}") - tools_disable_enable_command( - Namespace(tools_action=subcommand, names=names, platform="cli")) + _run_capture(Namespace(tools_action=subcommand, names=names, platform="cli")) # Reset session so the new tool config is picked up from a clean state from hermes_cli.tools_config import _get_platform_tools @@ -3893,23 +4410,14 @@ def show_toolsets(self): def _handle_profile_command(self): """Display active profile name and home directory.""" - from hermes_constants import get_hermes_home, display_hermes_home + from hermes_constants import display_hermes_home + from hermes_cli.profiles import get_active_profile_name - home = get_hermes_home() display = display_hermes_home() - - profiles_parent = Path.home() / ".hermes" / "profiles" - try: - rel = home.relative_to(profiles_parent) - profile_name = str(rel).split("/")[0] - except ValueError: - profile_name = None + profile_name = get_active_profile_name() print() - if profile_name: - print(f" Profile: {profile_name}") - else: - print(" Profile: default") + print(f" Profile: {profile_name}") print(f" Home: {display}") print() @@ -4096,6 +4604,8 @@ def new_session(self, silent=False): self.agent.flush_memories(self.conversation_history) except (Exception, KeyboardInterrupt): pass + # Trigger memory extraction on the old session before session_id rotates. + self.agent.commit_memory_session(self.conversation_history) self._notify_session_boundary("on_session_finalize") elif self.agent: # First session or empty history — still finalize the old session @@ -4474,53 +4984,6 @@ def _ask(): _ask() return result[0] - def _interactive_provider_selection( - self, providers: list, current_model: str, current_provider: str - ) -> str | None: - """Show provider picker, return slug or None on cancel.""" - choices = [] - for p in providers: - count = p.get("total_models", len(p.get("models", []))) - label = f"{p['name']} ({count} model{'s' if count != 1 else ''})" - if p.get("is_current"): - label += " ← current" - choices.append(label) - - default_idx = next( - (i for i, p in enumerate(providers) if p.get("is_current")), 0 - ) - - idx = self._run_curses_picker( - f"Select a provider (current: {current_model} on {current_provider}):", - choices, - default_index=default_idx, - ) - if idx is None: - return None - return providers[idx]["slug"] - - def _interactive_model_selection( - self, model_list: list, provider_data: dict - ) -> str | None: - """Show model picker for a given provider, return model_id or None on cancel.""" - pname = provider_data.get("name", provider_data.get("slug", "")) - total = provider_data.get("total_models", len(model_list)) - - if not model_list: - _cprint(f"\n No models listed for {pname}.") - return self._prompt_text_input(" Enter model name manually (or Enter to cancel): ") - - choices = list(model_list) + ["Enter custom model name"] - idx = self._run_curses_picker( - f"Select model from {pname} ({len(model_list)} of {total}):", - choices, - ) - if idx is None: - return None - if idx < len(model_list): - return model_list[idx] - return self._prompt_text_input(" Enter model name: ") - def _open_model_picker(self, providers: list, current_model: str, current_provider: str, user_provs=None, custom_provs=None) -> None: """Open prompt_toolkit-native /model picker modal.""" self._capture_modal_input_snapshot() @@ -4541,6 +5004,34 @@ def _close_model_picker(self) -> None: self._restore_modal_input_snapshot() self._invalidate(min_interval=0.0) + @staticmethod + def _compute_model_picker_viewport( + selected: int, + scroll_offset: int, + n: int, + term_rows: int, + reserved_below: int = 6, + panel_chrome: int = 6, + min_visible: int = 3, + ) -> tuple[int, int]: + """Resolve (scroll_offset, visible) for the /model picker viewport. + + ``reserved_below`` matches the approval / clarify panels — input area, + status bar, and separators below the panel. ``panel_chrome`` covers + this panel's own borders + blanks + hint row. The remaining rows hold + the scrollable list, with the offset slid to keep ``selected`` on screen. + """ + max_visible = max(min_visible, term_rows - reserved_below - panel_chrome) + if n <= max_visible: + return 0, n + visible = max_visible + if selected < scroll_offset: + scroll_offset = selected + elif selected >= scroll_offset + visible: + scroll_offset = selected - visible + 1 + scroll_offset = max(0, min(scroll_offset, n - visible)) + return scroll_offset, visible + def _apply_model_switch_result(self, result, persist_global: bool) -> None: if not result.success: _cprint(f" ✗ {result.error_message}") @@ -4604,7 +5095,7 @@ def _apply_model_switch_result(self, result, persist_global: bool) -> None: pass cache_enabled = ( - ("openrouter" in (result.base_url or "").lower() and "claude" in result.new_model.lower()) + (base_url_host_matches(result.base_url or "", "openrouter.ai") and "claude" in result.new_model.lower()) or result.api_mode == "anthropic_messages" ) if cache_enabled: @@ -4631,16 +5122,19 @@ def _handle_model_picker_selection(self, persist_global: bool = False) -> None: self._close_model_picker() return provider_data = providers[selected] - model_list = [] - try: - from hermes_cli.models import provider_model_ids - live = provider_model_ids(provider_data["slug"]) - if live: - model_list = live - except Exception: - pass + # Use the curated model list from list_authenticated_providers() + # (same lists as `hermes model` and gateway pickers). + # Only fall back to the live provider catalog when the curated + # list is empty (e.g. user-defined endpoints with no curated list). + model_list = provider_data.get("models", []) if not model_list: - model_list = provider_data.get("models", []) + try: + from hermes_cli.models import provider_model_ids + live = provider_model_ids(provider_data["slug"]) + if live: + model_list = live + except Exception: + pass state["stage"] = "model" state["provider_data"] = provider_data state["model_list"] = model_list @@ -4829,7 +5323,7 @@ def _handle_model_switch(self, cmd_original: str): # Cache notice cache_enabled = ( - ("openrouter" in (result.base_url or "").lower() and "claude" in result.new_model.lower()) + (base_url_host_matches(result.base_url or "", "openrouter.ai") and "claude" in result.new_model.lower()) or result.api_mode == "anthropic_messages" ) if cache_enabled: @@ -4860,6 +5354,30 @@ def _should_handle_model_command_inline(self, text: str, has_images: bool = Fals except Exception: return False + def _should_handle_steer_command_inline(self, text: str, has_images: bool = False) -> bool: + """Return True when /steer should be dispatched immediately while the agent is running. + + /steer MUST bypass the normal _pending_input → process_loop path when + the agent is active, because process_loop is blocked inside + self.chat() for the duration of the run. By the time the queued + command is pulled from _pending_input, _agent_running has already + flipped back to False, and process_command() takes the idle + fallback — delivering the steer as a next-turn message instead of + injecting it mid-run. Dispatching inline on the UI thread calls + agent.steer() directly, which is thread-safe (uses _pending_steer_lock). + """ + if not text or has_images or not _looks_like_slash_command(text): + return False + if not getattr(self, "_agent_running", False): + return False + try: + from hermes_cli.commands import resolve_command + base = text.split(None, 1)[0].lower().lstrip('/') + cmd = resolve_command(base) + return bool(cmd and cmd.name == "steer") + except Exception: + return False + def _show_model_and_providers(self): """Show current model + provider and list all authenticated providers. @@ -4933,8 +5451,15 @@ def _show_model_and_providers(self): print(" To change model or provider, use: hermes model") + def _output_console(self): + """Use prompt_toolkit-safe Rich rendering once the TUI is live.""" + if getattr(self, "_app", None): + return ChatConsole() + return self.console - + def _console_print(self, *args, **kwargs): + """Print through the active command-safe console.""" + self._output_console().print(*args, **kwargs) @staticmethod def _resolve_personality_prompt(value) -> str: @@ -4948,6 +5473,52 @@ def _resolve_personality_prompt(value) -> str: return "\n".join(p for p in parts if p) return str(value) + def _handle_gquota_command(self, cmd_original: str) -> None: + """Show Google Gemini Code Assist quota usage for the current OAuth account.""" + try: + from agent.google_oauth import get_valid_access_token, GoogleOAuthError, load_credentials + from agent.google_code_assist import retrieve_user_quota, CodeAssistError + except ImportError as exc: + self._console_print(f" [red]Gemini modules unavailable: {exc}[/]") + return + + try: + access_token = get_valid_access_token() + except GoogleOAuthError as exc: + self._console_print(f" [yellow]{exc}[/]") + self._console_print(" Run [bold]/model[/] and pick 'Google Gemini (OAuth)' to sign in.") + return + + creds = load_credentials() + project_id = (creds.project_id if creds else "") or "" + + try: + buckets = retrieve_user_quota(access_token, project_id=project_id) + except CodeAssistError as exc: + self._console_print(f" [red]Quota lookup failed:[/] {exc}") + return + + if not buckets: + self._console_print(" [dim]No quota buckets reported (account may be on legacy/unmetered tier).[/]") + return + + # Sort for stable display, group by model + buckets.sort(key=lambda b: (b.model_id, b.token_type)) + self._console_print() + self._console_print(f" [bold]Gemini Code Assist quota[/] (project: {project_id or '(auto / free-tier)'})") + self._console_print() + for b in buckets: + pct = max(0.0, min(1.0, b.remaining_fraction)) + width = 20 + filled = int(round(pct * width)) + bar = "▓" * filled + "░" * (width - filled) + pct_str = f"{int(pct * 100):3d}%" + header = b.model_id + if b.token_type: + header += f" [{b.token_type}]" + self._console_print(f" {header:40s} {bar} {pct_str}") + self._console_print() + def _handle_personality_command(self, cmd: str): """Handle the /personality command to set predefined personalities.""" parts = cmd.split(maxsplit=1) @@ -5077,7 +5648,7 @@ def _parse_flags(tokens): print(" /cron list") print(' /cron add "every 2h" "Check server status" [--skill blogwatcher]') print(' /cron edit --schedule "every 4h" --prompt "New task"') - print(" /cron edit --skill blogwatcher --skill find-nearby") + print(" /cron edit --skill blogwatcher --skill maps") print(" /cron edit --remove-skill blogwatcher") print(" /cron edit --clear-skills") print(" /cron pause ") @@ -5394,7 +5965,7 @@ def process_command(self, command: str) -> bool: _tip_color = get_active_skin().get_color("banner_dim", "#B8860B") except Exception: _tip_color = "#B8860B" - self.console.print(f"[dim {_tip_color}]✦ Tip: {_tip}[/]") + self._console_print(f"[dim {_tip_color}]✦ Tip: {_tip}[/]") except Exception: pass elif canonical == "history": @@ -5457,6 +6028,8 @@ def process_command(self, command: str) -> bool: self._handle_model_switch(cmd_original) elif canonical == "provider": self._show_model_and_providers() + elif canonical == "gquota": + self._handle_gquota_command(cmd_original) elif canonical == "personality": # Use original case (handler lowercases the personality name itself) @@ -5486,7 +6059,7 @@ def process_command(self, command: str) -> bool: elif canonical == "statusbar": self._status_bar_visible = not self._status_bar_visible state = "visible" if self._status_bar_visible else "hidden" - self.console.print(f" Status bar {state}") + self._console_print(f" Status bar {state}") elif canonical == "verbose": self._toggle_verbose() elif canonical == "yolo": @@ -5501,6 +6074,8 @@ def process_command(self, command: str) -> bool: self._show_usage() elif canonical == "insights": self._show_insights(cmd_original) + elif canonical == "copy": + self._handle_copy_command(cmd_original) elif canonical == "debug": self._handle_debug_command() elif canonical == "paste": @@ -5531,7 +6106,8 @@ def process_command(self, command: str) -> bool: version = f" v{p['version']}" if p["version"] else "" tools = f"{p['tools']} tools" if p["tools"] else "" hooks = f"{p['hooks']} hooks" if p["hooks"] else "" - parts = [x for x in [tools, hooks] if x] + commands = f"{p['commands']} commands" if p.get("commands") else "" + parts = [x for x in [tools, hooks, commands] if x] detail = f" ({', '.join(parts)})" if parts else "" error = f" — {p['error']}" if p["error"] else "" print(f" {status} {p['name']}{version}{detail}{error}") @@ -5543,6 +6119,8 @@ def process_command(self, command: str) -> bool: self._handle_snapshot_command(cmd_original) elif canonical == "stop": self._handle_stop_command() + elif canonical == "agents": + self._handle_agents_command() elif canonical == "background": self._handle_background_command(cmd_original) elif canonical == "btw": @@ -5559,6 +6137,30 @@ def process_command(self, command: str) -> bool: _cprint(f" Queued for the next turn: {payload[:80]}{'...' if len(payload) > 80 else ''}") else: _cprint(f" Queued: {payload[:80]}{'...' if len(payload) > 80 else ''}") + elif canonical == "steer": + # Inject a message after the next tool call without interrupting. + # If the agent is actively running, push the text into the agent's + # pending_steer slot — the drain hook in _execute_tool_calls_* + # will append it to the next tool result's content. If no agent + # is running, fall back to queue semantics (same as /queue). + parts = cmd_original.split(None, 1) + payload = parts[1].strip() if len(parts) > 1 else "" + if not payload: + _cprint(" Usage: /steer ") + elif self._agent_running and self.agent is not None and hasattr(self.agent, "steer"): + try: + accepted = self.agent.steer(payload) + except Exception as exc: + _cprint(f" Steer failed: {exc}") + else: + if accepted: + _cprint(f" ⏩ Steer queued — arrives after the next tool call: {payload[:80]}{'...' if len(payload) > 80 else ''}") + else: + _cprint(" Steer rejected (empty payload).") + else: + # No active run — treat as a normal next-turn message. + self._pending_input.put(payload) + _cprint(f" No agent running; queued as next turn: {payload[:80]}{'...' if len(payload) > 80 else ''}") elif canonical == "skin": self._handle_skin_command(cmd_original) elif canonical == "voice": @@ -5580,15 +6182,15 @@ def process_command(self, command: str) -> bool: ) output = result.stdout.strip() or result.stderr.strip() if output: - self.console.print(_rich_text_from_ansi(output)) + self._console_print(_rich_text_from_ansi(output)) else: - self.console.print("[dim]Command returned no output[/]") + self._console_print("[dim]Command returned no output[/]") except subprocess.TimeoutExpired: - self.console.print("[bold red]Quick command timed out (30s)[/]") + self._console_print("[bold red]Quick command timed out (30s)[/]") except Exception as e: - self.console.print(f"[bold red]Quick command error: {e}[/]") + self._console_print(f"[bold red]Quick command error: {e}[/]") else: - self.console.print(f"[bold red]Quick command '{base_cmd}' has no command defined[/]") + self._console_print(f"[bold red]Quick command '{base_cmd}' has no command defined[/]") elif qcmd.get("type") == "alias": target = qcmd.get("target", "").strip() if target: @@ -5597,9 +6199,9 @@ def process_command(self, command: str) -> bool: aliased_command = f"{target} {user_args}".strip() return self.process_command(aliased_command) else: - self.console.print(f"[bold red]Quick command '{base_cmd}' has no target defined[/]") + self._console_print(f"[bold red]Quick command '{base_cmd}' has no target defined[/]") else: - self.console.print(f"[bold red]Quick command '{base_cmd}' has unsupported type (supported: 'exec', 'alias')[/]") + self._console_print(f"[bold red]Quick command '{base_cmd}' has unsupported type (supported: 'exec', 'alias')[/]") # Check for plugin-registered slash commands elif base_cmd.lstrip("/") in _get_plugin_cmd_handler_names(): from hermes_cli.plugins import get_plugin_command_handler @@ -5778,8 +6380,7 @@ def _bg_thinking(text: str) -> None: # with the output (fixes #2718). if self._app: self._app.invalidate() - import time as _tmod - _tmod.sleep(0.05) # brief pause for refresh + time.sleep(0.05) # brief pause for refresh print() ChatConsole().print(f"[{_accent_hex()}]{'─' * 40}[/]") _cprint(f" ✅ Background task #{task_num} complete") @@ -5799,13 +6400,13 @@ def _bg_thinking(text: str) -> None: _chat_console = ChatConsole() _chat_console.print(Panel( - _rich_text_from_ansi(response), + _render_final_assistant_content(response, mode=self.final_response_markdown), title=f"[{_resp_color} bold]{label} (background #{task_num})[/]", title_align="left", border_style=_resp_color, style=_resp_text, box=rich_box.HORIZONTALS, - padding=(1, 2), + padding=(1, 4), )) else: _cprint(" (No response generated)") @@ -5819,8 +6420,7 @@ def _bg_thinking(text: str) -> None: # Same TUI refresh pattern as success path (#2718) if self._app: self._app.invalidate() - import time as _tmod - _tmod.sleep(0.05) + time.sleep(0.05) print() _cprint(f" ❌ Background task #{task_num} failed: {e}") finally: @@ -5924,12 +6524,12 @@ def run_btw(): _resp_color = "#4F6D4A" ChatConsole().print(Panel( - _rich_text_from_ansi(response), + _render_final_assistant_content(response, mode=self.final_response_markdown), title=f"[{_resp_color} bold]⚕ /btw[/]", title_align="left", border_style=_resp_color, box=rich_box.HORIZONTALS, - padding=(1, 2), + padding=(1, 4), )) else: _cprint(" 💬 /btw: (no response)") @@ -5996,7 +6596,7 @@ def _handle_browser_command(self, cmd: str): parts = cmd.strip().split(None, 1) sub = parts[1].lower().strip() if len(parts) > 1 else "status" - _DEFAULT_CDP = "http://localhost:9222" + _DEFAULT_CDP = "http://127.0.0.1:9222" current = os.environ.get("BROWSER_CDP_URL", "").strip() if sub.startswith("connect"): @@ -6040,7 +6640,6 @@ def _handle_browser_command(self, cmd: str): _launched = self._try_launch_chrome_debug(_port, _plat.system()) if _launched: # Wait for the port to come up - import time as _time for _wait in range(10): try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) @@ -6050,7 +6649,7 @@ def _handle_browser_command(self, cmd: str): _already_open = True break except (OSError, socket.timeout): - _time.sleep(0.5) + time.sleep(0.5) if _already_open: print(f" ✓ Chrome launched and listening on port {_port}") else: @@ -6203,6 +6802,7 @@ def _handle_skin_command(self, cmd: str): set_active_skin(new_skin) _ACCENT.reset() # Re-resolve ANSI color for the new skin + _DIM.reset() # Re-resolve dim/secondary ANSI color for the new skin if save_config_value("display.skin", new_skin): print(f" Skin set to: {new_skin} (saved)") else: @@ -6242,13 +6842,21 @@ def _toggle_verbose(self): def _toggle_yolo(self): """Toggle YOLO mode — skip all dangerous command approval prompts.""" import os + from hermes_cli.colors import Colors as _Colors + current = bool(os.environ.get("HERMES_YOLO_MODE")) if current: os.environ.pop("HERMES_YOLO_MODE", None) - self.console.print(" ⚠ YOLO mode [bold red]OFF[/] — dangerous commands will require approval.") + _cprint( + f" ⚠ YOLO mode {_Colors.BOLD}{_Colors.RED}OFF{_Colors.RESET}" + " — dangerous commands will require approval." + ) else: os.environ["HERMES_YOLO_MODE"] = "1" - self.console.print(" ⚡ YOLO mode [bold green]ON[/] — all commands auto-approved. Use with caution.") + _cprint( + f" ⚡ YOLO mode {_Colors.BOLD}{_Colors.GREEN}ON{_Colors.RESET}" + " — all commands auto-approved. Use with caution." + ) def _handle_reasoning_command(self, cmd: str): """Handle /reasoning — manage effort level and display toggle. @@ -6407,6 +7015,18 @@ def _manual_compress(self, cmd_original: str = ""): focus_topic=focus_topic or None, ) self.conversation_history = compressed + # _compress_context ends the old session and creates a new child + # session on the agent (run_agent.py::_compress_context). Sync the + # CLI's session_id so /status, /resume, exit summary, and title + # generation all point at the live continuation session, not the + # ended parent. Without this, subsequent end_session() calls target + # the already-closed parent and the child is orphaned. + if ( + getattr(self.agent, "session_id", None) + and self.agent.session_id != self.session_id + ): + self.session_id = self.agent.session_id + self._pending_title = None new_tokens = estimate_messages_tokens_rough(self.conversation_history) summary = summarize_manual_compression( original_history, @@ -6509,6 +7129,27 @@ def _show_usage(self): if cost_result.status == "unknown": print(f" Note: Pricing unknown for {agent.model}") + # Account limits -- fetched off-thread with a hard timeout so slow + # provider APIs don't hang the prompt. + provider = getattr(agent, "provider", None) or getattr(self, "provider", None) + base_url = getattr(agent, "base_url", None) or getattr(self, "base_url", None) + api_key = getattr(agent, "api_key", None) or getattr(self, "api_key", None) + account_snapshot = None + if provider: + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as _pool: + try: + account_snapshot = _pool.submit( + fetch_account_usage, provider, + base_url=base_url, api_key=api_key, + ).result(timeout=10.0) + except (concurrent.futures.TimeoutError, Exception): + account_snapshot = None + account_lines = [f" {line}" for line in render_account_usage_lines(account_snapshot)] + if account_lines: + print() + for line in account_lines: + print(line) + if self.verbose: logging.getLogger().setLevel(logging.DEBUG) for noisy in ('openai', 'openai._base_client', 'httpx', 'httpcore', 'asyncio', 'hpack', 'grpc', 'modal'): @@ -6559,7 +7200,6 @@ def _check_config_mcp_changes(self) -> None: known state. When a change is detected, triggers _reload_mcp() and informs the user so they know the tool list has been refreshed. """ - import time import yaml as _yaml CONFIG_WATCH_INTERVAL = 5.0 # seconds between config.yaml stat() calls @@ -6651,7 +7291,6 @@ def _reload_mcp(self): # Refresh the agent's tool list so the model can call new tools if self.agent is not None: - from model_tools import get_tool_definitions self.agent.tools = get_tool_definitions( enabled_toolsets=self.agent.enabled_toolsets if hasattr(self.agent, "enabled_toolsets") else None, @@ -6734,7 +7373,6 @@ def _on_tool_progress(self, event_type: str, function_name: str = None, preview: full history of tool calls (not just the current one in the spinner). """ if event_type == "tool.completed": - import time as _time self._tool_start_time = 0.0 # Print stacked scrollback line for "all" / "new" modes if function_name and self.tool_progress_mode in ("all", "new"): @@ -6763,7 +7401,6 @@ def _on_tool_progress(self, event_type: str, function_name: str = None, preview: if event_type != "tool.started": return if function_name and not function_name.startswith("_"): - import time as _time from agent.display import get_tool_emoji emoji = get_tool_emoji(function_name) label = preview or function_name @@ -6772,7 +7409,7 @@ def _on_tool_progress(self, event_type: str, function_name: str = None, preview: if _pl > 0 and len(label) > _pl: label = label[:_pl - 3] + "..." self._spinner_text = f"{emoji} {label}" - self._tool_start_time = _time.monotonic() + self._tool_start_time = time.monotonic() # Store args for stacked scrollback line on completion self._pending_tool_info.setdefault(function_name, []).append( function_args if function_args is not None else {} @@ -6847,8 +7484,7 @@ def _voice_start_recording(self): ) raise RuntimeError( "Voice mode requires sounddevice and numpy.\n" - "Install with: pip install sounddevice numpy\n" - "Or: pip install hermes-agent[voice]" + f"Install with: {sys.executable} -m pip install sounddevice numpy" ) if not reqs.get("stt_available", reqs.get("stt_key_set")): raise RuntimeError( @@ -6890,11 +7526,12 @@ def _on_silence(): self._voice_stop_and_transcribe() # Audio cue: single beep BEFORE starting stream (avoid CoreAudio conflict) - try: - from tools.voice_mode import play_beep - play_beep(frequency=880, count=1) - except Exception: - pass + if self._voice_beeps_enabled(): + try: + from tools.voice_mode import play_beep + play_beep(frequency=880, count=1) + except Exception: + pass try: self._voice_recorder.start(on_silence_stop=_on_silence) @@ -6942,11 +7579,12 @@ def _voice_stop_and_transcribe(self): wav_path = self._voice_recorder.stop() # Audio cue: double beep after stream stopped (no CoreAudio conflict) - try: - from tools.voice_mode import play_beep - play_beep(frequency=660, count=2) - except Exception: - pass + if self._voice_beeps_enabled(): + try: + from tools.voice_mode import play_beep + play_beep(frequency=660, count=2) + except Exception: + pass if wav_path is None: _cprint(f"{_DIM}No speech detected.{_RST}") @@ -7029,7 +7667,6 @@ def _voice_speak_response(self, text: str): try: from tools.tts_tool import text_to_speech_tool from tools.voice_mode import play_audio_file - import re # Strip markdown and non-speech content for cleaner TTS tts_text = text[:4000] if len(text) > 4000 else text @@ -7097,6 +7734,17 @@ def _handle_voice_command(self, command: str): _cprint(f"Unknown voice subcommand: {subcommand}") _cprint("Usage: /voice [on|off|tts|status]") + def _voice_beeps_enabled(self) -> bool: + """Return whether CLI voice mode should play record start/stop beeps.""" + try: + from hermes_cli.config import load_config + voice_cfg = load_config().get("voice", {}) + if isinstance(voice_cfg, dict): + return bool(voice_cfg.get("beep_enabled", True)) + except Exception: + pass + return True + def _enable_voice_mode(self): """Enable voice mode after checking requirements.""" if self._voice_mode: @@ -7124,8 +7772,7 @@ def _enable_voice_mode(self): _cprint(f" {_DIM}Then install/update the Termux:API Android app for microphone capture{_RST}") _cprint(f" {_BOLD}Option 2: pkg install python-numpy portaudio && python -m pip install sounddevice{_RST}") else: - _cprint(f"\n {_BOLD}Install: pip install {' '.join(reqs['missing_packages'])}{_RST}") - _cprint(f" {_DIM}Or: pip install hermes-agent[voice]{_RST}") + _cprint(f"\n {_BOLD}Install: {sys.executable} -m pip install {' '.join(reqs['missing_packages'])}{_RST}") return with self._voice_lock: @@ -7407,7 +8054,9 @@ def _handle_approval_selection(self) -> None: return selected = state.get("selected", 0) - choices = state.get("choices") or [] + choices = state.get("choices") + if not isinstance(choices, list): + choices = [] if not (0 <= selected < len(choices)): return @@ -7425,7 +8074,15 @@ def _handle_approval_selection(self) -> None: self._invalidate() def _get_approval_display_fragments(self): - """Render the dangerous-command approval panel for the prompt_toolkit UI.""" + """Render the dangerous-command approval panel for the prompt_toolkit UI. + + Layout priority: title + command + choices must always render, even if + the terminal is short or the description is long. Description is placed + at the bottom of the panel and gets truncated to fit the remaining row + budget. This prevents HSplit from clipping approve/deny off-screen when + tirith findings produce multi-paragraph descriptions or when the user + runs in a compact terminal pane. + """ state = self._approval_state if not state: return [] @@ -7484,22 +8141,99 @@ def _append_blank_panel_line(lines, border_style: str, box_width: int) -> None: box_width = _panel_box_width(title, preview_lines) inner_text_width = max(8, box_width - 2) + # Pre-wrap the mandatory content — command + choices must always render. + cmd_wrapped = _wrap_panel_text(cmd_display, inner_text_width) + + # (choice_index, wrapped_line) so we can re-apply selected styling below + choice_wrapped: list[tuple[int, str]] = [] + for i, choice in enumerate(choices): + label = choice_labels.get(choice, choice) + # Show number prefix for quick selection (1-9 for items 1-9, 0 for 10th item) + if i < 9: + num_prefix = str(i + 1) + elif i == 9: + num_prefix = '0' + else: + num_prefix = ' ' # No number for items beyond 10th + if i == selected: + prefix = f'❯ {num_prefix}. ' + else: + prefix = f' {num_prefix}. ' + for wrapped in _wrap_panel_text(f"{prefix}{label}", inner_text_width, subsequent_indent=" "): + choice_wrapped.append((i, wrapped)) + + # Budget vertical space so HSplit never clips the command or choices. + # Panel chrome (full layout with separators): + # top border + title + blank_after_title + # + blank_between_cmd_choices + bottom border = 5 rows. + # In tight terminals we collapse to: + # top border + title + bottom border = 3 rows (no blanks). + # + # reserved_below: rows consumed below the approval panel by the + # spinner/tool-progress line, status bar, input area, separators, and + # prompt symbol. Measured at ~6 rows during live PTY approval prompts; + # budget 6 so we don't overestimate the panel's room. + term_rows = shutil.get_terminal_size((100, 24)).lines + chrome_full = 5 + chrome_tight = 3 + reserved_below = 6 + + available = max(0, term_rows - reserved_below) + mandatory_full = chrome_full + len(cmd_wrapped) + len(choice_wrapped) + + # If the full-chrome panel doesn't fit, drop the separator blanks. + # This keeps the command and every choice on-screen in compact terminals. + use_compact_chrome = mandatory_full > available + chrome_rows = chrome_tight if use_compact_chrome else chrome_full + + # If the command itself is too long to leave room for choices (e.g. user + # hit "view" on a multi-hundred-character command), truncate it so the + # approve/deny buttons still render. Keep at least 1 row of command. + max_cmd_rows = max(1, available - chrome_rows - len(choice_wrapped)) + if len(cmd_wrapped) > max_cmd_rows: + keep = max(1, max_cmd_rows - 1) if max_cmd_rows > 1 else 1 + cmd_wrapped = cmd_wrapped[:keep] + ["… (command truncated — use /logs or /debug for full text)"] + + # Allocate any remaining rows to description. The extra -1 in full mode + # accounts for the blank separator between choices and description. + mandatory_no_desc = chrome_rows + len(cmd_wrapped) + len(choice_wrapped) + desc_sep_cost = 0 if use_compact_chrome else 1 + available_for_desc = available - mandatory_no_desc - desc_sep_cost + # Even on huge terminals, cap description height so the panel stays compact. + available_for_desc = max(0, min(available_for_desc, 10)) + + desc_wrapped = _wrap_panel_text(description, inner_text_width) if description else [] + if available_for_desc < 1 or not desc_wrapped: + desc_wrapped = [] + elif len(desc_wrapped) > available_for_desc: + keep = max(1, available_for_desc - 1) + desc_wrapped = desc_wrapped[:keep] + ["… (description truncated)"] + + # Render: title → command → choices → description (description last so + # any remaining overflow clips from the bottom of the least-critical + # content, never from the command or choices). Use compact chrome (no + # blank separators) when the terminal is tight. lines = [] lines.append(('class:approval-border', '╭' + ('─' * box_width) + '╮\n')) _append_panel_line(lines, 'class:approval-border', 'class:approval-title', title, box_width) - _append_blank_panel_line(lines, 'class:approval-border', box_width) - for wrapped in _wrap_panel_text(description, inner_text_width): - _append_panel_line(lines, 'class:approval-border', 'class:approval-desc', wrapped, box_width) - for wrapped in _wrap_panel_text(cmd_display, inner_text_width): + if not use_compact_chrome: + _append_blank_panel_line(lines, 'class:approval-border', box_width) + + for wrapped in cmd_wrapped: _append_panel_line(lines, 'class:approval-border', 'class:approval-cmd', wrapped, box_width) - _append_blank_panel_line(lines, 'class:approval-border', box_width) - for i, choice in enumerate(choices): - label = choice_labels.get(choice, choice) + if not use_compact_chrome: + _append_blank_panel_line(lines, 'class:approval-border', box_width) + + for i, wrapped in choice_wrapped: style = 'class:approval-selected' if i == selected else 'class:approval-choice' - prefix = '❯ ' if i == selected else ' ' - for wrapped in _wrap_panel_text(f"{prefix}{label}", inner_text_width, subsequent_indent=" "): - _append_panel_line(lines, 'class:approval-border', style, wrapped, box_width) - _append_blank_panel_line(lines, 'class:approval-border', box_width) + _append_panel_line(lines, 'class:approval-border', style, wrapped, box_width) + + if desc_wrapped: + if not use_compact_chrome: + _append_blank_panel_line(lines, 'class:approval-border', box_width) + for wrapped in desc_wrapped: + _append_panel_line(lines, 'class:approval-border', 'class:approval-desc', wrapped, box_width) + lines.append(('class:approval-border', '╰' + ('─' * box_width) + '╯\n')) return lines @@ -7588,7 +8322,6 @@ def chat(self, message, images: list = None) -> Optional[str]: if not self._init_agent( model_override=turn_route["model"], runtime_override=turn_route["runtime"], - route_label=turn_route["label"], request_overrides=turn_route.get("request_overrides"), ): return None @@ -7691,7 +8424,7 @@ def display_callback(sentence: str): label = " ⚕ Hermes " fill = w - 2 - len(label) _cprint(f"\n{_ACCENT}╭─{label}{'─' * max(fill - 1, 0)}╮{_RST}") - _cprint(sentence.rstrip()) + _cprint(f"{_STREAM_PAD}{sentence.rstrip()}") tts_thread = threading.Thread( target=stream_tts_to_speaker, @@ -7717,6 +8450,17 @@ def stream_callback(delta: str): def run_agent(): nonlocal result + # Set callbacks inside the agent thread so thread-local storage + # in terminal_tool is populated for this thread. The main thread + # registration (run() line ~9046) is invisible here because + # _callback_tls is threading.local(). Matches the pattern used + # by acp_adapter/server.py for ACP sessions. + set_sudo_password_callback(self._sudo_password_callback) + set_approval_callback(self._approval_callback) + try: + set_secret_capture_callback(self._secret_capture_callback) + except Exception: + pass agent_message = _voice_prefix + message if _voice_prefix else message # Prepend pending model switch note so the model knows about the switch _msn = getattr(self, '_pending_model_switch_note', None) @@ -7742,10 +8486,23 @@ def run_agent(): "failed": True, "error": _summary, } + finally: + # Clear thread-local callbacks so a reused thread doesn't + # hold stale references to a disposed CLI instance. + try: + set_sudo_password_callback(None) + set_approval_callback(None) + set_secret_capture_callback(None) + except Exception: + pass # Start agent in background thread (daemon so it cannot keep the # process alive when the user closes the terminal tab — SIGHUP # exits the main thread and daemon threads are reaped automatically). + # Start per-prompt elapsed timer — frozen after the agent thread + # finishes; reset on the next turn. + self._prompt_start_time = time.time() + self._prompt_duration = 0.0 agent_thread = threading.Thread(target=run_agent, daemon=True) agent_thread.start() @@ -7775,8 +8532,7 @@ def run_agent(): try: _dbg = _hermes_home / "interrupt_debug.log" with open(_dbg, "a") as _f: - import time as _t - _f.write(f"{_t.strftime('%H:%M:%S')} interrupt fired: msg={str(interrupt_msg)[:60]!r}, " + _f.write(f"{time.strftime('%H:%M:%S')} interrupt fired: msg={str(interrupt_msg)[:60]!r}, " f"children={len(self.agent._active_children)}, " f"parent._interrupt={self.agent._interrupt_requested}\n") for _ci, _ch in enumerate(self.agent._active_children): @@ -7795,7 +8551,39 @@ def run_agent(): # Fallback for non-interactive mode (e.g., single-query) agent_thread.join(0.1) - agent_thread.join() # Ensure agent thread completes + # Wait for the agent thread to finish. After an interrupt the + # agent may take a few seconds to clean up (kill subprocess, persist + # session). Poll instead of a blocking join so the process_loop + # stays responsive — if the user sent another interrupt or the + # agent gets stuck, we can break out instead of freezing forever. + if interrupt_msg is not None: + # Interrupt path: poll briefly, then move on. The agent + # thread is daemon — it dies on process exit regardless. + for _wait_tick in range(50): # 50 * 0.2s = 10s max + agent_thread.join(timeout=0.2) + if not agent_thread.is_alive(): + break + # Check if user fired ANOTHER interrupt (Ctrl+C sets + # _should_exit which process_loop checks on next pass). + if getattr(self, '_should_exit', False): + break + if agent_thread.is_alive(): + logger.warning( + "Agent thread still alive after interrupt " + "(thread %s). Daemon thread will be cleaned up " + "on exit.", + agent_thread.ident, + ) + else: + # Normal completion: agent thread should be done already, + # but guard against edge cases. + agent_thread.join(timeout=30) + + # Freeze per-prompt elapsed timer once the agent thread has + # exited (or been abandoned as a daemon after interrupt). + if self._prompt_start_time is not None: + self._prompt_duration = max(0.0, time.time() - self._prompt_start_time) + self._prompt_start_time = None # Proactively clean up async clients whose event loop is dead. # The agent thread may have created AsyncOpenAI clients bound @@ -7820,13 +8608,26 @@ def run_agent(): # buffer so tool/status lines render ABOVE our response box. # The flush pushes data into the renderer queue; the short # sleep lets the renderer actually paint it before we draw. - import time as _time sys.stdout.flush() - _time.sleep(0.15) + time.sleep(0.15) # Update history with full conversation self.conversation_history = result.get("messages", self.conversation_history) if result else self.conversation_history + # If auto-compression fired mid-turn, the agent created a new + # continuation session and mutated self.agent.session_id. Sync + # the CLI's session_id so /status, /resume, title generation, + # and the exit summary all target the live child session rather + # than the ended parent. Mirrors the gateway's post-run sync + # (gateway/run.py around line 9983). + if ( + self.agent + and getattr(self.agent, "session_id", None) + and self.agent.session_id != self.session_id + ): + self.session_id = self.agent.session_id + self._pending_title = None + # Get the final response response = result.get("final_response", "") if result else "" @@ -7916,13 +8717,13 @@ def run_agent(): else: _chat_console = ChatConsole() _chat_console.print(Panel( - _rich_text_from_ansi(response), + _render_final_assistant_content(response, mode=self.final_response_markdown), title=f"[{_resp_color} bold]{label}[/]", title_align="left", border_style=_resp_color, style=_resp_text, box=rich_box.HORIZONTALS, - padding=(1, 2), + padding=(1, 4), )) @@ -7975,7 +8776,15 @@ def run_agent(): else: print(f"\n⚡ Sending after interrupt: '{preview}'") self._pending_input.put(combined) - + + # If a /steer was left over (agent finished before another tool + # batch could absorb it), deliver it as the next user turn. + _leftover_steer = result.get("pending_steer") if result else None + if _leftover_steer and hasattr(self, '_pending_input'): + preview = _leftover_steer[:60] + ("..." if len(_leftover_steer) > 60 else "") + print(f"\n⏩ Delivering leftover /steer as next turn: '{preview}'") + self._pending_input.put(_leftover_steer) + return response except Exception as e: @@ -8253,7 +9062,7 @@ def run(self): except Exception: _welcome_text = "Welcome to Hermes Agent! Type your message or /help for commands." _welcome_color = "#FFF8DC" - self.console.print(f"[{_welcome_color}]{_welcome_text}[/]") + self._console_print(f"[{_welcome_color}]{_welcome_text}[/]") # Show a random tip to help users discover features try: from hermes_cli.tips import get_random_tip @@ -8262,16 +9071,16 @@ def run(self): _tip_color = _welcome_skin.get_color("banner_dim", "#B8860B") except Exception: _tip_color = "#B8860B" - self.console.print(f"[dim {_tip_color}]✦ Tip: {_tip}[/]") + self._console_print(f"[dim {_tip_color}]✦ Tip: {_tip}[/]") except Exception: pass # Tips are non-critical — never break startup if self.preloaded_skills and not self._startup_skills_line_shown: skills_label = ", ".join(self.preloaded_skills) - self.console.print( + self._console_print( f"[bold {_accent_hex()}]Activated skills:[/] {skills_label}" ) self._startup_skills_line_shown = True - self.console.print() + self._console_print() # State for async operation self._agent_running = False @@ -8393,6 +9202,7 @@ def handle_enter(event): # --- /model picker modal --- if self._model_picker_state: self._handle_model_picker_selection() + event.app.current_buffer.reset() event.app.invalidate() return @@ -8436,6 +9246,17 @@ def handle_enter(event): event.app.current_buffer.reset(append_to_history=True) return + # Handle /steer while the agent is running immediately on the + # UI thread. Queuing through _pending_input would deadlock the + # steer until after the agent loop finishes (process_loop is + # blocked inside self.chat()), which turns /steer into a + # post-run next-turn message — defeating mid-run injection. + # agent.steer() is thread-safe (holds _pending_steer_lock). + if self._should_handle_steer_command_inline(text, has_images=has_images): + self.process_command(text) + event.app.current_buffer.reset(append_to_history=True) + return + # Snapshot and clear attached images images = list(self._attached_images) self._attached_images.clear() @@ -8454,8 +9275,7 @@ def handle_enter(event): try: _dbg = _hermes_home / "interrupt_debug.log" with open(_dbg, "a") as _f: - import time as _t - _f.write(f"{_t.strftime('%H:%M:%S')} ENTER: queued interrupt msg={str(payload)[:60]!r}, " + _f.write(f"{time.strftime('%H:%M:%S')} ENTER: queued interrupt msg={str(payload)[:60]!r}, " f"agent_running={self._agent_running}\n") except Exception: pass @@ -8473,6 +9293,16 @@ def handle_ctrl_enter(event): """Ctrl+Enter (c-j) inserts a newline. Most terminals send c-j for Ctrl+Enter.""" event.current_buffer.insert_text('\n') + @kb.add( + 'c-g', + filter=Condition( + lambda: not self._clarify_state and not self._approval_state and not self._sudo_state and not self._secret_state + ), + ) + def handle_open_in_editor(event): + """Ctrl+G opens the current draft in an external editor.""" + cli_ref._open_external_editor(event.current_buffer) + @kb.add('tab', eager=True) def handle_tab(event): """Tab: accept completion, auto-suggestion, or start completions. @@ -8524,6 +9354,29 @@ def clarify_down(event): self._clarify_state["selected"] = min(max_idx, self._clarify_state["selected"] + 1) event.app.invalidate() + # Number keys for quick clarify selection (1-9, 0 for 10th item) + def _make_clarify_number_handler(idx): + def handler(event): + if self._clarify_state and not self._clarify_freetext: + choices = self._clarify_state.get("choices") or [] + # Map index to choice (treating "Other" as the last option) + if idx < len(choices): + # Select a numbered choice + self._clarify_state["response_queue"].put(choices[idx]) + self._clarify_state = None + self._clarify_freetext = False + event.app.invalidate() + elif idx == len(choices): + # Select "Other" option + self._clarify_freetext = True + event.app.invalidate() + return handler + + for _num in range(10): + # 1-9 select items 0-8, 0 selects item 9 (10thitem) + _idx = 9 if _num == 0 else _num - 1 + kb.add(str(_num), filter=Condition(lambda: bool(self._clarify_state) and not self._clarify_freetext))(_make_clarify_number_handler(_idx)) + # --- Dangerous command approval: arrow-key navigation --- @kb.add('up', filter=Condition(lambda: bool(self._approval_state))) @@ -8558,6 +9411,27 @@ def model_picker_down(event): state["selected"] = min(max_idx, state.get("selected", 0) + 1) event.app.invalidate() + @kb.add('escape', filter=Condition(lambda: bool(self._model_picker_state)), eager=True) + def model_picker_escape(event): + """ESC closes the /model picker.""" + self._close_model_picker() + event.app.current_buffer.reset() + event.app.invalidate() + + # Number keys for quick approval selection (1-9, 0 for 10th item) + def _make_approval_number_handler(idx): + def handler(event): + if self._approval_state and idx < len(self._approval_state["choices"]): + self._approval_state["selected"] = idx + self._handle_approval_selection() + event.app.invalidate() + return handler + + for _num in range(10): + # 1-9 select items 0-8, 0 selects item 9 (10th item) + _idx = 9 if _num == 0 else _num - 1 + kb.add(str(_num), filter=Condition(lambda: bool(self._approval_state)))(_make_approval_number_handler(_idx)) + # --- History navigation: up/down browse history in normal input mode --- # The TextArea is multiline, so by default up/down only move the cursor. # Buffer.auto_up/auto_down handle both: cursor movement when multi-line, @@ -8586,8 +9460,7 @@ def handle_ctrl_c(event): 2. Interrupt the running agent (first press) 3. Force exit (second press within 2s, or when idle) """ - import time as _time - now = _time.time() + now = time.time() # Cancel active voice recording. # Run cancel() in a background thread to prevent blocking the @@ -8674,15 +9547,32 @@ def handle_ctrl_d(event): self._should_exit = True event.app.exit() + _modal_prompt_active = Condition( + lambda: bool(self._secret_state or self._sudo_state) + ) + + @kb.add('escape', filter=_modal_prompt_active, eager=True) + def handle_escape_modal(event): + """ESC cancels active secret/sudo prompts.""" + if self._secret_state: + self._cancel_secret_capture() + event.app.current_buffer.reset() + event.app.invalidate() + return + if self._sudo_state: + self._sudo_state["response_queue"].put("") + self._sudo_state = None + event.app.invalidate() + return + @kb.add('c-z') def handle_ctrl_z(event): """Handle Ctrl+Z - suspend process to background (Unix only).""" - import sys if sys.platform == 'win32': _cprint(f"\n{_DIM}Suspend (Ctrl+Z) is not supported on Windows.{_RST}") event.app.invalidate() return - import os, signal as _sig + import signal as _sig from prompt_toolkit.application import run_in_terminal from hermes_cli.skin_engine import get_active_skin agent_name = get_active_skin().get_branding("agent_name", "Hermes Agent") @@ -8899,6 +9789,7 @@ def _input_height(): _prev_text_len = [0] _prev_newline_count = [0] _paste_just_collapsed = [False] + self._skip_paste_collapse = False def _on_text_changed(buf): """Detect large pastes and collapse them to a file reference. @@ -8918,8 +9809,9 @@ def _on_text_changed(buf): text = buf.text chars_added = len(text) - _prev_text_len[0] _prev_text_len[0] = len(text) - if _paste_just_collapsed[0]: + if _paste_just_collapsed[0] or self._skip_paste_collapse: _paste_just_collapsed[0] = False + self._skip_paste_collapse = False _prev_newline_count[0] = text.count('\n') return line_count = text.count('\n') @@ -8928,12 +9820,10 @@ def _on_text_changed(buf): is_paste = chars_added > 1 or newlines_added >= 4 if line_count >= 5 and is_paste and not text.startswith('/'): _paste_counter[0] += 1 - # Save to temp file paste_dir = _hermes_home / "pastes" paste_dir.mkdir(parents=True, exist_ok=True) paste_file = paste_dir / f"paste_{_paste_counter[0]}_{datetime.now().strftime('%H%M%S')}.txt" paste_file.write_text(text, encoding="utf-8") - # Replace buffer with compact reference _paste_just_collapsed[0] = True buf.text = f"[Pasted text #{_paste_counter[0]}: {line_count + 1} lines \u2192 {paste_file}]" buf.cursor_position = len(buf.text) @@ -8971,9 +9861,9 @@ def _get_placeholder(): if cli_ref._voice_processing: return "transcribing..." if cli_ref._sudo_state: - return "type password (hidden), Enter to skip" + return "type password (hidden), Enter to submit · ESC to skip" if cli_ref._secret_state: - return "type secret (hidden), Enter to skip" + return "type secret (hidden), Enter to submit · ESC to skip" if cli_ref._approval_state: return "" if cli_ref._clarify_freetext: @@ -8996,31 +9886,29 @@ def _get_placeholder(): # extra instructions (sudo countdown, approval navigation, clarify). # The agent-running interrupt hint is now an inline placeholder above. def get_hint_text(): - import time as _time - if cli_ref._sudo_state: - remaining = max(0, int(cli_ref._sudo_deadline - _time.monotonic())) + remaining = max(0, int(cli_ref._sudo_deadline - time.monotonic())) return [ ('class:hint', ' password hidden · Enter to skip'), ('class:clarify-countdown', f' ({remaining}s)'), ] if cli_ref._secret_state: - remaining = max(0, int(cli_ref._secret_deadline - _time.monotonic())) + remaining = max(0, int(cli_ref._secret_deadline - time.monotonic())) return [ ('class:hint', ' secret hidden · Enter to skip'), ('class:clarify-countdown', f' ({remaining}s)'), ] if cli_ref._approval_state: - remaining = max(0, int(cli_ref._approval_deadline - _time.monotonic())) + remaining = max(0, int(cli_ref._approval_deadline - time.monotonic())) return [ ('class:hint', ' ↑/↓ to select, Enter to confirm'), ('class:clarify-countdown', f' ({remaining}s)'), ] if cli_ref._clarify_state: - remaining = max(0, int(cli_ref._clarify_deadline - _time.monotonic())) + remaining = max(0, int(cli_ref._clarify_deadline - time.monotonic())) countdown = f' ({remaining}s)' if cli_ref._clarify_deadline else '' if cli_ref._clarify_freetext: return [ @@ -9048,21 +9936,10 @@ def get_hint_height(): return cli_ref._agent_spacer_height() def get_spinner_text(): - txt = cli_ref._spinner_text - if not txt: + spinner_line = cli_ref._render_spinner_text() + if not spinner_line: return [] - # Append live elapsed timer when a tool is running - t0 = cli_ref._tool_start_time - if t0 > 0: - import time as _time - elapsed = _time.monotonic() - t0 - if elapsed >= 60: - _m, _s = int(elapsed // 60), int(elapsed % 60) - elapsed_str = f"{_m}m {_s}s" - else: - elapsed_str = f"{elapsed:.1f}s" - return [('class:hint', f' {txt} ({elapsed_str})')] - return [('class:hint', f' {txt}')] + return [('class:hint', spinner_line)] def get_spinner_height(): return cli_ref._spinner_widget_height() @@ -9070,6 +9947,7 @@ def get_spinner_height(): spinner_widget = Window( content=FormattedTextControl(get_spinner_text), height=get_spinner_height, + wrap_lines=True, ) spacer = Window( @@ -9106,7 +9984,13 @@ def _append_blank_panel_line(lines, border_style: str, box_width: int) -> None: lines.append((border_style, "│" + (" " * box_width) + "│\n")) def _get_clarify_display(): - """Build styled text for the clarify question/choices panel.""" + """Build styled text for the clarify question/choices panel. + + Layout priority: choices + Other option must always render even if + the question is very long. The question is budgeted to leave enough + rows for the choices and trailing chrome; anything over the budget + is truncated with a marker. + """ state = cli_ref._clarify_state if not state: return [] @@ -9116,59 +10000,152 @@ def _get_clarify_display(): selected = state.get("selected", 0) preview_lines = _wrap_panel_text(question, 60) for i, choice in enumerate(choices): - prefix = "❯ " if i == selected and not cli_ref._clarify_freetext else " " - preview_lines.extend(_wrap_panel_text(f"{prefix}{choice}", 60, subsequent_indent=" ")) + # Show number prefix for quick selection (1-9 for items 1-9, 0 for 10th item) + if i < 9: + num_prefix = str(i + 1) + elif i == 9: + num_prefix = '0' + else: + num_prefix = ' ' + if i == selected and not cli_ref._clarify_freetext: + prefix = f"❯ {num_prefix}. " + else: + prefix = f" {num_prefix}. " + preview_lines.extend(_wrap_panel_text(f"{prefix}{choice}", 60, subsequent_indent=" ")) + # "Other" option in preview + other_num = len(choices) + 1 + if other_num < 10: + other_num_prefix = str(other_num) + elif other_num == 10: + other_num_prefix = '0' + else: + other_num_prefix = ' ' other_label = ( - "❯ Other (type below)" if cli_ref._clarify_freetext - else "❯ Other (type your answer)" if selected == len(choices) - else " Other (type your answer)" + f"❯ {other_num_prefix}. Other (type below)" if cli_ref._clarify_freetext + else f"❯ {other_num_prefix}. Other (type your answer)" if selected == len(choices) + else f" {other_num_prefix}. Other (type your answer)" ) - preview_lines.extend(_wrap_panel_text(other_label, 60, subsequent_indent=" ")) + preview_lines.extend(_wrap_panel_text(other_label, 60, subsequent_indent=" ")) box_width = _panel_box_width("Hermes needs your input", preview_lines) inner_text_width = max(8, box_width - 2) + # Pre-wrap choices + Other option — these are mandatory. + choice_wrapped: list[tuple[int, str]] = [] + if choices: + for i, choice in enumerate(choices): + # Show number prefix for quick selection (1-9 for items 1-9, 0 for 10th item) + if i < 9: + num_prefix = str(i + 1) + elif i == 9: + num_prefix = '0' + else: + num_prefix = ' ' + if i == selected and not cli_ref._clarify_freetext: + prefix = f'❯ {num_prefix}. ' + else: + prefix = f' {num_prefix}. ' + for wrapped in _wrap_panel_text(f"{prefix}{choice}", inner_text_width, subsequent_indent=" "): + choice_wrapped.append((i, wrapped)) + # Trailing Other row(s) + other_idx = len(choices) + other_num = other_idx + 1 + if other_num < 10: + other_num_prefix = str(other_num) + elif other_num == 10: + other_num_prefix = '0' + else: + other_num_prefix = ' ' + if selected == other_idx and not cli_ref._clarify_freetext: + other_label_mand = f'❯ {other_num_prefix}. Other (type your answer)' + elif cli_ref._clarify_freetext: + other_label_mand = f'❯ {other_num_prefix}. Other (type below)' + else: + other_label_mand = f' {other_num_prefix}. Other (type your answer)' + other_wrapped = _wrap_panel_text(other_label_mand, inner_text_width, subsequent_indent=" ") + elif cli_ref._clarify_freetext: + # Freetext-only mode: the guidance line takes the place of choices. + other_wrapped = _wrap_panel_text( + "Type your answer in the prompt below, then press Enter.", + inner_text_width, + ) + else: + other_wrapped = [] + + # Budget the question so mandatory rows always render. + # Chrome layouts: + # full : top border + blank_after_title + blank_after_question + # + blank_before_bottom + bottom border = 5 rows + # tight: top border + bottom border = 2 rows (drop all blanks) + # + # reserved_below matches the approval-panel budget (~6 rows for + # spinner/tool-progress + status + input + separators + prompt). + term_rows = shutil.get_terminal_size((100, 24)).lines + chrome_full = 5 + chrome_tight = 2 + reserved_below = 6 + + available = max(0, term_rows - reserved_below) + mandatory_full = chrome_full + len(choice_wrapped) + len(other_wrapped) + + use_compact_chrome = mandatory_full > available + chrome_rows = chrome_tight if use_compact_chrome else chrome_full + + max_question_rows = max(1, available - chrome_rows - len(choice_wrapped) - len(other_wrapped)) + max_question_rows = min(max_question_rows, 12) # soft cap on huge terminals + + question_wrapped = _wrap_panel_text(question, inner_text_width) + if len(question_wrapped) > max_question_rows: + keep = max(1, max_question_rows - 1) + question_wrapped = question_wrapped[:keep] + ["… (question truncated)"] + lines = [] # Box top border lines.append(('class:clarify-border', '╭─ ')) lines.append(('class:clarify-title', 'Hermes needs your input')) lines.append(('class:clarify-border', ' ' + ('─' * max(0, box_width - len("Hermes needs your input") - 3)) + '╮\n')) - _append_blank_panel_line(lines, 'class:clarify-border', box_width) + if not use_compact_chrome: + _append_blank_panel_line(lines, 'class:clarify-border', box_width) - # Question text - for wrapped in _wrap_panel_text(question, inner_text_width): + # Question text (bounded) + for wrapped in question_wrapped: _append_panel_line(lines, 'class:clarify-border', 'class:clarify-question', wrapped, box_width) - _append_blank_panel_line(lines, 'class:clarify-border', box_width) + if not use_compact_chrome: + _append_blank_panel_line(lines, 'class:clarify-border', box_width) if cli_ref._clarify_freetext and not choices: - guidance = "Type your answer in the prompt below, then press Enter." - for wrapped in _wrap_panel_text(guidance, inner_text_width): + for wrapped in other_wrapped: _append_panel_line(lines, 'class:clarify-border', 'class:clarify-choice', wrapped, box_width) - _append_blank_panel_line(lines, 'class:clarify-border', box_width) + if not use_compact_chrome: + _append_blank_panel_line(lines, 'class:clarify-border', box_width) if choices: # Multiple-choice mode: show selectable options - for i, choice in enumerate(choices): + for i, wrapped in choice_wrapped: style = 'class:clarify-selected' if i == selected and not cli_ref._clarify_freetext else 'class:clarify-choice' - prefix = '❯ ' if i == selected and not cli_ref._clarify_freetext else ' ' - wrapped_lines = _wrap_panel_text(f"{prefix}{choice}", inner_text_width, subsequent_indent=" ") - for wrapped in wrapped_lines: - _append_panel_line(lines, 'class:clarify-border', style, wrapped, box_width) + _append_panel_line(lines, 'class:clarify-border', style, wrapped, box_width) - # "Other" option (5th line, only shown when choices exist) + # "Other" option (trailing row(s), only shown when choices exist) other_idx = len(choices) + # Calculate number prefix for "Other" option + other_num = other_idx + 1 + if other_num < 10: + other_num_prefix = str(other_num) + elif other_num == 10: + other_num_prefix = '0' + else: + other_num_prefix = ' ' + if selected == other_idx and not cli_ref._clarify_freetext: other_style = 'class:clarify-selected' - other_label = '❯ Other (type your answer)' elif cli_ref._clarify_freetext: other_style = 'class:clarify-active-other' - other_label = '❯ Other (type below)' else: other_style = 'class:clarify-choice' - other_label = ' Other (type your answer)' - for wrapped in _wrap_panel_text(other_label, inner_text_width, subsequent_indent=" "): + for wrapped in other_wrapped: _append_panel_line(lines, 'class:clarify-border', other_style, wrapped, box_width) - _append_blank_panel_line(lines, 'class:clarify-border', box_width) + if not use_compact_chrome: + _append_blank_panel_line(lines, 'class:clarify-border', box_width) lines.append(('class:clarify-border', '╰' + ('─' * box_width) + '╯\n')) return lines @@ -9216,7 +10193,7 @@ def _get_secret_display(): prompt = state.get("prompt") or f"Enter value for {state.get('var_name', 'secret')}" metadata = state.get("metadata") or {} help_text = metadata.get("help") - body = 'Enter secret below (hidden), or press Enter to skip' + body = 'Enter secret below (hidden), ESC or Ctrl+C to skip' content_lines = [prompt, body] if help_text: content_lines.insert(1, str(help_text)) @@ -9265,7 +10242,8 @@ def _get_model_picker_display(): if stage == "provider": title = "⚙ Model Picker — Select Provider" choices = [] - for p in state.get("providers") or []: + _providers = state.get("providers") + for p in _providers if isinstance(_providers, list) else []: count = p.get("total_models", len(p.get("models", []))) label = f"{p['name']} ({count} model{'s' if count != 1 else ''})" if p.get("is_current"): @@ -9285,6 +10263,22 @@ def _get_model_picker_display(): box_width = _panel_box_width(title, [hint] + choices, min_width=46, max_width=84) inner_text_width = max(8, box_width - 6) + selected = state.get("selected", 0) + + # Scrolling viewport: the panel renders into a Window with no max + # height, so without limiting visible items the bottom border and + # any items past the available terminal rows get clipped on long + # provider catalogs (e.g. Ollama Cloud's 36+ models). + try: + from prompt_toolkit.application import get_app + term_rows = get_app().output.get_size().rows + except Exception: + term_rows = shutil.get_terminal_size((100, 24)).lines + scroll_offset, visible = HermesCLI._compute_model_picker_viewport( + selected, state.get("_scroll_offset", 0), len(choices), term_rows, + ) + state["_scroll_offset"] = scroll_offset + lines = [] lines.append(('class:clarify-border', '╭─ ')) lines.append(('class:clarify-title', title)) @@ -9292,8 +10286,8 @@ def _get_model_picker_display(): _append_blank_panel_line(lines, 'class:clarify-border', box_width) _append_panel_line(lines, 'class:clarify-border', 'class:clarify-hint', hint, box_width) _append_blank_panel_line(lines, 'class:clarify-border', box_width) - selected = state.get("selected", 0) - for idx, choice in enumerate(choices): + for idx in range(scroll_offset, scroll_offset + visible): + choice = choices[idx] style = 'class:clarify-selected' if idx == selected else 'class:clarify-choice' prefix = '❯ ' if idx == selected else ' ' for wrapped in _wrap_panel_text(prefix + choice, inner_text_width, subsequent_indent=' '): @@ -9506,22 +10500,20 @@ def _resize_clear_ghosts(): app._on_resize = _resize_clear_ghosts def spinner_loop(): - import time as _time - last_idle_refresh = 0.0 while not self._should_exit: if not self._app: - _time.sleep(0.1) + time.sleep(0.1) continue if self._command_running: self._invalidate(min_interval=0.1) - _time.sleep(0.1) + time.sleep(0.1) else: - now = _time.monotonic() + now = time.monotonic() if now - last_idle_refresh >= 1.0: last_idle_refresh = now self._invalidate(min_interval=1.0) - _time.sleep(0.2) + time.sleep(0.2) spinner_thread = threading.Thread(target=spinner_loop, daemon=True) spinner_thread.start() @@ -9590,49 +10582,12 @@ def process_loop(): continue # Expand paste references back to full content - import re as _re - _paste_ref_re = _re.compile(r'\[Pasted text #\d+: \d+ lines \u2192 (.+?)\]') + _paste_ref_re = re.compile(r'\[Pasted text #\d+: \d+ lines \u2192 (.+?)\]') paste_refs = list(_paste_ref_re.finditer(user_input)) if isinstance(user_input, str) else [] if paste_refs: - def _expand_ref(m): - p = Path(m.group(1)) - return p.read_text(encoding="utf-8") if p.exists() else m.group(0) - expanded = _paste_ref_re.sub(_expand_ref, user_input) - total_lines = expanded.count('\n') + 1 - n_pastes = len(paste_refs) - _user_bar = f"[{_accent_hex()}]{'─' * 40}[/]" - print() - ChatConsole().print(_user_bar) - # Show any surrounding user text alongside the paste summary - split_parts = _paste_ref_re.split(user_input) - visible_user_text = " ".join( - split_parts[i].strip() for i in range(0, len(split_parts), 2) if split_parts[i].strip() - ) - if visible_user_text: - ChatConsole().print( - f"[bold {_accent_hex()}]\u25cf[/] [bold]{_escape(visible_user_text)}[/] " - f"[dim]({n_pastes} pasted block{'s' if n_pastes > 1 else ''}, {total_lines} lines total)[/]" - ) - else: - ChatConsole().print( - f"[bold {_accent_hex()}]\u25cf[/] [bold]{_escape(f'[Pasted text: {total_lines} lines]')}[/]" - ) - user_input = expanded - else: - _user_bar = f"[{_accent_hex()}]{'─' * 40}[/]" - if '\n' in user_input: - first_line = user_input.split('\n')[0] - line_count = user_input.count('\n') + 1 - print() - ChatConsole().print(_user_bar) - ChatConsole().print( - f"[bold {_accent_hex()}]●[/] [bold]{_escape(first_line)}[/] " - f"[dim](+{line_count - 1} lines)[/]" - ) - else: - print() - ChatConsole().print(_user_bar) - ChatConsole().print(f"[bold {_accent_hex()}]●[/] [bold]{_escape(user_input)}[/]") + user_input = self._expand_paste_references(user_input) + print() + self._print_user_message_preview(user_input) # Show image attachment count if submit_images: @@ -9698,8 +10653,35 @@ def _restart_recording(): # Register signal handlers for graceful shutdown on SSH disconnect / SIGTERM def _signal_handler(signum, frame): - """Handle SIGHUP/SIGTERM by triggering graceful cleanup.""" + """Handle SIGHUP/SIGTERM by triggering graceful cleanup. + + Calls ``self.agent.interrupt()`` first so the agent daemon + thread's poll loop sees the per-thread interrupt and kills the + tool's subprocess group via ``_kill_process`` (os.killpg). + Without this, the main thread dies from KeyboardInterrupt and + the daemon thread is killed with it — before it can run one + more poll iteration to clean up the subprocess, which was + spawned with ``os.setsid`` and therefore survives as an orphan + with PPID=1. + + Grace window (``HERMES_SIGTERM_GRACE``, default 1.5 s) gives + the daemon time to: detect the interrupt (next 200 ms poll) → + call _kill_process (SIGTERM + 1 s wait + SIGKILL if needed) → + return from _wait_for_process. ``time.sleep`` releases the + GIL so the daemon actually runs during the window. + """ logger.debug("Received signal %s, triggering graceful shutdown", signum) + try: + if getattr(self, "agent", None) and getattr(self, "_agent_running", False): + self.agent.interrupt(f"received signal {signum}") + try: + _grace = float(os.getenv("HERMES_SIGTERM_GRACE", "1.5")) + except (TypeError, ValueError): + _grace = 1.5 + if _grace > 0: + time.sleep(_grace) + except Exception: + pass # never block signal handling raise KeyboardInterrupt() try: @@ -9730,8 +10712,7 @@ def _suppress_closed_loop_errors(loop, context): # uv-managed Python, fd 0 can be invalid or unregisterable with the # asyncio selector, causing "KeyError: '0 is not registered'" (#6393). try: - import os as _os - _os.fstat(0) + os.fstat(0) except OSError: print( "Error: stdin (fd 0) is not available.\n" @@ -9853,6 +10834,8 @@ def main( w: bool = False, checkpoints: bool = False, pass_session_id: bool = False, + ignore_user_config: bool = False, + ignore_rules: bool = False, ): """ Hermes Agent CLI - Interactive AI Assistant @@ -9962,6 +10945,7 @@ def main( resume=resume, checkpoints=checkpoints, pass_session_id=pass_session_id, + ignore_rules=ignore_rules, ) if parsed_skills: @@ -10002,6 +10986,44 @@ def main( # Register cleanup for single-query mode (interactive mode registers in run()) atexit.register(_run_cleanup) + + # Also install signal handlers in single-query / `-q` mode. Interactive + # mode registers its own inside HermesCLI.run(), but `-q` runs + # cli.agent.run_conversation() below and AIAgent spawns worker threads + # for tools — so when SIGTERM arrives on the main thread, raising + # KeyboardInterrupt only unwinds the main thread, not the worker + # running _wait_for_process. Python then exits, the child subprocess + # (spawned with os.setsid, its own process group) is reparented to + # init and keeps running as an orphan. + # + # Fix: route SIGTERM/SIGHUP through agent.interrupt() which sets the + # per-thread interrupt flag the worker's poll loop checks every 200 ms. + # Give the worker a grace window to call _kill_process (SIGTERM to the + # process group, then SIGKILL after 1 s), then raise KeyboardInterrupt + # so main unwinds normally. HERMES_SIGTERM_GRACE overrides the 1.5 s + # default for debugging. + def _signal_handler_q(signum, frame): + logger.debug("Received signal %s in single-query mode", signum) + try: + _agent = getattr(cli, "agent", None) + if _agent is not None: + _agent.interrupt(f"received signal {signum}") + try: + _grace = float(os.getenv("HERMES_SIGTERM_GRACE", "1.5")) + except (TypeError, ValueError): + _grace = 1.5 + if _grace > 0: + time.sleep(_grace) + except Exception: + pass # never block signal handling + raise KeyboardInterrupt() + try: + import signal as _signal + _signal.signal(_signal.SIGTERM, _signal_handler_q) + if hasattr(_signal, "SIGHUP"): + _signal.signal(_signal.SIGHUP, _signal_handler_q) + except Exception: + pass # signal handler may fail in restricted environments # Handle single query mode if query or image: @@ -10024,19 +11046,33 @@ def main( if cli._init_agent( model_override=turn_route["model"], runtime_override=turn_route["runtime"], - route_label=turn_route["label"], request_overrides=turn_route.get("request_overrides"), ): cli.agent.quiet_mode = True cli.agent.suppress_status_output = True + # Suppress streaming display callbacks so stdout stays + # machine-readable (no styled "Hermes" box, no tool-gen + # status lines). The response is printed once below. + cli.agent.stream_delta_callback = None + cli.agent.tool_gen_callback = None result = cli.agent.run_conversation( user_message=effective_query, conversation_history=cli.conversation_history, ) + # Sync session_id if mid-run compression created a + # continuation session. The exit line below reports + # session_id to stderr for automation wrappers; without + # this sync it would point at the ended parent. + if ( + getattr(cli.agent, "session_id", None) + and cli.agent.session_id != cli.session_id + ): + cli.session_id = cli.agent.session_id response = result.get("final_response", "") if isinstance(result, dict) else str(result) if response: print(response) - print(f"\nsession_id: {cli.session_id}") + # Session ID goes to stderr so piped stdout is clean. + print(f"\nsession_id: {cli.session_id}", file=sys.stderr) # Ensure proper exit code for automation wrappers sys.exit(1 if isinstance(result, dict) and result.get("failed") else 0) diff --git a/cron/jobs.py b/cron/jobs.py index 47e0b66efa55..4d34b1534bfe 100644 --- a/cron/jobs.py +++ b/cron/jobs.py @@ -9,6 +9,7 @@ import json import logging import tempfile +import threading import os import re import uuid @@ -34,6 +35,11 @@ HERMES_DIR = get_hermes_home().resolve() CRON_DIR = HERMES_DIR / "cron" JOBS_FILE = CRON_DIR / "jobs.json" + +# In-process lock protecting load_jobs→modify→save_jobs cycles. +# Required when tick() runs jobs in parallel threads — without this, +# concurrent mark_job_run / advance_next_run calls can clobber each other. +_jobs_file_lock = threading.Lock() OUTPUT_DIR = CRON_DIR / "output" ONESHOT_GRACE_SECONDS = 120 @@ -378,6 +384,7 @@ def create_job( provider: Optional[str] = None, base_url: Optional[str] = None, script: Optional[str] = None, + enabled_toolsets: Optional[List[str]] = None, ) -> Dict[str, Any]: """ Create a new cron job. @@ -397,6 +404,9 @@ def create_job( script: Optional path to a Python script whose stdout is injected into the prompt each run. The script runs before the agent turn, and its output is prepended as context. Useful for data collection / change detection. + enabled_toolsets: Optional list of toolset names to restrict the agent to. + When set, only tools from these toolsets are loaded, reducing + token overhead. When omitted, all default tools are loaded. Returns: The created job dict @@ -427,6 +437,8 @@ def create_job( normalized_base_url = normalized_base_url or None normalized_script = str(script).strip() if isinstance(script, str) else None normalized_script = normalized_script or None + normalized_toolsets = [str(t).strip() for t in enabled_toolsets if str(t).strip()] if enabled_toolsets else None + normalized_toolsets = normalized_toolsets or None label_source = (prompt or (normalized_skills[0] if normalized_skills else None)) or "cron job" job = { @@ -458,6 +470,7 @@ def create_job( # Delivery configuration "deliver": deliver, "origin": origin, # Tracks where job was created for "origin" delivery + "enabled_toolsets": normalized_toolsets, } jobs = load_jobs() @@ -501,6 +514,12 @@ def update_job(job_id: str, updates: Dict[str, Any]) -> Optional[Dict[str, Any]] if schedule_changed: updated_schedule = updated["schedule"] + # The API may pass schedule as a raw string (e.g. "every 10m") + # instead of a pre-parsed dict. Normalize it the same way + # create_job() does so downstream code can call .get() safely. + if isinstance(updated_schedule, str): + updated_schedule = parse_schedule(updated_schedule) + updated["schedule"] = updated_schedule updated["schedule_display"] = updates.get( "schedule_display", updated_schedule.get("display", updated.get("schedule_display")), @@ -588,43 +607,44 @@ def mark_job_run(job_id: str, success: bool, error: Optional[str] = None, ``delivery_error`` is tracked separately from the agent error — a job can succeed (agent produced output) but fail delivery (platform down). """ - jobs = load_jobs() - for i, job in enumerate(jobs): - if job["id"] == job_id: - now = _hermes_now().isoformat() - job["last_run_at"] = now - job["last_status"] = "ok" if success else "error" - job["last_error"] = error if not success else None - # Track delivery failures separately — cleared on successful delivery - job["last_delivery_error"] = delivery_error - - # Increment completed count - if job.get("repeat"): - job["repeat"]["completed"] = job["repeat"].get("completed", 0) + 1 + with _jobs_file_lock: + jobs = load_jobs() + for i, job in enumerate(jobs): + if job["id"] == job_id: + now = _hermes_now().isoformat() + job["last_run_at"] = now + job["last_status"] = "ok" if success else "error" + job["last_error"] = error if not success else None + # Track delivery failures separately — cleared on successful delivery + job["last_delivery_error"] = delivery_error - # Check if we've hit the repeat limit - times = job["repeat"].get("times") - completed = job["repeat"]["completed"] - if times is not None and times > 0 and completed >= times: - # Remove the job (limit reached) - jobs.pop(i) - save_jobs(jobs) - return - - # Compute next run - job["next_run_at"] = compute_next_run(job["schedule"], now) + # Increment completed count + if job.get("repeat"): + job["repeat"]["completed"] = job["repeat"].get("completed", 0) + 1 + + # Check if we've hit the repeat limit + times = job["repeat"].get("times") + completed = job["repeat"]["completed"] + if times is not None and times > 0 and completed >= times: + # Remove the job (limit reached) + jobs.pop(i) + save_jobs(jobs) + return + + # Compute next run + job["next_run_at"] = compute_next_run(job["schedule"], now) - # If no next run (one-shot completed), disable - if job["next_run_at"] is None: - job["enabled"] = False - job["state"] = "completed" - elif job.get("state") != "paused": - job["state"] = "scheduled" + # If no next run (one-shot completed), disable + if job["next_run_at"] is None: + job["enabled"] = False + job["state"] = "completed" + elif job.get("state") != "paused": + job["state"] = "scheduled" - save_jobs(jobs) - return + save_jobs(jobs) + return - logger.warning("mark_job_run: job_id %s not found, skipping save", job_id) + logger.warning("mark_job_run: job_id %s not found, skipping save", job_id) def advance_next_run(job_id: str) -> bool: @@ -639,20 +659,21 @@ def advance_next_run(job_id: str) -> bool: Returns True if next_run_at was advanced, False otherwise. """ - jobs = load_jobs() - for job in jobs: - if job["id"] == job_id: - kind = job.get("schedule", {}).get("kind") - if kind not in ("cron", "interval"): + with _jobs_file_lock: + jobs = load_jobs() + for job in jobs: + if job["id"] == job_id: + kind = job.get("schedule", {}).get("kind") + if kind not in ("cron", "interval"): + return False + now = _hermes_now().isoformat() + new_next = compute_next_run(job["schedule"], now) + if new_next and new_next != job.get("next_run_at"): + job["next_run_at"] = new_next + save_jobs(jobs) + return True return False - now = _hermes_now().isoformat() - new_next = compute_next_run(job["schedule"], now) - if new_next and new_next != job.get("next_run_at"): - job["next_run_at"] = new_next - save_jobs(jobs) - return True - return False - return False + return False def get_due_jobs() -> List[Dict[str, Any]]: diff --git a/cron/scheduler.py b/cron/scheduler.py index e6db77c0989a..979770374435 100644 --- a/cron/scheduler.py +++ b/cron/scheduler.py @@ -10,6 +10,7 @@ import asyncio import concurrent.futures +import contextvars import json import logging import os @@ -26,7 +27,7 @@ except ImportError: msvcrt = None from pathlib import Path -from typing import Optional +from typing import List, Optional # Add parent directory to path for imports BEFORE repo-level imports. # Without this, standalone invocations (e.g. after `hermes update` reloads @@ -45,8 +46,36 @@ "telegram", "discord", "slack", "whatsapp", "signal", "matrix", "mattermost", "homeassistant", "dingtalk", "feishu", "wecom", "wecom_callback", "weixin", "sms", "email", "webhook", "bluebubbles", + "qqbot", }) +# Platforms that support a configured cron/notification home target, mapped to +# the environment variable used by gateway setup/runtime config. +_HOME_TARGET_ENV_VARS = { + "matrix": "MATRIX_HOME_ROOM", + "telegram": "TELEGRAM_HOME_CHANNEL", + "discord": "DISCORD_HOME_CHANNEL", + "slack": "SLACK_HOME_CHANNEL", + "signal": "SIGNAL_HOME_CHANNEL", + "mattermost": "MATTERMOST_HOME_CHANNEL", + "sms": "SMS_HOME_CHANNEL", + "email": "EMAIL_HOME_ADDRESS", + "dingtalk": "DINGTALK_HOME_CHANNEL", + "feishu": "FEISHU_HOME_CHANNEL", + "wecom": "WECOM_HOME_CHANNEL", + "weixin": "WEIXIN_HOME_CHANNEL", + "bluebubbles": "BLUEBUBBLES_HOME_CHANNEL", + "qqbot": "QQBOT_HOME_CHANNEL", +} + +# Legacy env var names kept for back-compat. Each entry is the current +# primary env var → the previous name. _get_home_target_chat_id falls +# back to the legacy name if the primary is unset, so users who set the +# old name before the rename keep working until they migrate. +_LEGACY_HOME_TARGET_ENV_VARS = { + "QQBOT_HOME_CHANNEL": "QQ_HOME_CHANNEL", +} + from cron.jobs import get_due_jobs, mark_job_run, save_job_output, advance_next_run # Sentinel: when a cron agent has nothing new to report, it can start its @@ -74,15 +103,28 @@ def _resolve_origin(job: dict) -> Optional[dict]: return None -def _resolve_delivery_target(job: dict) -> Optional[dict]: - """Resolve the concrete auto-delivery target for a cron job, if any.""" - deliver = job.get("deliver", "local") +def _get_home_target_chat_id(platform_name: str) -> str: + """Return the configured home target chat/room ID for a delivery platform.""" + env_var = _HOME_TARGET_ENV_VARS.get(platform_name.lower()) + if not env_var: + return "" + value = os.getenv(env_var, "") + if not value: + legacy = _LEGACY_HOME_TARGET_ENV_VARS.get(env_var) + if legacy: + value = os.getenv(legacy, "") + return value + + +def _resolve_single_delivery_target(job: dict, deliver_value: str) -> Optional[dict]: + """Resolve one concrete auto-delivery target for a cron job.""" + origin = _resolve_origin(job) - if deliver == "local": + if deliver_value == "local": return None - if deliver == "origin": + if deliver_value == "origin": if origin: return { "platform": origin["platform"], @@ -91,8 +133,8 @@ def _resolve_delivery_target(job: dict) -> Optional[dict]: } # Origin missing (e.g. job created via API/script) — try each # platform's home channel as a fallback instead of silently dropping. - for platform_name in ("matrix", "telegram", "discord", "slack", "bluebubbles"): - chat_id = os.getenv(f"{platform_name.upper()}_HOME_CHANNEL", "") + for platform_name in _HOME_TARGET_ENV_VARS: + chat_id = _get_home_target_chat_id(platform_name) if chat_id: logger.info( "Job '%s' has deliver=origin but no origin; falling back to %s home channel", @@ -106,8 +148,8 @@ def _resolve_delivery_target(job: dict) -> Optional[dict]: } return None - if ":" in deliver: - platform_name, rest = deliver.split(":", 1) + if ":" in deliver_value: + platform_name, rest = deliver_value.split(":", 1) platform_key = platform_name.lower() from tools.send_message_tool import _parse_target_ref @@ -137,7 +179,7 @@ def _resolve_delivery_target(job: dict) -> Optional[dict]: "thread_id": thread_id, } - platform_name = deliver + platform_name = deliver_value if origin and origin.get("platform") == platform_name: return { "platform": platform_name, @@ -147,7 +189,7 @@ def _resolve_delivery_target(job: dict) -> Optional[dict]: if platform_name.lower() not in _KNOWN_DELIVERY_PLATFORMS: return None - chat_id = os.getenv(f"{platform_name.upper()}_HOME_CHANNEL", "") + chat_id = _get_home_target_chat_id(platform_name) if not chat_id: return None @@ -158,6 +200,30 @@ def _resolve_delivery_target(job: dict) -> Optional[dict]: } +def _resolve_delivery_targets(job: dict) -> List[dict]: + """Resolve all concrete auto-delivery targets for a cron job (supports comma-separated deliver).""" + deliver = job.get("deliver", "local") + if deliver == "local": + return [] + parts = [p.strip() for p in str(deliver).split(",") if p.strip()] + seen = set() + targets = [] + for part in parts: + target = _resolve_single_delivery_target(job, part) + if target: + key = (target["platform"].lower(), str(target["chat_id"]), target.get("thread_id")) + if key not in seen: + seen.add(key) + targets.append(target) + return targets + + +def _resolve_delivery_target(job: dict) -> Optional[dict]: + """Resolve the concrete auto-delivery target for a cron job, if any.""" + targets = _resolve_delivery_targets(job) + return targets[0] if targets else None + + # Media extension sets — keep in sync with gateway/platforms/base.py:_process_message_background _AUDIO_EXTS = frozenset({'.ogg', '.opus', '.mp3', '.wav', '.m4a'}) _VIDEO_EXTS = frozenset({'.mp4', '.mov', '.avi', '.mkv', '.webm', '.3gp'}) @@ -186,7 +252,11 @@ def _send_media_via_adapter(adapter, chat_id: str, media_files: list, metadata: coro = adapter.send_document(chat_id=chat_id, file_path=media_path, metadata=metadata) future = asyncio.run_coroutine_threadsafe(coro, loop) - result = future.result(timeout=30) + try: + result = future.result(timeout=30) + except TimeoutError: + future.cancel() + raise if result and not getattr(result, "success", True): logger.warning( "Job '%s': media send failed for %s: %s", @@ -198,7 +268,7 @@ def _send_media_via_adapter(adapter, chat_id: str, media_files: list, metadata: def _deliver_result(job: dict, content: str, adapters=None, loop=None) -> Optional[str]: """ - Deliver job output to the configured target (origin chat, specific platform, etc.). + Deliver job output to the configured target(s) (origin chat, specific platform, etc.). When ``adapters`` and ``loop`` are provided (gateway is running), tries to use the live adapter first — this supports E2EE rooms (e.g. Matrix) where @@ -207,33 +277,14 @@ def _deliver_result(job: dict, content: str, adapters=None, loop=None) -> Option Returns None on success, or an error string on failure. """ - target = _resolve_delivery_target(job) - if not target: + targets = _resolve_delivery_targets(job) + if not targets: if job.get("deliver", "local") != "local": msg = f"no delivery target resolved for deliver={job.get('deliver', 'local')}" logger.warning("Job '%s': %s", job["id"], msg) return msg return None # local-only jobs don't deliver — not a failure - platform_name = target["platform"] - chat_id = target["chat_id"] - thread_id = target.get("thread_id") - - # Diagnostic: log thread_id for topic-aware delivery debugging - origin = job.get("origin") or {} - origin_thread = origin.get("thread_id") - if origin_thread and not thread_id: - logger.warning( - "Job '%s': origin has thread_id=%s but delivery target lost it " - "(deliver=%s, target=%s)", - job["id"], origin_thread, job.get("deliver", "local"), target, - ) - elif thread_id: - logger.debug( - "Job '%s': delivering to %s:%s thread_id=%s", - job["id"], platform_name, chat_id, thread_id, - ) - from tools.send_message_tool import _send_to_platform from gateway.config import load_gateway_config, Platform @@ -254,25 +305,8 @@ def _deliver_result(job: dict, content: str, adapters=None, loop=None) -> Option "email": Platform.EMAIL, "sms": Platform.SMS, "bluebubbles": Platform.BLUEBUBBLES, + "qqbot": Platform.QQBOT, } - platform = platform_map.get(platform_name.lower()) - if not platform: - msg = f"unknown platform '{platform_name}'" - logger.warning("Job '%s': %s", job["id"], msg) - return msg - - try: - config = load_gateway_config() - except Exception as e: - msg = f"failed to load gateway config: {e}" - logger.error("Job '%s': %s", job["id"], msg) - return msg - - pconfig = config.platforms.get(platform) - if not pconfig or not pconfig.enabled: - msg = f"platform '{platform_name}' not configured/enabled" - logger.warning("Job '%s': %s", job["id"], msg) - return msg # Optionally wrap the content with a header/footer so the user knows this # is a cron delivery. Wrapping is on by default; set cron.wrap_response: false @@ -286,11 +320,13 @@ def _deliver_result(job: dict, content: str, adapters=None, loop=None) -> Option if wrap_response: task_name = job.get("name", job["id"]) + job_id = job.get("id", "") delivery_content = ( f"Cronjob Response: {task_name}\n" + f"(job_id: {job_id})\n" f"-------------\n\n" f"{content}\n\n" - f"Note: The agent cannot see this message, and therefore cannot respond to it." + f"To stop or manage this job, send me a new message (e.g. \"stop reminder {task_name}\")." ) else: delivery_content = content @@ -299,67 +335,120 @@ def _deliver_result(job: dict, content: str, adapters=None, loop=None) -> Option from gateway.platforms.base import BasePlatformAdapter media_files, cleaned_delivery_content = BasePlatformAdapter.extract_media(delivery_content) - # Prefer the live adapter when the gateway is running — this supports E2EE - # rooms (e.g. Matrix) where the standalone HTTP path cannot encrypt. - runtime_adapter = (adapters or {}).get(platform) - if runtime_adapter is not None and loop is not None and getattr(loop, "is_running", lambda: False)(): - send_metadata = {"thread_id": thread_id} if thread_id else None - try: - # Send cleaned text (MEDIA tags stripped) — not the raw content - text_to_send = cleaned_delivery_content.strip() - adapter_ok = True - if text_to_send: - future = asyncio.run_coroutine_threadsafe( - runtime_adapter.send(chat_id, text_to_send, metadata=send_metadata), - loop, - ) - send_result = future.result(timeout=60) - if send_result and not getattr(send_result, "success", True): - err = getattr(send_result, "error", "unknown") - logger.warning( - "Job '%s': live adapter send to %s:%s failed (%s), falling back to standalone", - job["id"], platform_name, chat_id, err, - ) - adapter_ok = False # fall through to standalone path + try: + config = load_gateway_config() + except Exception as e: + msg = f"failed to load gateway config: {e}" + logger.error("Job '%s': %s", job["id"], msg) + return msg - # Send extracted media files as native attachments via the live adapter - if adapter_ok and media_files: - _send_media_via_adapter(runtime_adapter, chat_id, media_files, send_metadata, loop, job) + delivery_errors = [] - if adapter_ok: - logger.info("Job '%s': delivered to %s:%s via live adapter", job["id"], platform_name, chat_id) - return None - except Exception as e: + for target in targets: + platform_name = target["platform"] + chat_id = target["chat_id"] + thread_id = target.get("thread_id") + + # Diagnostic: log thread_id for topic-aware delivery debugging + origin = job.get("origin") or {} + origin_thread = origin.get("thread_id") + if origin_thread and not thread_id: logger.warning( - "Job '%s': live adapter delivery to %s:%s failed (%s), falling back to standalone", - job["id"], platform_name, chat_id, e, + "Job '%s': origin has thread_id=%s but delivery target lost it " + "(deliver=%s, target=%s)", + job["id"], origin_thread, job.get("deliver", "local"), target, + ) + elif thread_id: + logger.debug( + "Job '%s': delivering to %s:%s thread_id=%s", + job["id"], platform_name, chat_id, thread_id, ) - # Standalone path: run the async send in a fresh event loop (safe from any thread) - coro = _send_to_platform(platform, pconfig, chat_id, cleaned_delivery_content, thread_id=thread_id, media_files=media_files) - try: - result = asyncio.run(coro) - except RuntimeError: - # asyncio.run() checks for a running loop before awaiting the coroutine; - # when it raises, the original coro was never started — close it to - # prevent "coroutine was never awaited" RuntimeWarning, then retry in a - # fresh thread that has no running loop. - coro.close() - import concurrent.futures - with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: - future = pool.submit(asyncio.run, _send_to_platform(platform, pconfig, chat_id, cleaned_delivery_content, thread_id=thread_id, media_files=media_files)) - result = future.result(timeout=30) - except Exception as e: - msg = f"delivery to {platform_name}:{chat_id} failed: {e}" - logger.error("Job '%s': %s", job["id"], msg) - return msg + platform = platform_map.get(platform_name.lower()) + if not platform: + msg = f"unknown platform '{platform_name}'" + logger.warning("Job '%s': %s", job["id"], msg) + delivery_errors.append(msg) + continue - if result and result.get("error"): - msg = f"delivery error: {result['error']}" - logger.error("Job '%s': %s", job["id"], msg) - return msg + # Prefer the live adapter when the gateway is running — this supports E2EE + # rooms (e.g. Matrix) where the standalone HTTP path cannot encrypt. + runtime_adapter = (adapters or {}).get(platform) + delivered = False + if runtime_adapter is not None and loop is not None and getattr(loop, "is_running", lambda: False)(): + send_metadata = {"thread_id": thread_id} if thread_id else None + try: + # Send cleaned text (MEDIA tags stripped) — not the raw content + text_to_send = cleaned_delivery_content.strip() + adapter_ok = True + if text_to_send: + future = asyncio.run_coroutine_threadsafe( + runtime_adapter.send(chat_id, text_to_send, metadata=send_metadata), + loop, + ) + try: + send_result = future.result(timeout=60) + except TimeoutError: + future.cancel() + raise + if send_result and not getattr(send_result, "success", True): + err = getattr(send_result, "error", "unknown") + logger.warning( + "Job '%s': live adapter send to %s:%s failed (%s), falling back to standalone", + job["id"], platform_name, chat_id, err, + ) + adapter_ok = False # fall through to standalone path + + # Send extracted media files as native attachments via the live adapter + if adapter_ok and media_files: + _send_media_via_adapter(runtime_adapter, chat_id, media_files, send_metadata, loop, job) + + if adapter_ok: + logger.info("Job '%s': delivered to %s:%s via live adapter", job["id"], platform_name, chat_id) + delivered = True + except Exception as e: + logger.warning( + "Job '%s': live adapter delivery to %s:%s failed (%s), falling back to standalone", + job["id"], platform_name, chat_id, e, + ) + + if not delivered: + pconfig = config.platforms.get(platform) + if not pconfig or not pconfig.enabled: + msg = f"platform '{platform_name}' not configured/enabled" + logger.warning("Job '%s': %s", job["id"], msg) + delivery_errors.append(msg) + continue + + # Standalone path: run the async send in a fresh event loop (safe from any thread) + coro = _send_to_platform(platform, pconfig, chat_id, cleaned_delivery_content, thread_id=thread_id, media_files=media_files) + try: + result = asyncio.run(coro) + except RuntimeError: + # asyncio.run() checks for a running loop before awaiting the coroutine; + # when it raises, the original coro was never started — close it to + # prevent "coroutine was never awaited" RuntimeWarning, then retry in a + # fresh thread that has no running loop. + coro.close() + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: + future = pool.submit(asyncio.run, _send_to_platform(platform, pconfig, chat_id, cleaned_delivery_content, thread_id=thread_id, media_files=media_files)) + result = future.result(timeout=30) + except Exception as e: + msg = f"delivery to {platform_name}:{chat_id} failed: {e}" + logger.error("Job '%s': %s", job["id"], msg) + delivery_errors.append(msg) + continue - logger.info("Job '%s': delivered to %s:%s", job["id"], platform_name, chat_id) + if result and result.get("error"): + msg = f"delivery error: {result['error']}" + logger.error("Job '%s': %s", job["id"], msg) + delivery_errors.append(msg) + continue + + logger.info("Job '%s': delivered to %s:%s", job["id"], platform_name, chat_id) + + if delivery_errors: + return "; ".join(delivery_errors) return None @@ -482,15 +571,53 @@ def _run_job_script(script_path: str) -> tuple[bool, str]: return False, f"Script execution failed: {exc}" -def _build_job_prompt(job: dict) -> str: - """Build the effective prompt for a cron job, optionally loading one or more skills first.""" +def _parse_wake_gate(script_output: str) -> bool: + """Parse the last non-empty stdout line of a cron job's pre-check script + as a wake gate. + + The convention (ported from nanoclaw #1232): if the last stdout line is + JSON like ``{"wakeAgent": false}``, the agent is skipped entirely — no + LLM run, no delivery. Any other output (non-JSON, missing flag, gate + absent, or ``wakeAgent: true``) means wake the agent normally. + + Returns True if the agent should wake, False to skip. + """ + if not script_output: + return True + stripped_lines = [line for line in script_output.splitlines() if line.strip()] + if not stripped_lines: + return True + last_line = stripped_lines[-1].strip() + try: + gate = json.loads(last_line) + except (json.JSONDecodeError, ValueError): + return True + if not isinstance(gate, dict): + return True + return gate.get("wakeAgent", True) is not False + + +def _build_job_prompt(job: dict, prerun_script: Optional[tuple] = None) -> str: + """Build the effective prompt for a cron job, optionally loading one or more skills first. + + Args: + job: The cron job dict. + prerun_script: Optional ``(success, stdout)`` from a script that has + already been executed by the caller (e.g. for a wake-gate check). + When provided, the script is not re-executed and the cached + result is used for prompt injection. When omitted, the script + (if any) runs inline as before. + """ prompt = job.get("prompt", "") skills = job.get("skills") # Run data-collection script if configured, inject output as context. script_path = job.get("script") if script_path: - success, script_output = _run_job_script(script_path) + if prerun_script is not None: + success, script_output = prerun_script + else: + success, script_output = _run_job_script(script_path) if success: if script_output: prompt = ( @@ -592,21 +719,52 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: job_id = job["id"] job_name = job["name"] - prompt = _build_job_prompt(job) + + # Wake-gate: if this job has a pre-check script, run it BEFORE building + # the prompt so a ``{"wakeAgent": false}`` response can short-circuit + # the whole agent run. We pass the result into _build_job_prompt so + # the script is only executed once. + prerun_script = None + script_path = job.get("script") + if script_path: + prerun_script = _run_job_script(script_path) + _ran_ok, _script_output = prerun_script + if _ran_ok and not _parse_wake_gate(_script_output): + logger.info( + "Job '%s' (ID: %s): wakeAgent=false, skipping agent run", + job_name, job_id, + ) + silent_doc = ( + f"# Cron Job: {job_name}\n\n" + f"**Job ID:** {job_id}\n" + f"**Run Time:** {_hermes_now().strftime('%Y-%m-%d %H:%M:%S')}\n\n" + "Script gate returned `wakeAgent=false` — agent skipped.\n" + ) + return True, silent_doc, SILENT_MARKER, None + + prompt = _build_job_prompt(job, prerun_script=prerun_script) origin = _resolve_origin(job) _cron_session_id = f"cron_{job_id}_{_hermes_now().strftime('%Y%m%d_%H%M%S')}" logger.info("Running job '%s' (ID: %s)", job_name, job_id) logger.info("Prompt: %s", prompt[:100]) + # Mark this as a cron session so the approval system can apply cron_mode. + # This env var is process-wide and persists for the lifetime of the + # scheduler process — every job this process runs is a cron job. + os.environ["HERMES_CRON_SESSION"] = "1" + + # Use ContextVars for per-job session/delivery state so parallel jobs + # don't clobber each other's targets (os.environ is process-global). + from gateway.session_context import set_session_vars, clear_session_vars, _VAR_MAP + + _ctx_tokens = set_session_vars( + platform=origin["platform"] if origin else "", + chat_id=str(origin["chat_id"]) if origin else "", + chat_name=origin.get("chat_name", "") if origin else "", + ) + try: - # Inject origin context so the agent's send_message tool knows the chat. - # Must be INSIDE the try block so the finally cleanup always runs. - if origin: - os.environ["HERMES_SESSION_PLATFORM"] = origin["platform"] - os.environ["HERMES_SESSION_CHAT_ID"] = str(origin["chat_id"]) - if origin.get("chat_name"): - os.environ["HERMES_SESSION_CHAT_NAME"] = origin["chat_name"] # Re-read .env and config.yaml fresh every run so provider/key # changes take effect without a gateway restart. from dotenv import load_dotenv @@ -617,10 +775,10 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: delivery_target = _resolve_delivery_target(job) if delivery_target: - os.environ["HERMES_CRON_AUTO_DELIVER_PLATFORM"] = delivery_target["platform"] - os.environ["HERMES_CRON_AUTO_DELIVER_CHAT_ID"] = str(delivery_target["chat_id"]) + _VAR_MAP["HERMES_CRON_AUTO_DELIVER_PLATFORM"].set(delivery_target["platform"]) + _VAR_MAP["HERMES_CRON_AUTO_DELIVER_CHAT_ID"].set(str(delivery_target["chat_id"])) if delivery_target.get("thread_id") is not None: - os.environ["HERMES_CRON_AUTO_DELIVER_THREAD_ID"] = str(delivery_target["thread_id"]) + _VAR_MAP["HERMES_CRON_AUTO_DELIVER_THREAD_ID"].set(str(delivery_target["thread_id"])) model = job.get("model") or os.getenv("HERMES_MODEL") or "" @@ -659,14 +817,13 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: prefill_messages = None prefill_file = os.getenv("HERMES_PREFILL_MESSAGES_FILE", "") or _cfg.get("prefill_messages_file", "") if prefill_file: - import json as _json pfpath = Path(prefill_file).expanduser() if not pfpath.is_absolute(): pfpath = _hermes_home / pfpath if pfpath.exists(): try: with open(pfpath, "r", encoding="utf-8") as _pf: - prefill_messages = _json.load(_pf) + prefill_messages = json.load(_pf) if not isinstance(prefill_messages, list): prefill_messages = None except Exception as e: @@ -678,7 +835,6 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: # Provider routing pr = _cfg.get("provider_routing", {}) - smart_routing = _cfg.get("smart_model_routing", {}) or {} from hermes_cli.runtime_provider import ( resolve_runtime_provider, @@ -695,24 +851,9 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: message = format_runtime_provider_error(exc) raise RuntimeError(message) from exc - from agent.smart_model_routing import resolve_turn_route - turn_route = resolve_turn_route( - prompt, - smart_routing, - { - "model": model, - "api_key": runtime.get("api_key"), - "base_url": runtime.get("base_url"), - "provider": runtime.get("provider"), - "api_mode": runtime.get("api_mode"), - "command": runtime.get("command"), - "args": list(runtime.get("args") or []), - }, - ) - fallback_model = _cfg.get("fallback_providers") or _cfg.get("fallback_model") or None credential_pool = None - runtime_provider = str(turn_route["runtime"].get("provider") or "").strip().lower() + runtime_provider = str(runtime.get("provider") or "").strip().lower() if runtime_provider: try: from agent.credential_pool import load_pool @@ -729,13 +870,13 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: logger.debug("Job '%s': failed to load credential pool for %s: %s", job_id, runtime_provider, e) agent = AIAgent( - model=turn_route["model"], - api_key=turn_route["runtime"].get("api_key"), - base_url=turn_route["runtime"].get("base_url"), - provider=turn_route["runtime"].get("provider"), - api_mode=turn_route["runtime"].get("api_mode"), - acp_command=turn_route["runtime"].get("command"), - acp_args=turn_route["runtime"].get("args"), + model=model, + api_key=runtime.get("api_key"), + base_url=runtime.get("base_url"), + provider=runtime.get("provider"), + api_mode=runtime.get("api_mode"), + acp_command=runtime.get("command"), + acp_args=runtime.get("args"), max_iterations=max_iterations, reasoning_config=reasoning_config, prefill_messages=prefill_messages, @@ -745,6 +886,7 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: providers_ignored=pr.get("ignore"), providers_order=pr.get("order"), provider_sort=pr.get("sort"), + enabled_toolsets=job.get("enabled_toolsets") or None, disabled_toolsets=["cronjob", "messaging", "clarify"], quiet_mode=True, skip_context_files=True, # Don't inject SOUL.md/AGENTS.md from scheduler cwd @@ -766,7 +908,11 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: _cron_inactivity_limit = _cron_timeout if _cron_timeout > 0 else None _POLL_INTERVAL = 5.0 _cron_pool = concurrent.futures.ThreadPoolExecutor(max_workers=1) - _cron_future = _cron_pool.submit(agent.run_conversation, prompt) + # Preserve scheduler-scoped ContextVar state (for example skill-declared + # env passthrough registrations) when the cron run hops into the worker + # thread used for inactivity timeout monitoring. + _cron_context = contextvars.copy_context() + _cron_future = _cron_pool.submit(_cron_context.run, agent.run_conversation, prompt) _inactivity_timeout = False try: if _cron_inactivity_limit is None: @@ -827,7 +973,16 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: f"— last activity: {_last_desc}" ) + # Guard against non-dict returns from run_conversation under error conditions + if not isinstance(result, dict): + raise RuntimeError( + f"agent.run_conversation returned {type(result).__name__} instead of dict: {result!r}" + ) + final_response = result.get("final_response", "") or "" + # Strip leaked placeholder text that upstream may inject on empty completions. + if final_response.strip() == "(No response generated)": + final_response = "" # Use a separate variable for log display; keep final_response clean # for delivery logic (empty response = no delivery). logged_response = final_response if final_response else "(No response generated)" @@ -873,16 +1028,8 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: return False, output, "", error_msg finally: - # Clean up injected env vars so they don't leak to other jobs - for key in ( - "HERMES_SESSION_PLATFORM", - "HERMES_SESSION_CHAT_ID", - "HERMES_SESSION_CHAT_NAME", - "HERMES_CRON_AUTO_DELIVER_PLATFORM", - "HERMES_CRON_AUTO_DELIVER_CHAT_ID", - "HERMES_CRON_AUTO_DELIVER_THREAD_ID", - ): - os.environ.pop(key, None) + # Clean up ContextVar session/delivery state for this job. + clear_session_vars(_ctx_tokens) if _session_db: try: _session_db.end_session(_cron_session_id, "cron_complete") @@ -935,15 +1082,41 @@ def tick(verbose: bool = True, adapters=None, loop=None) -> int: if verbose: logger.info("%s - %s job(s) due", _hermes_now().strftime('%H:%M:%S'), len(due_jobs)) - executed = 0 + # Advance next_run_at for all recurring jobs FIRST, under the file lock, + # before any execution begins. This preserves at-most-once semantics. for job in due_jobs: + advance_next_run(job["id"]) + + # Resolve max parallel workers: env var > config.yaml > unbounded. + # Set HERMES_CRON_MAX_PARALLEL=1 to restore old serial behaviour. + _max_workers: Optional[int] = None + try: + _env_par = os.getenv("HERMES_CRON_MAX_PARALLEL", "").strip() + if _env_par: + _max_workers = int(_env_par) or None + except (ValueError, TypeError): + logger.warning("Invalid HERMES_CRON_MAX_PARALLEL value; defaulting to unbounded") + if _max_workers is None: try: - # For recurring jobs (cron/interval), advance next_run_at to the - # next future occurrence BEFORE execution. This way, if the - # process crashes mid-run, the job won't re-fire on restart. - # One-shot jobs are left alone so they can retry on restart. - advance_next_run(job["id"]) + _ucfg = load_config() or {} + _cfg_par = ( + _ucfg.get("cron", {}) if isinstance(_ucfg, dict) else {} + ).get("max_parallel_jobs") + if _cfg_par is not None: + _max_workers = int(_cfg_par) or None + except Exception: + pass + if verbose: + logger.info( + "Running %d job(s) in parallel (max_workers=%s)", + len(due_jobs), + _max_workers if _max_workers else "unbounded", + ) + + def _process_job(job: dict) -> bool: + """Run one due job end-to-end: execute, save, deliver, mark.""" + try: success, output, final_response, error = run_job(job) output_file = save_job_output(job["id"], output) @@ -967,14 +1140,31 @@ def tick(verbose: bool = True, adapters=None, loop=None) -> int: delivery_error = str(de) logger.error("Delivery failed for job %s: %s", job["id"], de) + # Treat empty final_response as a soft failure so last_status + # is not "ok" — the agent ran but produced nothing useful. + # (issue #8585) + if success and not final_response: + success = False + error = "Agent completed but produced empty response (model error, timeout, or misconfiguration)" + mark_job_run(job["id"], success, error, delivery_error=delivery_error) - executed += 1 + return True except Exception as e: logger.error("Error processing job %s: %s", job['id'], e) mark_job_run(job["id"], False, str(e)) - - return executed + return False + + # Run all due jobs concurrently, each in its own ContextVar copy + # so session/delivery state stays isolated per-thread. + with concurrent.futures.ThreadPoolExecutor(max_workers=_max_workers) as _tick_pool: + _futures = [] + for job in due_jobs: + _ctx = contextvars.copy_context() + _futures.append(_tick_pool.submit(_ctx.run, _process_job, job)) + _results = [f.result() for f in _futures] + + return sum(_results) finally: if fcntl: fcntl.flock(lock_fd, fcntl.LOCK_UN) diff --git a/desktop/.env.example b/desktop/.env.example index 470187528d66..785d04d498f5 100644 --- a/desktop/.env.example +++ b/desktop/.env.example @@ -1,2 +1,8 @@ # PostHog analytics key — used by both main (posthog-node) and renderer (posthog-js). VITE_POSTHOG_API_KEY=phc_your_key_here + +# atomic-bot-backend base URL. The renderer's atomic-backend-api client falls +# back to https://api.atomicbot.ai when this is unset. Set to a local dev +# instance (e.g. http://localhost:8787) when working against atomic-bot-backend +# directly. +VITE_ATOMIC_BACKEND_URL=https://api-stage.atomicbot.ai diff --git a/desktop/package-lock.json b/desktop/package-lock.json index 7d0bf350de63..27714673e911 100644 --- a/desktop/package-lock.json +++ b/desktop/package-lock.json @@ -1,12 +1,13 @@ { "name": "hermes-desktop", - "version": "0.1.31", + "version": "0.1.35", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "hermes-desktop", - "version": "0.1.31", + "version": "0.1.35", + "hasInstallScript": true, "dependencies": { "chokidar": "^3.6.0", "electron-store": "^8.2.0", @@ -21,18 +22,22 @@ "@electron/rebuild": "^4.0.3", "@monaco-editor/react": "^4.7.0", "@reduxjs/toolkit": "^2.9.0", + "@testing-library/jest-dom": "^6.9.1", + "@testing-library/react": "^16.3.2", "@types/js-yaml": "^4.0.9", "@types/node": "^22.0.0", "@types/react": "^19.2.10", "@types/react-dom": "^19.2.3", "@types/remove-markdown": "^0.3.4", "@vitejs/plugin-react": "^5.1.3", + "@vitest/ui": "^4.1.5", "@xterm/addon-fit": "^0.11.0", "@xterm/xterm": "^6.0.0", "cross-env": "^10.1.0", "electron": "^33.0.0", "electron-builder": "^25.0.0", "esbuild": "^0.28.0", + "jsdom": "^29.1.0", "monaco-editor": "^0.55.1", "react": "^19.2.4", "react-dom": "^19.2.4", @@ -44,7 +49,8 @@ "remark-math": "^6.0.0", "remove-markdown": "^0.6.3", "typescript": "^5.6.0", - "vite": "^7.3.1" + "vite": "^7.3.1", + "vitest": "^4.1.5" }, "engines": { "node": ">=18.0.0" @@ -53,6 +59,64 @@ "appdmg": "^0.6.6" } }, + "node_modules/@adobe/css-tools": { + "version": "4.4.4", + "resolved": "https://registry.npmjs.org/@adobe/css-tools/-/css-tools-4.4.4.tgz", + "integrity": "sha512-Elp+iwUx5rN5+Y8xLt5/GRoG20WGoDCQ/1Fb+1LiGtvwbDavuSk0jhD/eZdckHAuzcDzccnkv+rEjyWfRx18gg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@asamuzakjp/css-color": { + "version": "5.1.11", + "resolved": "https://registry.npmjs.org/@asamuzakjp/css-color/-/css-color-5.1.11.tgz", + "integrity": "sha512-KVw6qIiCTUQhByfTd78h2yD1/00waTmm9uy/R7Ck/ctUyAPj+AEDLkQIdJW0T8+qGgj3j5bpNKK7Q3G+LedJWg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@asamuzakjp/generational-cache": "^1.0.1", + "@csstools/css-calc": "^3.2.0", + "@csstools/css-color-parser": "^4.1.0", + "@csstools/css-parser-algorithms": "^4.0.0", + "@csstools/css-tokenizer": "^4.0.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/@asamuzakjp/dom-selector": { + "version": "7.1.1", + "resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-7.1.1.tgz", + "integrity": "sha512-67RZDnYRc8H/8MLDgQCDE//zoqVFwajkepHZgmXrbwybzXOEwOWGPYGmALYl9J2DOLfFPPs6kKCqmbzV895hTQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@asamuzakjp/generational-cache": "^1.0.1", + "@asamuzakjp/nwsapi": "^2.3.9", + "bidi-js": "^1.0.3", + "css-tree": "^3.2.1", + "is-potential-custom-element-name": "^1.0.1" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/@asamuzakjp/generational-cache": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@asamuzakjp/generational-cache/-/generational-cache-1.0.1.tgz", + "integrity": "sha512-wajfB8KqzMCN2KGNFdLkReeHncd0AslUSrvHVvvYWuU8ghncRJoA50kT3zP9MVL0+9g4/67H+cdvBskj9THPzg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/@asamuzakjp/nwsapi": { + "version": "2.3.9", + "resolved": "https://registry.npmjs.org/@asamuzakjp/nwsapi/-/nwsapi-2.3.9.tgz", + "integrity": "sha512-n8GuYSrI9bF7FFZ/SjhwevlHc8xaVlb/7HmHelnc/PZXBD2ZR49NnN9sMMuDdEGPeeRQ5d0hqlSlEpgCX3Wl0Q==", + "dev": true, + "license": "MIT" + }, "node_modules/@babel/code-frame": { "version": "7.29.0", "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.0.tgz", @@ -304,6 +368,16 @@ "@babel/core": "^7.0.0-0" } }, + "node_modules/@babel/runtime": { + "version": "7.29.2", + "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.2.tgz", + "integrity": "sha512-JiDShH45zKHWyGe4ZNVRrCjBz8Nh9TMmZG1kh4QTK8hCBTWBi8Da+i7s1fJw7/lYpM4ccepSNfqzZ/QvABBi5g==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, "node_modules/@babel/template": { "version": "7.28.6", "resolved": "https://registry.npmjs.org/@babel/template/-/template-7.28.6.tgz", @@ -352,6 +426,159 @@ "node": ">=6.9.0" } }, + "node_modules/@bramus/specificity": { + "version": "2.4.2", + "resolved": "https://registry.npmjs.org/@bramus/specificity/-/specificity-2.4.2.tgz", + "integrity": "sha512-ctxtJ/eA+t+6q2++vj5j7FYX3nRu311q1wfYH3xjlLOsczhlhxAg2FWNUXhpGvAw3BWo1xBcvOV6/YLc2r5FJw==", + "dev": true, + "license": "MIT", + "dependencies": { + "css-tree": "^3.0.0" + }, + "bin": { + "specificity": "bin/cli.js" + } + }, + "node_modules/@csstools/color-helpers": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-6.0.2.tgz", + "integrity": "sha512-LMGQLS9EuADloEFkcTBR3BwV/CGHV7zyDxVRtVDTwdI2Ca4it0CCVTT9wCkxSgokjE5Ho41hEPgb8OEUwoXr6Q==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT-0", + "engines": { + "node": ">=20.19.0" + } + }, + "node_modules/@csstools/css-calc": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/@csstools/css-calc/-/css-calc-3.2.0.tgz", + "integrity": "sha512-bR9e6o2BDB12jzN/gIbjHa5wLJ4UjD1CB9pM7ehlc0ddk6EBz+yYS1EV2MF55/HUxrHcB/hehAyt5vhsA3hx7w==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-parser-algorithms": "^4.0.0", + "@csstools/css-tokenizer": "^4.0.0" + } + }, + "node_modules/@csstools/css-color-parser": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/@csstools/css-color-parser/-/css-color-parser-4.1.0.tgz", + "integrity": "sha512-U0KhLYmy2GVj6q4T3WaAe6NPuFYCPQoE3b0dRGxejWDgcPp8TP7S5rVdM5ZrFaqu4N67X8YaPBw14dQSYx3IyQ==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "dependencies": { + "@csstools/color-helpers": "^6.0.2", + "@csstools/css-calc": "^3.2.0" + }, + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-parser-algorithms": "^4.0.0", + "@csstools/css-tokenizer": "^4.0.0" + } + }, + "node_modules/@csstools/css-parser-algorithms": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@csstools/css-parser-algorithms/-/css-parser-algorithms-4.0.0.tgz", + "integrity": "sha512-+B87qS7fIG3L5h3qwJ/IFbjoVoOe/bpOdh9hAjXbvx0o8ImEmUsGXN0inFOnk2ChCFgqkkGFQ+TpM5rbhkKe4w==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-tokenizer": "^4.0.0" + } + }, + "node_modules/@csstools/css-syntax-patches-for-csstree": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/@csstools/css-syntax-patches-for-csstree/-/css-syntax-patches-for-csstree-1.1.3.tgz", + "integrity": "sha512-SH60bMfrRCJF3morcdk57WklujF4Jr/EsQUzqkarfHXEFcAR1gg7fS/chAE922Sehgzc1/+Tz5H3Ypa1HiEKrg==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT-0", + "peerDependencies": { + "css-tree": "^3.2.1" + }, + "peerDependenciesMeta": { + "css-tree": { + "optional": true + } + } + }, + "node_modules/@csstools/css-tokenizer": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@csstools/css-tokenizer/-/css-tokenizer-4.0.0.tgz", + "integrity": "sha512-QxULHAm7cNu72w97JUNCBFODFaXpbDg+dP8b/oWFAZ2MTRppA3U00Y2L1HqaS4J6yBqxwa/Y3nMBaxVKbB/NsA==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + } + }, "node_modules/@develar/schema-utils": { "version": "2.6.5", "resolved": "https://registry.npmjs.org/@develar/schema-utils/-/schema-utils-2.6.5.tgz", @@ -1208,6 +1435,24 @@ "node": ">=18" } }, + "node_modules/@exodus/bytes": { + "version": "1.15.0", + "resolved": "https://registry.npmjs.org/@exodus/bytes/-/bytes-1.15.0.tgz", + "integrity": "sha512-UY0nlA+feH81UGSHv92sLEPLCeZFjXOuHhrIo0HQydScuQc8s0A7kL/UdgwgDq8g8ilksmuoF35YVTNphV2aBQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + }, + "peerDependencies": { + "@noble/hashes": "^1.8.0 || ^2.0.0" + }, + "peerDependenciesMeta": { + "@noble/hashes": { + "optional": true + } + } + }, "node_modules/@gar/promisify": { "version": "1.1.3", "resolved": "https://registry.npmjs.org/@gar/promisify/-/promisify-1.1.3.tgz", @@ -1816,6 +2061,13 @@ "node": ">=14" } }, + "node_modules/@polka/url": { + "version": "1.0.0-next.29", + "resolved": "https://registry.npmjs.org/@polka/url/-/url-1.0.0-next.29.tgz", + "integrity": "sha512-wwQAWhWSuHaag8c4q/KN/vCoeOJYshAIvMQwD4GpSb3OiZklFfvAgmj0VCBBImRpuF/aFgIRzllXlVX93Jevww==", + "dev": true, + "license": "MIT" + }, "node_modules/@posthog/core": { "version": "1.25.2", "resolved": "https://registry.npmjs.org/@posthog/core/-/core-1.25.2.tgz", @@ -2316,6 +2568,82 @@ "node": ">=10" } }, + "node_modules/@testing-library/dom": { + "version": "10.4.1", + "resolved": "https://registry.npmjs.org/@testing-library/dom/-/dom-10.4.1.tgz", + "integrity": "sha512-o4PXJQidqJl82ckFaXUeoAW+XysPLauYI43Abki5hABd853iMhitooc6znOnczgbTYmEP6U6/y1ZyKAIsvMKGg==", + "dev": true, + "license": "MIT", + "peer": true, + "dependencies": { + "@babel/code-frame": "^7.10.4", + "@babel/runtime": "^7.12.5", + "@types/aria-query": "^5.0.1", + "aria-query": "5.3.0", + "dom-accessibility-api": "^0.5.9", + "lz-string": "^1.5.0", + "picocolors": "1.1.1", + "pretty-format": "^27.0.2" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/@testing-library/jest-dom": { + "version": "6.9.1", + "resolved": "https://registry.npmjs.org/@testing-library/jest-dom/-/jest-dom-6.9.1.tgz", + "integrity": "sha512-zIcONa+hVtVSSep9UT3jZ5rizo2BsxgyDYU7WFD5eICBE7no3881HGeb/QkGfsJs6JTkY1aQhT7rIPC7e+0nnA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@adobe/css-tools": "^4.4.0", + "aria-query": "^5.0.0", + "css.escape": "^1.5.1", + "dom-accessibility-api": "^0.6.3", + "picocolors": "^1.1.1", + "redent": "^3.0.0" + }, + "engines": { + "node": ">=14", + "npm": ">=6", + "yarn": ">=1" + } + }, + "node_modules/@testing-library/jest-dom/node_modules/dom-accessibility-api": { + "version": "0.6.3", + "resolved": "https://registry.npmjs.org/dom-accessibility-api/-/dom-accessibility-api-0.6.3.tgz", + "integrity": "sha512-7ZgogeTnjuHbo+ct10G9Ffp0mif17idi0IyWNVA/wcwcm7NPOD/WEHVP3n7n3MhXqxoIYm8d6MuZohYWIZ4T3w==", + "dev": true, + "license": "MIT" + }, + "node_modules/@testing-library/react": { + "version": "16.3.2", + "resolved": "https://registry.npmjs.org/@testing-library/react/-/react-16.3.2.tgz", + "integrity": "sha512-XU5/SytQM+ykqMnAnvB2umaJNIOsLF3PVv//1Ew4CTcpz0/BRyy/af40qqrt7SjKpDdT1saBMc42CUok5gaw+g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/runtime": "^7.12.5" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "@testing-library/dom": "^10.0.0", + "@types/react": "^18.0.0 || ^19.0.0", + "@types/react-dom": "^18.0.0 || ^19.0.0", + "react": "^18.0.0 || ^19.0.0", + "react-dom": "^18.0.0 || ^19.0.0" + }, + "peerDependenciesMeta": { + "@types/react": { + "optional": true + }, + "@types/react-dom": { + "optional": true + } + } + }, "node_modules/@tootallnate/once": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/@tootallnate/once/-/once-2.0.0.tgz", @@ -2326,6 +2654,14 @@ "node": ">= 10" } }, + "node_modules/@types/aria-query": { + "version": "5.0.4", + "resolved": "https://registry.npmjs.org/@types/aria-query/-/aria-query-5.0.4.tgz", + "integrity": "sha512-rfT93uj5s0PRL7EzccGMs3brplhcrghnDoV26NqKhCAS1hVo+WdNsPvE/yb6ilfr5hi2MEk6d5EWJTKdxg8jVw==", + "dev": true, + "license": "MIT", + "peer": true + }, "node_modules/@types/babel__core": { "version": "7.20.5", "resolved": "https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz", @@ -2384,6 +2720,17 @@ "@types/responselike": "^1.0.0" } }, + "node_modules/@types/chai": { + "version": "5.2.3", + "resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz", + "integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/deep-eql": "*", + "assertion-error": "^2.0.1" + } + }, "node_modules/@types/debug": { "version": "4.1.13", "resolved": "https://registry.npmjs.org/@types/debug/-/debug-4.1.13.tgz", @@ -2394,6 +2741,13 @@ "@types/ms": "*" } }, + "node_modules/@types/deep-eql": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz", + "integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==", + "dev": true, + "license": "MIT" + }, "node_modules/@types/estree": { "version": "1.0.8", "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz", @@ -2605,6 +2959,148 @@ "vite": "^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0" } }, + "node_modules/@vitest/expect": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.5.tgz", + "integrity": "sha512-PWBaRY5JoKuRnHlUHfpV/KohFylaDZTupcXN1H9vYryNLOnitSw60Mw9IAE2r67NbwwzBw/Cc/8q9BK3kIX8Kw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@standard-schema/spec": "^1.1.0", + "@types/chai": "^5.2.2", + "@vitest/spy": "4.1.5", + "@vitest/utils": "4.1.5", + "chai": "^6.2.2", + "tinyrainbow": "^3.1.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/mocker": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.5.tgz", + "integrity": "sha512-/x2EmFC4mT4NNzqvC3fmesuV97w5FC903KPmey4gsnJiMQ3Be1IlDKVaDaG8iqaLFHqJ2FVEkxZk5VmeLjIItw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/spy": "4.1.5", + "estree-walker": "^3.0.3", + "magic-string": "^0.30.21" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "msw": "^2.4.9", + "vite": "^6.0.0 || ^7.0.0 || ^8.0.0" + }, + "peerDependenciesMeta": { + "msw": { + "optional": true + }, + "vite": { + "optional": true + } + } + }, + "node_modules/@vitest/pretty-format": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.5.tgz", + "integrity": "sha512-7I3q6l5qr03dVfMX2wCo9FxwSJbPdwKjy2uu/YPpU3wfHvIL4QHwVRp57OfGrDFeUJ8/8QdfBKIV12FTtLn00g==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^3.1.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/runner": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.5.tgz", + "integrity": "sha512-2D+o7Pr82IEO46YPpoA/YU0neeyr6FTerQb5Ro7BUnBuv6NQtT/kmVnczngiMEBhzgqz2UZYl5gArejsyERDSQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/utils": "4.1.5", + "pathe": "^2.0.3" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/snapshot": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.5.tgz", + "integrity": "sha512-zypXEt4KH/XgKGPUz4eC2AvErYx0My5hfL8oDb1HzGFpEk1P62bxSohdyOmvz+d9UJwanI68MKwr2EquOaOgMQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "4.1.5", + "@vitest/utils": "4.1.5", + "magic-string": "^0.30.21", + "pathe": "^2.0.3" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/spy": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.5.tgz", + "integrity": "sha512-2lNOsh6+R2Idnf1TCZqSwYlKN2E/iDlD8sgU59kYVl+OMDmvldO1VDk39smRfpUNwYpNRVn3w4YfuC7KfbBnkQ==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/ui": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/@vitest/ui/-/ui-4.1.5.tgz", + "integrity": "sha512-3Z9HNFiV0IF1fk0JPiK+7kE1GcaIPefQQIBYur6PM5yFIq6agys3uqP/0t966e1wXfmjbRCHDe7qW236Xjwnag==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/utils": "4.1.5", + "fflate": "^0.8.2", + "flatted": "^3.4.2", + "pathe": "^2.0.3", + "sirv": "^3.0.2", + "tinyglobby": "^0.2.15", + "tinyrainbow": "^3.1.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "vitest": "4.1.5" + } + }, + "node_modules/@vitest/ui/node_modules/fflate": { + "version": "0.8.2", + "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.2.tgz", + "integrity": "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==", + "dev": true, + "license": "MIT" + }, + "node_modules/@vitest/utils": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.5.tgz", + "integrity": "sha512-76wdkrmfXfqGjueGgnb45ITPyUi1ycZ4IHgC2bhPDUfWHklY/q3MdLOAB+TF1e6xfl8NxNY0ZYaPCFNWSsw3Ug==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "4.1.5", + "convert-source-map": "^2.0.0", + "tinyrainbow": "^3.1.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, "node_modules/@xmldom/xmldom": { "version": "0.8.12", "resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.12.tgz", @@ -3439,6 +3935,16 @@ "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==", "license": "Python-2.0" }, + "node_modules/aria-query": { + "version": "5.3.0", + "resolved": "https://registry.npmjs.org/aria-query/-/aria-query-5.3.0.tgz", + "integrity": "sha512-b0P0sZPKtyu8HkeRAfCq0IfURZK+SuwMjY1UXGBU27wpAiTwQAIlq56IbIO+ytk/JjS1fMR14ee5WBBfKi5J6A==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "dequal": "^2.0.3" + } + }, "node_modules/assert-plus": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz", @@ -3450,6 +3956,16 @@ "node": ">=0.8" } }, + "node_modules/assertion-error": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", + "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + } + }, "node_modules/astral-regex": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/astral-regex/-/astral-regex-2.0.0.tgz", @@ -3569,6 +4085,16 @@ "node": ">=6.0.0" } }, + "node_modules/bidi-js": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/bidi-js/-/bidi-js-1.0.3.tgz", + "integrity": "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw==", + "dev": true, + "license": "MIT", + "dependencies": { + "require-from-string": "^2.0.2" + } + }, "node_modules/binary-extensions": { "version": "2.3.0", "resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz", @@ -4041,6 +4567,16 @@ "url": "https://github.com/sponsors/wooorm" } }, + "node_modules/chai": { + "version": "6.2.2", + "resolved": "https://registry.npmjs.org/chai/-/chai-6.2.2.tgz", + "integrity": "sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, "node_modules/chalk": { "version": "4.1.2", "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz", @@ -4602,12 +5138,47 @@ "node": ">= 8" } }, + "node_modules/css-tree": { + "version": "3.2.1", + "resolved": "https://registry.npmjs.org/css-tree/-/css-tree-3.2.1.tgz", + "integrity": "sha512-X7sjQzceUhu1u7Y/ylrRZFU2FS6LRiFVp6rKLPg23y3x3c3DOKAwuXGDp+PAGjh6CSnCjYeAul8pcT8bAl+lSA==", + "dev": true, + "license": "MIT", + "dependencies": { + "mdn-data": "2.27.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0" + } + }, + "node_modules/css.escape": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/css.escape/-/css.escape-1.5.1.tgz", + "integrity": "sha512-YUifsXXuknHlUsmlgyY0PKzgPOr7/FjCePfHNt0jxm83wHZi44VDMQ7/fGNkjY3/jV1MC+1CmZbaHzugyeRtpg==", + "dev": true, + "license": "MIT" + }, "node_modules/csstype": { "version": "3.2.3", "resolved": "https://registry.npmjs.org/csstype/-/csstype-3.2.3.tgz", "integrity": "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==", "license": "MIT" }, + "node_modules/data-urls": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-7.0.0.tgz", + "integrity": "sha512-23XHcCF+coGYevirZceTVD7NdJOqVn+49IHyxgszm+JIiHLoB2TkmPtsYkNWT1pvRSGkc35L6NHs0yHkN2SumA==", + "dev": true, + "license": "MIT", + "dependencies": { + "whatwg-mimetype": "^5.0.0", + "whatwg-url": "^16.0.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, "node_modules/debounce-fn": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/debounce-fn/-/debounce-fn-4.0.0.tgz", @@ -4640,6 +5211,13 @@ } } }, + "node_modules/decimal.js": { + "version": "10.6.0", + "resolved": "https://registry.npmjs.org/decimal.js/-/decimal.js-10.6.0.tgz", + "integrity": "sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg==", + "dev": true, + "license": "MIT" + }, "node_modules/decode-named-character-reference": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/decode-named-character-reference/-/decode-named-character-reference-1.3.0.tgz", @@ -4928,6 +5506,14 @@ "node": ">=8" } }, + "node_modules/dom-accessibility-api": { + "version": "0.5.16", + "resolved": "https://registry.npmjs.org/dom-accessibility-api/-/dom-accessibility-api-0.5.16.tgz", + "integrity": "sha512-X7BJ2yElsnOJ30pZF4uIIDfBEVgF4XEBxL9Bxhy6dnrm5hkzqmsWHGTiHqRiITNhMyFLyAiWndIJP7Z1NTteDg==", + "dev": true, + "license": "MIT", + "peer": true + }, "node_modules/dompurify": { "version": "3.2.7", "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.2.7.tgz", @@ -5414,6 +6000,13 @@ "node": ">= 0.4" } }, + "node_modules/es-module-lexer": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.1.0.tgz", + "integrity": "sha512-n27zTYMjYu1aj4MjCWzSP7G9r75utsaoc8m61weK+W8JMBGGQybd43GstCXZ3WNmSFtGT9wi59qQTW6mhTR5LQ==", + "dev": true, + "license": "MIT" + }, "node_modules/es-object-atoms": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz", @@ -5528,6 +6121,16 @@ "url": "https://opencollective.com/unified" } }, + "node_modules/estree-walker": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz", + "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0" + } + }, "node_modules/execa": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/execa/-/execa-1.0.0.tgz", @@ -5633,6 +6236,16 @@ "which": "bin/which" } }, + "node_modules/expect-type": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.3.0.tgz", + "integrity": "sha512-knvyeauYhqjOYvQ66MznSMs83wmHrCycNEN6Ao+2AeYEfxUIkuiVxdEa1qlGEPK+We3n0THiDciYSsCcgW/DoA==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=12.0.0" + } + }, "node_modules/exponential-backoff": { "version": "3.1.3", "resolved": "https://registry.npmjs.org/exponential-backoff/-/exponential-backoff-3.1.3.tgz", @@ -5806,6 +6419,13 @@ "node": ">=6" } }, + "node_modules/flatted": { + "version": "3.4.2", + "resolved": "https://registry.npmjs.org/flatted/-/flatted-3.4.2.tgz", + "integrity": "sha512-PjDse7RzhcPkIJwy5t7KPWQSZ9cAbzQXcafsetQoD7sOJRQlGikNbx7yZp2OotDnJyrDcbyRq3Ttb18iYOqkxA==", + "dev": true, + "license": "ISC" + }, "node_modules/fmix": { "version": "0.1.0", "resolved": "https://registry.npmjs.org/fmix/-/fmix-0.1.0.tgz", @@ -6504,6 +7124,19 @@ "node": ">=10" } }, + "node_modules/html-encoding-sniffer": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-6.0.0.tgz", + "integrity": "sha512-CV9TW3Y3f8/wT0BRFc1/KAVQ3TUHiXmaAb6VW9vtiMFf7SLoMd1PdAc4W3KFOFETBJUb90KatHqlsZMWV+R9Gg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@exodus/bytes": "^1.6.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, "node_modules/html-url-attributes": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/html-url-attributes/-/html-url-attributes-3.0.1.tgz", @@ -6896,6 +7529,13 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/is-potential-custom-element-name": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/is-potential-custom-element-name/-/is-potential-custom-element-name-1.0.1.tgz", + "integrity": "sha512-bCYeRA2rVibKZd+s2625gGnGF/t7DSqDs4dP7CrLA1m7jKWz6pps0LpYLJN8Q64HtmPKJ1hrN3nzPNKFEKOUiQ==", + "dev": true, + "license": "MIT" + }, "node_modules/is-property": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/is-property/-/is-property-1.0.2.tgz", @@ -7007,6 +7647,83 @@ "js-yaml": "bin/js-yaml.js" } }, + "node_modules/jsdom": { + "version": "29.1.0", + "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-29.1.0.tgz", + "integrity": "sha512-YNUc7fB9QuvSSQWfrH0xF+TyABkxUwx8sswgIDaCrw4Hol8BghdZDkITtZheRJeMtzWlnTfsM3bBBusRvpO1wg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@asamuzakjp/css-color": "^5.1.11", + "@asamuzakjp/dom-selector": "^7.1.1", + "@bramus/specificity": "^2.4.2", + "@csstools/css-syntax-patches-for-csstree": "^1.1.3", + "@exodus/bytes": "^1.15.0", + "css-tree": "^3.2.1", + "data-urls": "^7.0.0", + "decimal.js": "^10.6.0", + "html-encoding-sniffer": "^6.0.0", + "is-potential-custom-element-name": "^1.0.1", + "lru-cache": "^11.3.5", + "parse5": "^8.0.1", + "saxes": "^6.0.0", + "symbol-tree": "^3.2.4", + "tough-cookie": "^6.0.1", + "undici": "^7.25.0", + "w3c-xmlserializer": "^5.0.0", + "webidl-conversions": "^8.0.1", + "whatwg-mimetype": "^5.0.0", + "whatwg-url": "^16.0.1", + "xml-name-validator": "^5.0.0" + }, + "engines": { + "node": "^20.19.0 || ^22.13.0 || >=24.0.0" + }, + "peerDependencies": { + "canvas": "^3.0.0" + }, + "peerDependenciesMeta": { + "canvas": { + "optional": true + } + } + }, + "node_modules/jsdom/node_modules/entities": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/entities/-/entities-8.0.0.tgz", + "integrity": "sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=20.19.0" + }, + "funding": { + "url": "https://github.com/fb55/entities?sponsor=1" + } + }, + "node_modules/jsdom/node_modules/lru-cache": { + "version": "11.3.5", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.3.5.tgz", + "integrity": "sha512-NxVFwLAnrd9i7KUBxC4DrUhmgjzOs+1Qm50D3oF1/oL+r1NpZ4gA7xvG0/zJ8evR7zIKn4vLf7qTNduWFtCrRw==", + "dev": true, + "license": "BlueOak-1.0.0", + "engines": { + "node": "20 || >=22" + } + }, + "node_modules/jsdom/node_modules/parse5": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/parse5/-/parse5-8.0.1.tgz", + "integrity": "sha512-z1e/HMG90obSGeidlli3hj7cbocou0/wa5HacvI3ASx34PecNjNQeaHNo5WIZpWofN9kgkqV1q5YvXe3F0FoPw==", + "dev": true, + "license": "MIT", + "dependencies": { + "entities": "^8.0.0" + }, + "funding": { + "url": "https://github.com/inikulin/parse5?sponsor=1" + } + }, "node_modules/jsesc": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/jsesc/-/jsesc-3.1.0.tgz", @@ -7304,6 +8021,17 @@ "node": ">=10" } }, + "node_modules/lz-string": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/lz-string/-/lz-string-1.5.0.tgz", + "integrity": "sha512-h5bgJWpxJNswbU7qCrV0tIKQCaS3blPDrqKWx+QxzuzL1zGUzij9XCWLrSLsJPu5t+eWA/ycetzYAO5IOMcWAQ==", + "dev": true, + "license": "MIT", + "peer": true, + "bin": { + "lz-string": "bin/bin.js" + } + }, "node_modules/macos-alias": { "version": "0.2.12", "resolved": "https://registry.npmjs.org/macos-alias/-/macos-alias-0.2.12.tgz", @@ -7318,6 +8046,16 @@ "nan": "^2.4.0" } }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, "node_modules/make-fetch-happen": { "version": "14.0.3", "resolved": "https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-14.0.3.tgz", @@ -7717,6 +8455,13 @@ "url": "https://opencollective.com/unified" } }, + "node_modules/mdn-data": { + "version": "2.27.1", + "resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.27.1.tgz", + "integrity": "sha512-9Yubnt3e8A0OKwxYSXyhLymGW4sCufcLG6VdiDdUGVkPhpqLxlvP5vl1983gQjJl3tqbrM731mjaZaP68AgosQ==", + "dev": true, + "license": "CC0-1.0" + }, "node_modules/micromark": { "version": "4.0.2", "resolved": "https://registry.npmjs.org/micromark/-/micromark-4.0.2.tgz", @@ -8383,6 +9128,16 @@ "node": ">=4" } }, + "node_modules/min-indent": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/min-indent/-/min-indent-1.0.1.tgz", + "integrity": "sha512-I9jwMn07Sy/IwOj3zVkVik2JTvgpaykDZEigL6Rx6N9LbMywwUSMtxET+7lVoDLLd3O3IXwJwvuuns8UB/HeAg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=4" + } + }, "node_modules/minimatch": { "version": "10.2.5", "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz", @@ -8563,6 +9318,16 @@ "marked": "14.0.0" } }, + "node_modules/mrmime": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/mrmime/-/mrmime-2.0.1.tgz", + "integrity": "sha512-Y3wQdFg2Va6etvQ5I82yUhGdsKrcYox6p7FfL1LbK2J4V01F9TGlepTIhnK24t7koZibmg82KGglhA1XK5IsLQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=10" + } + }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", @@ -8917,6 +9682,17 @@ "node": ">= 0.4" } }, + "node_modules/obug": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/obug/-/obug-2.1.1.tgz", + "integrity": "sha512-uTqF9MuPraAQ+IsnPf366RG4cP9RtUi7MLO1N3KEc+wb0a6yKpeL0lmk2IB1jY5KHPAlTc6T/JRdC/YqxHNwkQ==", + "dev": true, + "funding": [ + "https://github.com/sponsors/sxzz", + "https://opencollective.com/debug" + ], + "license": "MIT" + }, "node_modules/once": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", @@ -9186,6 +9962,13 @@ "node": ">=16 || 14 >=14.17" } }, + "node_modules/pathe": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", + "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==", + "dev": true, + "license": "MIT" + }, "node_modules/pe-library": { "version": "0.4.1", "resolved": "https://registry.npmjs.org/pe-library/-/pe-library-0.4.1.tgz", @@ -9344,6 +10127,36 @@ "url": "https://opencollective.com/preact" } }, + "node_modules/pretty-format": { + "version": "27.5.1", + "resolved": "https://registry.npmjs.org/pretty-format/-/pretty-format-27.5.1.tgz", + "integrity": "sha512-Qb1gy5OrP5+zDf2Bvnzdl3jsTf1qXVMazbvCoKhtKqVs4/YK4ozX4gKQJJVyNe+cajNPn0KoC0MC3FUmaHWEmQ==", + "dev": true, + "license": "MIT", + "peer": true, + "dependencies": { + "ansi-regex": "^5.0.1", + "ansi-styles": "^5.0.0", + "react-is": "^17.0.1" + }, + "engines": { + "node": "^10.13.0 || ^12.13.0 || ^14.15.0 || >=15.0.0" + } + }, + "node_modules/pretty-format/node_modules/ansi-styles": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-5.2.0.tgz", + "integrity": "sha512-Cxwpt2SfTzTtXcfOlzGEee8O+c+MmUgGrNiBcXnuWxuFJHe6a5Hz7qwhwe5OgaSYI0IJvkLqWX1ASG+cJOkEiA==", + "dev": true, + "license": "MIT", + "peer": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/ansi-styles?sponsor=1" + } + }, "node_modules/proc-log": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/proc-log/-/proc-log-5.0.0.tgz", @@ -9517,6 +10330,14 @@ "react-dom": ">=16" } }, + "node_modules/react-is": { + "version": "17.0.2", + "resolved": "https://registry.npmjs.org/react-is/-/react-is-17.0.2.tgz", + "integrity": "sha512-w2GsyukL62IJnlaff/nRegPQR94C/XXamvMWmSHRJ4y7Ts/4ocGRmTHvOs8PSE6pB3dWOrD/nueuU5sduBsQ4w==", + "dev": true, + "license": "MIT", + "peer": true + }, "node_modules/react-markdown": { "version": "10.1.0", "resolved": "https://registry.npmjs.org/react-markdown/-/react-markdown-10.1.0.tgz", @@ -9715,6 +10536,20 @@ "url": "https://github.com/sponsors/jonschlinkert" } }, + "node_modules/redent": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/redent/-/redent-3.0.0.tgz", + "integrity": "sha512-6tDA8g98We0zd0GvVeMT9arEOnTw9qM03L9cJXaCjrip1OO764RDBLBfrB4cwzNGDj5OA5ioymC9GkizgWJDUg==", + "dev": true, + "license": "MIT", + "dependencies": { + "indent-string": "^4.0.0", + "strip-indent": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/redux": { "version": "5.0.1", "resolved": "https://registry.npmjs.org/redux/-/redux-5.0.1.tgz", @@ -10072,6 +10907,19 @@ "node": ">=11.0.0" } }, + "node_modules/saxes": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/saxes/-/saxes-6.0.0.tgz", + "integrity": "sha512-xAg7SOnEhrm5zI3puOOKyy1OMcMlIJZYNJY7xLBwSze0UjhPLnWfj2GF2EpT0jmzaJKIWKHLsaSSajf35bcYnA==", + "dev": true, + "license": "ISC", + "dependencies": { + "xmlchars": "^2.2.0" + }, + "engines": { + "node": ">=v12.22.7" + } + }, "node_modules/scheduler": { "version": "0.27.0", "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.27.0.tgz", @@ -10164,6 +11012,13 @@ "node": ">=8" } }, + "node_modules/siginfo": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz", + "integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==", + "dev": true, + "license": "ISC" + }, "node_modules/signal-exit": { "version": "3.0.7", "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.7.tgz", @@ -10197,6 +11052,21 @@ "node": ">=10" } }, + "node_modules/sirv": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/sirv/-/sirv-3.0.2.tgz", + "integrity": "sha512-2wcC/oGxHis/BoHkkPwldgiPSYcpZK3JU28WoMVv55yHJgcZ8rlXvuG9iZggz+sU1d4bRgIGASwyWqjxu3FM0g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@polka/url": "^1.0.0-next.24", + "mrmime": "^2.0.0", + "totalist": "^3.0.0" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/slice-ansi": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/slice-ansi/-/slice-ansi-3.0.0.tgz", @@ -10327,6 +11197,13 @@ "node": ">=16 || 14 >=14.17" } }, + "node_modules/stackback": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz", + "integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==", + "dev": true, + "license": "MIT" + }, "node_modules/stat-mode": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/stat-mode/-/stat-mode-1.0.0.tgz", @@ -10344,6 +11221,13 @@ "dev": true, "license": "MIT" }, + "node_modules/std-env": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/std-env/-/std-env-4.1.0.tgz", + "integrity": "sha512-Rq7ybcX2RuC55r9oaPVEW7/xu3tj8u4GeBYHBWCychFtzMIr86A7e3PPEBPT37sHStKX3+TiX/Fr/ACmJLVlLQ==", + "dev": true, + "license": "MIT" + }, "node_modules/stream-buffers": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/stream-buffers/-/stream-buffers-2.2.0.tgz", @@ -10447,6 +11331,19 @@ "node": ">=0.10.0" } }, + "node_modules/strip-indent": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/strip-indent/-/strip-indent-3.0.0.tgz", + "integrity": "sha512-laJTa3Jb+VQpaC6DseHhF7dXVqHTfJPCRDaEbid/drOhgitgYku/letMUqOXFoWV0zIIUbjpdH2t+tYj4bQMRQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "min-indent": "^1.0.0" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/style-to-js": { "version": "1.1.21", "resolved": "https://registry.npmjs.org/style-to-js/-/style-to-js-1.1.21.tgz", @@ -10493,6 +11390,13 @@ "node": ">=8" } }, + "node_modules/symbol-tree": { + "version": "3.2.4", + "resolved": "https://registry.npmjs.org/symbol-tree/-/symbol-tree-3.2.4.tgz", + "integrity": "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==", + "dev": true, + "license": "MIT" + }, "node_modules/tar": { "version": "6.2.1", "resolved": "https://registry.npmjs.org/tar/-/tar-6.2.1.tgz", @@ -10595,6 +11499,23 @@ "integrity": "sha512-qVtvMxeXbVej0cQWKqVSSAHmKZEHAvxdF8HEUBFWts8h+xEo5m/lEiPakuyZ3BnCBjOD8i24kzNOiOLLgsSxhA==", "license": "MIT" }, + "node_modules/tinybench": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", + "integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinyexec": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.1.2.tgz", + "integrity": "sha512-dAqSqE/RabpBKI8+h26GfLq6Vb3JVXs30XYQjdMjaj/c2tS8IYYMbIzP599KtRj7c57/wYApb3QjgRgXmrCukA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, "node_modules/tinyglobby": { "version": "0.2.16", "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz", @@ -10612,6 +11533,36 @@ "url": "https://github.com/sponsors/SuperchupuDev" } }, + "node_modules/tinyrainbow": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-3.1.0.tgz", + "integrity": "sha512-Bf+ILmBgretUrdJxzXM0SgXLZ3XfiaUuOj/IKQHuTXip+05Xn+uyEYdVg0kYDipTBcLrCVyUzAPz7QmArb0mmw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/tldts": { + "version": "7.0.29", + "resolved": "https://registry.npmjs.org/tldts/-/tldts-7.0.29.tgz", + "integrity": "sha512-JIXCerhudr/N6OWLwLF1HVsTTUo7ry6qHa5eWZEkiMuxsIiAACL55tGLfqfHfoH7QaMQUW8fngD7u7TxWexYQg==", + "dev": true, + "license": "MIT", + "dependencies": { + "tldts-core": "^7.0.29" + }, + "bin": { + "tldts": "bin/cli.js" + } + }, + "node_modules/tldts-core": { + "version": "7.0.29", + "resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-7.0.29.tgz", + "integrity": "sha512-W99NuU7b1DcG3uJ3v9k9VztCH3WialNbBkBft5wCs8V8mexu0XQqaZEYb9l9RNNzK8+3EJ9PKWB0/RUtTQ/o+Q==", + "dev": true, + "license": "MIT" + }, "node_modules/tmp": { "version": "0.2.5", "resolved": "https://registry.npmjs.org/tmp/-/tmp-0.2.5.tgz", @@ -10664,6 +11615,42 @@ "node": ">=8.0" } }, + "node_modules/totalist": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/totalist/-/totalist-3.0.1.tgz", + "integrity": "sha512-sf4i37nQ2LBx4m3wB74y+ubopq6W/dIzXg0FDGjsYnZHVa1Da8FH853wlL2gtUhg+xJXjfk3kUZS3BRoQeoQBQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/tough-cookie": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-6.0.1.tgz", + "integrity": "sha512-LktZQb3IeoUWB9lqR5EWTHgW/VTITCXg4D21M+lvybRVdylLrRMnqaIONLVb5mav8vM19m44HIcGq4qASeu2Qw==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "tldts": "^7.0.5" + }, + "engines": { + "node": ">=16" + } + }, + "node_modules/tr46": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/tr46/-/tr46-6.0.0.tgz", + "integrity": "sha512-bLVMLPtstlZ4iMQHpFHTR7GAGj2jxi8Dg0s2h2MafAE4uSWF98FC/3MomU51iQAMf8/qDUbKWf5GxuvvVcXEhw==", + "dev": true, + "license": "MIT", + "dependencies": { + "punycode": "^2.3.1" + }, + "engines": { + "node": ">=20" + } + }, "node_modules/trim-lines": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/trim-lines/-/trim-lines-3.0.1.tgz", @@ -10722,6 +11709,16 @@ "node": ">=14.17" } }, + "node_modules/undici": { + "version": "7.25.0", + "resolved": "https://registry.npmjs.org/undici/-/undici-7.25.0.tgz", + "integrity": "sha512-xXnp4kTyor2Zq+J1FfPI6Eq3ew5h6Vl0F/8d9XU5zZQf1tX9s2Su1/3PiMmUANFULpmksxkClamIZcaUqryHsQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20.18.1" + } + }, "node_modules/undici-types": { "version": "6.21.0", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", @@ -11582,6 +12579,109 @@ "@esbuild/win32-x64": "0.27.7" } }, + "node_modules/vitest": { + "version": "4.1.5", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.5.tgz", + "integrity": "sha512-9Xx1v3/ih3m9hN+SbfkUyy0JAs72ap3r7joc87XL6jwF0jGg6mFBvQ1SrwaX+h8BlkX6Hz9shdd1uo6AF+ZGpg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/expect": "4.1.5", + "@vitest/mocker": "4.1.5", + "@vitest/pretty-format": "4.1.5", + "@vitest/runner": "4.1.5", + "@vitest/snapshot": "4.1.5", + "@vitest/spy": "4.1.5", + "@vitest/utils": "4.1.5", + "es-module-lexer": "^2.0.0", + "expect-type": "^1.3.0", + "magic-string": "^0.30.21", + "obug": "^2.1.1", + "pathe": "^2.0.3", + "picomatch": "^4.0.3", + "std-env": "^4.0.0-rc.1", + "tinybench": "^2.9.0", + "tinyexec": "^1.0.2", + "tinyglobby": "^0.2.15", + "tinyrainbow": "^3.1.0", + "vite": "^6.0.0 || ^7.0.0 || ^8.0.0", + "why-is-node-running": "^2.3.0" + }, + "bin": { + "vitest": "vitest.mjs" + }, + "engines": { + "node": "^20.0.0 || ^22.0.0 || >=24.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "@edge-runtime/vm": "*", + "@opentelemetry/api": "^1.9.0", + "@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0", + "@vitest/browser-playwright": "4.1.5", + "@vitest/browser-preview": "4.1.5", + "@vitest/browser-webdriverio": "4.1.5", + "@vitest/coverage-istanbul": "4.1.5", + "@vitest/coverage-v8": "4.1.5", + "@vitest/ui": "4.1.5", + "happy-dom": "*", + "jsdom": "*", + "vite": "^6.0.0 || ^7.0.0 || ^8.0.0" + }, + "peerDependenciesMeta": { + "@edge-runtime/vm": { + "optional": true + }, + "@opentelemetry/api": { + "optional": true + }, + "@types/node": { + "optional": true + }, + "@vitest/browser-playwright": { + "optional": true + }, + "@vitest/browser-preview": { + "optional": true + }, + "@vitest/browser-webdriverio": { + "optional": true + }, + "@vitest/coverage-istanbul": { + "optional": true + }, + "@vitest/coverage-v8": { + "optional": true + }, + "@vitest/ui": { + "optional": true + }, + "happy-dom": { + "optional": true + }, + "jsdom": { + "optional": true + }, + "vite": { + "optional": false + } + } + }, + "node_modules/w3c-xmlserializer": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-5.0.0.tgz", + "integrity": "sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA==", + "dev": true, + "license": "MIT", + "dependencies": { + "xml-name-validator": "^5.0.0" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/wcwidth": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/wcwidth/-/wcwidth-1.0.1.tgz", @@ -11609,6 +12709,41 @@ "integrity": "sha512-i2z98bEmaCqSDiHEDu+gHl/dmR4Q+TxFmG3/13KkMO+o8UxQzCqWaDRCiLgEa41nlO4VpXSI0ASa1xWmO9sBlA==", "license": "Apache-2.0" }, + "node_modules/webidl-conversions": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-8.0.1.tgz", + "integrity": "sha512-BMhLD/Sw+GbJC21C/UgyaZX41nPt8bUTg+jWyDeg7e7YN4xOM05YPSIXceACnXVtqyEw/LMClUQMtMZ+PGGpqQ==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=20" + } + }, + "node_modules/whatwg-mimetype": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-5.0.0.tgz", + "integrity": "sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20" + } + }, + "node_modules/whatwg-url": { + "version": "16.0.1", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-16.0.1.tgz", + "integrity": "sha512-1to4zXBxmXHV3IiSSEInrreIlu02vUOvrhxJJH5vcxYTBDAx51cqZiKdyTxlecdKNSjj8EcxGBxNf6Vg+945gw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@exodus/bytes": "^1.11.0", + "tr46": "^6.0.0", + "webidl-conversions": "^8.0.1" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, "node_modules/which": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", @@ -11625,6 +12760,23 @@ "node": ">= 8" } }, + "node_modules/why-is-node-running": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz", + "integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==", + "dev": true, + "license": "MIT", + "dependencies": { + "siginfo": "^2.0.0", + "stackback": "0.0.2" + }, + "bin": { + "why-is-node-running": "cli.js" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/wide-align": { "version": "1.1.5", "resolved": "https://registry.npmjs.org/wide-align/-/wide-align-1.1.5.tgz", @@ -11679,6 +12831,16 @@ "devOptional": true, "license": "ISC" }, + "node_modules/xml-name-validator": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/xml-name-validator/-/xml-name-validator-5.0.0.tgz", + "integrity": "sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=18" + } + }, "node_modules/xmlbuilder": { "version": "15.1.1", "resolved": "https://registry.npmjs.org/xmlbuilder/-/xmlbuilder-15.1.1.tgz", @@ -11689,6 +12851,13 @@ "node": ">=8.0" } }, + "node_modules/xmlchars": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/xmlchars/-/xmlchars-2.2.0.tgz", + "integrity": "sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw==", + "dev": true, + "license": "MIT" + }, "node_modules/xtend": { "version": "4.0.2", "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz", diff --git a/desktop/package.json b/desktop/package.json index e04438074a24..3b4e2fa39e3d 100644 --- a/desktop/package.json +++ b/desktop/package.json @@ -1,17 +1,19 @@ { "name": "hermes-desktop", - "version": "0.1.31", + "version": "0.1.35", "description": "Atomic Hermes — self-contained macOS desktop app", "main": "dist/main/main.js", "private": true, "scripts": { "rebuild": "electron-rebuild -f -w node-pty", + "patch:electron": "node scripts/patch-electron-bundle-id.cjs", + "postinstall": "node scripts/patch-electron-bundle-id.cjs", "build:ts": "tsc && node scripts/define-main-env.mjs", "build:dashboard": "npm --prefix ../web install && npm --prefix ../web run build", "build:renderer": "node --max-old-space-size=4096 node_modules/vite/bin/vite.js build --config renderer/vite.config.ts", "build:all": "npm run build:dashboard && npm run build:ts && npm run build:renderer", - "start": "npm run rebuild && npm run build:all && electron .", - "dev": "npm run rebuild && bash scripts/bundle-dev.sh && npm run build:dashboard && npm run build:renderer && tsc && (tsc -w --preserveWatchOutput & electron .)", + "start": "npm run patch:electron && npm run rebuild && npm run build:all && electron .", + "dev": "npm run patch:electron && npm run rebuild && bash scripts/bundle-dev.sh && npm run build:dashboard && npm run build:renderer && tsc && (tsc -w --preserveWatchOutput & electron .)", "bundle": "bash scripts/bundle-all.sh", "bundle:dev": "bash scripts/bundle-dev.sh", "dist": "npm run build:all && electron-builder", @@ -19,7 +21,9 @@ "release": "bash scripts/release.sh", "release:patch": "bash scripts/release.sh patch", "release:minor": "bash scripts/release.sh minor", - "release:major": "bash scripts/release.sh major" + "release:major": "bash scripts/release.sh major", + "test": "vitest run", + "test:watch": "vitest" }, "dependencies": { "chokidar": "^3.6.0", @@ -35,18 +39,22 @@ "@electron/rebuild": "^4.0.3", "@monaco-editor/react": "^4.7.0", "@reduxjs/toolkit": "^2.9.0", + "@testing-library/jest-dom": "^6.9.1", + "@testing-library/react": "^16.3.2", "@types/js-yaml": "^4.0.9", "@types/node": "^22.0.0", "@types/react": "^19.2.10", "@types/react-dom": "^19.2.3", "@types/remove-markdown": "^0.3.4", "@vitejs/plugin-react": "^5.1.3", + "@vitest/ui": "^4.1.5", "@xterm/addon-fit": "^0.11.0", "@xterm/xterm": "^6.0.0", "cross-env": "^10.1.0", "electron": "^33.0.0", "electron-builder": "^25.0.0", "esbuild": "^0.28.0", + "jsdom": "^29.1.0", "monaco-editor": "^0.55.1", "react": "^19.2.4", "react-dom": "^19.2.4", @@ -58,7 +66,8 @@ "remark-math": "^6.0.0", "remove-markdown": "^0.6.3", "typescript": "^5.6.0", - "vite": "^7.3.1" + "vite": "^7.3.1", + "vitest": "^4.1.5" }, "optionalDependencies": { "appdmg": "^0.6.6" @@ -130,6 +139,14 @@ "repo": "atomic-hermes", "releaseType": "draft" }, + "protocols": [ + { + "name": "Atomic Hermes Protocol", + "schemes": [ + "atomicbot-hermes" + ] + } + ], "mac": { "artifactName": "Atomic-Hermes-${version}-${arch}-mac.${ext}", "icon": "assets/icon.icns", diff --git a/desktop/renderer/index.html b/desktop/renderer/index.html index 82f57a624f41..df5cb5fca77f 100644 --- a/desktop/renderer/index.html +++ b/desktop/renderer/index.html @@ -5,7 +5,7 @@ Atomic Hermes diff --git a/desktop/renderer/src/hooks/useAtomicDeepLink.ts b/desktop/renderer/src/hooks/useAtomicDeepLink.ts new file mode 100644 index 000000000000..cb62aab69408 --- /dev/null +++ b/desktop/renderer/src/hooks/useAtomicDeepLink.ts @@ -0,0 +1,73 @@ +import React from "react"; +import { getDesktopApiOrNull } from "@ipc/desktopApi"; + +export type AtomicDeepLinkAuthParams = { + jwt: string; + email: string; + userId: string; + isNewUser: boolean; +}; + +export type AtomicStripeSuccessParams = { + sessionId: string | null; +}; + +/** + * Subscribes to atomicbot-hermes:// deep links delivered by the main process. + * + * The backend emits three URL shapes (see atomic-bot-backend/docs/DESKTOP_INTEGRATION.md); + * we override the default `atomicbot://` scheme to `atomicbot-hermes://` via + * `?scheme=atomicbot-hermes` on the OAuth/topup initiator URLs: + * - atomicbot-hermes://auth?token=&email=&userId=&isNewUser= + * - atomicbot-hermes://stripe-success?session_id=cs_... + * - atomicbot-hermes://stripe-cancel + */ +export function useAtomicDeepLink(handlers: { + onAuth?: (params: AtomicDeepLinkAuthParams) => void; + onAuthError?: () => void; + onStripeSuccess?: (params: AtomicStripeSuccessParams) => void; + onStripeCancel?: () => void; +}): void { + const handlersRef = React.useRef(handlers); + handlersRef.current = handlers; + + React.useEffect(() => { + const api = getDesktopApiOrNull(); + if (!api?.onAtomicDeepLink) return; + + const unsub = api.onAtomicDeepLink((payload) => { + // macOS hands us the URL with `host = "auth"` and `pathname = ""`, + // while some Linux desktop environments fire it as `host = ""` and + // `pathname = "/auth"`. Accept both shapes. + const hostOrPath = payload.host || payload.pathname.replace(/^\//, ""); + + if (hostOrPath === "auth") { + const { token, email, userId, isNewUser } = payload.params; + if (token && userId) { + handlersRef.current.onAuth?.({ + jwt: token, + email: email ? decodeURIComponent(email) : "", + userId, + isNewUser: isNewUser === "true", + }); + } else { + handlersRef.current.onAuthError?.(); + } + return; + } + + if (hostOrPath === "stripe-success") { + handlersRef.current.onStripeSuccess?.({ + sessionId: payload.params.session_id ?? null, + }); + return; + } + + if (hostOrPath === "stripe-cancel") { + handlersRef.current.onStripeCancel?.(); + } + }); + + return unsub; + }, []); +} diff --git a/desktop/renderer/src/hooks/useAtomicTopupPolling.ts b/desktop/renderer/src/hooks/useAtomicTopupPolling.ts new file mode 100644 index 000000000000..9d09ea4b9fc8 --- /dev/null +++ b/desktop/renderer/src/hooks/useAtomicTopupPolling.ts @@ -0,0 +1,122 @@ +import React from "react"; +import toast from "react-hot-toast"; +import { useAppDispatch, useAppSelector } from "@store/hooks"; +import { + atomicAuthActions, + fetchAtomicBalance, +} from "@store/slices/atomicAuthSlice"; + +/** + * After a Stripe Checkout returns, the desktop app cannot rely on a + * `atomicbot-hermes://stripe-success` deep link to fire — Stripe rewrites + * unknown URI schemes (DNS error in the user's browser). Instead we poll + * the balance endpoint until the new credits land, mirroring the openclaw + * `usePaidStatusBridge` pattern. + * + * Total budget ≈ 2 minutes: + * - 30 polls × 2s = first minute + * - 12 polls × 5s = second minute + * If the balance increases versus the pre-topup snapshot, we stop early + * and notify the user. Otherwise the periodic background refresh (e.g. + * window focus) eventually picks the new balance up. + */ +const TOPUP_POLL_DELAYS_MS = [ + ...Array(30).fill(2000), + ...Array(12).fill(5000), +]; + +export function useAtomicTopupPolling(): void { + const dispatch = useAppDispatch(); + const jwt = useAppSelector((state) => state.atomicAuth.jwt); + const topupPending = useAppSelector((state) => state.atomicAuth.topupPending); + const balance = useAppSelector((state) => state.atomicAuth.balance); + + const balanceRef = React.useRef(balance); + React.useEffect(() => { + balanceRef.current = balance; + }, [balance]); + + // ── Polling: triggered by `topupPending` going truthy ── + React.useEffect(() => { + if (!topupPending || !jwt) return; + + let cancelled = false; + let pollTimer: ReturnType | null = null; + + const previousRemaining = balanceRef.current?.payg?.remaining ?? null; + + void (async () => { + for (const delay of TOPUP_POLL_DELAYS_MS) { + if (cancelled) return; + await new Promise((resolve) => { + pollTimer = setTimeout(resolve, delay); + }); + if (cancelled) return; + + // External refresh (focus/visibility) may have already updated the + // balance — short-circuit in that case. + const current = balanceRef.current?.payg?.remaining ?? null; + if ( + previousRemaining !== null && + current !== null && + current > previousRemaining + ) { + toast.success("Balance updated"); + dispatch(atomicAuthActions.setTopupPending(false)); + return; + } + + try { + const result = await dispatch( + fetchAtomicBalance({ sync: true }), + ).unwrap(); + const newRemaining = result.payg?.remaining ?? null; + if ( + newRemaining !== null && + (previousRemaining === null || newRemaining > previousRemaining) + ) { + toast.success("Balance updated"); + dispatch(atomicAuthActions.setTopupPending(false)); + return; + } + } catch { + // Transient — keep retrying. + } + } + // Stripe webhook is sometimes slower than our polling budget. The + // balance will land on the next manual / focus-triggered refresh. + toast("Balance is being updated…"); + dispatch(atomicAuthActions.setTopupPending(false)); + })(); + + return () => { + cancelled = true; + if (pollTimer) clearTimeout(pollTimer); + }; + }, [topupPending, jwt, dispatch]); + + // ── Window focus / visibility refresh ── + // Fires whenever the user comes back to the app, e.g. after switching to + // the browser to complete Stripe Checkout. Independent of `topupPending` + // so a manual return (without our explicit flow) still resyncs. + React.useEffect(() => { + if (!jwt) return; + + const refresh = () => { + void dispatch(fetchAtomicBalance({ sync: true })); + }; + + const onFocus = () => refresh(); + const onVisibilityChange = () => { + if (document.visibilityState === "visible") refresh(); + }; + + window.addEventListener("focus", onFocus); + document.addEventListener("visibilitychange", onVisibilityChange); + + return () => { + window.removeEventListener("focus", onFocus); + document.removeEventListener("visibilitychange", onVisibilityChange); + }; + }, [jwt, dispatch]); +} diff --git a/desktop/renderer/src/ipc/desktopApi.ts b/desktop/renderer/src/ipc/desktopApi.ts index 01723c58529e..cc0bf881d20c 100644 --- a/desktop/renderer/src/ipc/desktopApi.ts +++ b/desktop/renderer/src/ipc/desktopApi.ts @@ -79,6 +79,12 @@ export type UpdateErrorPayload = { message: string; }; +export type AtomicDeepLinkPayload = { + host: string; + pathname: string; + params: Record; +}; + // eslint-disable-next-line @typescript-eslint/no-empty-interface export interface DesktopApi { /** Node/Electron `process.platform` when running inside Electron preload. */ @@ -141,6 +147,11 @@ export interface DesktopApi { // Profile seeding seedProfileProvider?(source: string, target: string): Promise<{ ok: boolean; error?: string }>; + + // Atomic auth (PAYG / atomic-bot-backend) — JWT is persisted in + // window.localStorage on the renderer side; only the deep-link channel + // crosses the IPC boundary. + onAtomicDeepLink?(cb: (payload: AtomicDeepLinkPayload) => void): () => void; } export const DESKTOP_API_UNAVAILABLE = "Desktop API not available"; diff --git a/desktop/renderer/src/services/atomic-auth-storage.ts b/desktop/renderer/src/services/atomic-auth-storage.ts new file mode 100644 index 000000000000..35213276a8d3 --- /dev/null +++ b/desktop/renderer/src/services/atomic-auth-storage.ts @@ -0,0 +1,83 @@ +/** + * Renderer-side persistence for the Atomic Pay-as-you-go auth state. + * + * Stored in `window.localStorage` (Electron LevelDB profile, not the OS + * keystore). This is a deliberate downgrade from `safeStorage` to avoid the + * keystore unlock prompt — the JWT can always be re-issued via the OAuth + * deep-link flow if the storage is wiped. + */ + +const STORAGE_KEY = "atomic-auth"; + +export type AtomicAuthState = { + jwt: string; + email: string; + userId: string; +}; + +function getStorage(): Storage | null { + try { + return typeof window !== "undefined" ? window.localStorage : null; + } catch { + return null; + } +} + +function isAtomicAuthState(value: unknown): value is AtomicAuthState { + if (!value || typeof value !== "object") return false; + const obj = value as Record; + return ( + typeof obj.jwt === "string" && + obj.jwt.length > 0 && + typeof obj.userId === "string" && + obj.userId.length > 0 && + (typeof obj.email === "string" || obj.email === undefined) + ); +} + +export function readAtomicAuth(): AtomicAuthState | null { + const storage = getStorage(); + if (!storage) return null; + + let raw: string | null; + try { + raw = storage.getItem(STORAGE_KEY); + } catch (err) { + console.warn("[atomic-auth-storage] read failed:", err); + return null; + } + if (!raw) return null; + + try { + const parsed = JSON.parse(raw) as unknown; + if (!isAtomicAuthState(parsed)) return null; + return { + jwt: parsed.jwt, + email: parsed.email ?? "", + userId: parsed.userId, + }; + } catch (err) { + console.warn("[atomic-auth-storage] parse failed:", err); + return null; + } +} + +export function writeAtomicAuth(state: AtomicAuthState): void { + const storage = getStorage(); + if (!storage) return; + try { + storage.setItem(STORAGE_KEY, JSON.stringify(state)); + } catch (err) { + console.warn("[atomic-auth-storage] write failed:", err); + } +} + +export function clearAtomicAuth(): void { + const storage = getStorage(); + if (!storage) return; + try { + storage.removeItem(STORAGE_KEY); + } catch (err) { + console.warn("[atomic-auth-storage] clear failed:", err); + } +} diff --git a/desktop/renderer/src/services/atomic-backend-api.ts b/desktop/renderer/src/services/atomic-backend-api.ts new file mode 100644 index 000000000000..2fcb62ec0328 --- /dev/null +++ b/desktop/renderer/src/services/atomic-backend-api.ts @@ -0,0 +1,264 @@ +/** + * HTTP client for atomic-bot-backend (https://api.atomicbot.ai by default). + * Used by the PAYG onboarding flow to fetch the user's OpenRouter key, + * top up credits, read balance, and open the Stripe billing portal. + * + * See atomic-bot-backend/docs/DESKTOP_INTEGRATION.md for the full contract. + */ + +const DEFAULT_BACKEND_URL = "https://api.atomicbot.ai"; + +export function getAtomicBackendUrl(): string { + const fromEnv = import.meta.env.VITE_ATOMIC_BACKEND_URL; + if (typeof fromEnv === "string" && fromEnv.trim()) { + return fromEnv.trim().replace(/\/+$/, ""); + } + return DEFAULT_BACKEND_URL; +} + +export type SubscriptionPlan = "free" | "base" | "pro" | "max"; + +export type AuthMeResponse = { + userId: string; + email: string; + subscriptionPlan: SubscriptionPlan; +}; + +export type SubscriptionPool = { + limit: number; + remaining: number; + expiresAt: string | null; +}; + +export type PaygPool = { + limit: number; + remaining: number; +}; + +export type BalanceResponse = { + total: number; + subscriptionPlan: SubscriptionPlan; + subscription: SubscriptionPool | null; + payg: PaygPool | null; +}; + +export type PaygKeyResponse = { + key: string; + keyHash: string; + remaining: number; + limit: number; +}; + +export type PaygTopupRequest = { + amountUsd: number; + successUrl?: string; + cancelUrl?: string; +}; + +export type PaygTopupResponse = { + checkoutUrl: string; +}; + +export type PortalUrlResponse = { + portalUrl: string; +}; + +export type PurchaseHistoryItem = { + id: string; + type: "subscription" | "addon" | "payg_topup"; + amountUsd: number; + creditsAdded: number; + createdAt: string; +}; + +export type PurchaseHistoryResponse = { + purchases: PurchaseHistoryItem[]; +}; + +async function request( + path: string, + init: RequestInit & { jwt?: string | null } = {}, +): Promise { + const { jwt, ...rest } = init; + const headers = new Headers(rest.headers); + if (!headers.has("Content-Type") && rest.body !== undefined) { + headers.set("Content-Type", "application/json"); + } + if (jwt) { + headers.set("Authorization", `Bearer ${jwt}`); + } + + const url = `${getAtomicBackendUrl()}${path}`; + const res = await fetch(url, { ...rest, headers }); + + if (!res.ok) { + const body = await res.text().catch(() => ""); + let message = body; + try { + const parsed = JSON.parse(body) as { error?: string }; + if (parsed?.error) message = parsed.error; + } catch { + // body was not JSON — keep raw text + } + const error = new Error(message || `HTTP ${res.status}`); + (error as Error & { status?: number }).status = res.status; + throw error; + } + + return (await res.json()) as T; +} + +/** + * URL scheme this client claims for deep-link delivery. The backend reads + * this via `?scheme=` on `/auth/{google,apple}/desktop` and propagates it + * through OAuth `state`. Without it, the post-OAuth redirect falls back to + * the default `atomicbot://` scheme (used by openclaw) instead of our + * `atomicbot-hermes://`. The scheme must exactly match an entry in the + * backend's `ALLOWED_DESKTOP_SCHEMES` env var (see deploy-stage.yml / + * deploy-prod.yml in atomic-bot-backend), otherwise it is dropped + * server-side. + */ +export const DEEP_LINK_SCHEME = "atomicbot-hermes"; + +function withRedirectScheme(url: string): string { + const sep = url.includes("?") ? "&" : "?"; + return `${url}${sep}scheme=${encodeURIComponent(DEEP_LINK_SCHEME)}`; +} + +/** Build the URL the desktop opens in the system browser to start Google OAuth. */ +export function googleAuthDesktopUrl(): string { + return withRedirectScheme(`${getAtomicBackendUrl()}/auth/google/desktop`); +} + +/** Build the URL the desktop opens in the system browser to start Apple OAuth. */ +export function appleAuthDesktopUrl(): string { + return withRedirectScheme(`${getAtomicBackendUrl()}/auth/apple/desktop`); +} + +/** + * Localhost port (mirrors `STRIPE_THANKS_PORT` in + * `desktop/src/main/atomic-auth/stripe-thanks-server.ts`). Kept in sync by + * convention — both ends must agree on the same fixed port so a checkout + * URL minted in one process lifetime survives an app restart. + */ +const STRIPE_THANKS_LOCAL_PORT = 27871; + +/** + * Stripe Checkout `success_url` for PAYG top-ups. + * + * Points at the local thanks server hosted by the Electron main process. + * The page renders a "Payment successful" landing and triggers + * `atomicbot-hermes://stripe-success?session_id=…` via JS, which brings + * the Hermes window forward and triggers an immediate balance refresh. + * + * We deliberately avoid passing the custom URI scheme directly to Stripe: + * Stripe silently rewrites unknown schemes to `https://`, producing + * `https://atomicbot-hermes//stripe-success?…` (DNS error). Routing + * through `http://localhost` is the same trick the backend's Google OAuth + * flow uses (see atomic-bot-backend `auth.ts`, "Authentication Successful" + * HTML page) — Stripe accepts it, browser hits our local server, JS fires + * the deep link. + * + * `useAtomicTopupPolling` still polls the balance independently, so the + * flow is robust even if the deep link is blocked or the user dismisses + * the browser prompt. + */ +export function getStripePaygSuccessUrl(): string { + return `http://localhost:${STRIPE_THANKS_LOCAL_PORT}/stripe-thanks?session_id={CHECKOUT_SESSION_ID}`; +} + +/** + * Stripe Checkout `cancel_url` for PAYG top-ups. Must be a valid HTTP(S) + * URL — Stripe rejects custom URI schemes here too. + */ +export const STRIPE_PAYG_CANCEL_URL = "https://atomicbot.ai"; + +/** + * After Stripe Checkout, App.tsx consumes this and navigates (e.g. onboarding + * model step). TTL prevents a stale entry from navigating after an abandoned + * checkout session. + */ +const POST_PAYG_SUCCESS_NAV_SESSION_KEY = "hermes:postPaygSuccessNavigate"; +const POST_PAYG_SUCCESS_NAV_MAX_AGE_MS = 30 * 60 * 1000; + +export function rememberPostPaygSuccessNavigate(path: string): void { + try { + sessionStorage.setItem( + POST_PAYG_SUCCESS_NAV_SESSION_KEY, + JSON.stringify({ path, ts: Date.now() }), + ); + } catch { + // private mode / quota + } +} + +export function consumePostPaygSuccessNavigate(): string | null { + try { + const raw = sessionStorage.getItem(POST_PAYG_SUCCESS_NAV_SESSION_KEY); + sessionStorage.removeItem(POST_PAYG_SUCCESS_NAV_SESSION_KEY); + if (!raw) return null; + const parsed = JSON.parse(raw) as { path?: string; ts?: number }; + if (!parsed.path || typeof parsed.ts !== "number") return null; + if (Date.now() - parsed.ts > POST_PAYG_SUCCESS_NAV_MAX_AGE_MS) return null; + return parsed.path; + } catch { + return null; + } +} + +export function clearPostPaygSuccessNavigate(): void { + try { + sessionStorage.removeItem(POST_PAYG_SUCCESS_NAV_SESSION_KEY); + } catch { + // ignore + } +} + +export const atomicBackendApi = { + getMe(jwt: string): Promise { + return request("/auth/me", { jwt }); + }, + + getBalance(jwt: string, opts: { sync?: boolean } = {}): Promise { + const qs = opts.sync ? "?sync=true" : ""; + return request(`/billing/balance${qs}`, { jwt }); + }, + + /** + * Returns the decrypted PAYG OpenRouter API key. Idempotent — backend + * creates the key on demand if missing. TREAT THE RESULT AS A SECRET. + */ + getPaygKey(jwt: string): Promise { + return request("/billing/payg/key", { jwt }); + }, + + createPaygTopup(jwt: string, body: PaygTopupRequest): Promise { + return request("/billing/payg/topup", { + jwt, + method: "POST", + body: JSON.stringify(body), + }); + }, + + /** + * Stripe Customer Portal URL. Pass `mode: "payg"` for users without an + * active subscription (uses Customer ID from the latest PAYG payment). + */ + getPortalUrl(jwt: string, opts: { mode?: "payg" } = {}): Promise { + const qs = opts.mode === "payg" ? "?mode=payg" : ""; + return request(`/billing/portal${qs}`, { jwt }); + }, + + getHistory(jwt: string, limit = 20): Promise { + return request( + `/billing/history?limit=${encodeURIComponent(String(limit))}`, + { jwt }, + ); + }, +}; + +export type AtomicBackendError = Error & { status?: number }; + +export function isUnauthorizedError(err: unknown): boolean { + return Boolean(err && typeof err === "object" && (err as AtomicBackendError).status === 401); +} diff --git a/desktop/renderer/src/store/slices/atomicAuthSlice.ts b/desktop/renderer/src/store/slices/atomicAuthSlice.ts new file mode 100644 index 000000000000..487acbee2c2c --- /dev/null +++ b/desktop/renderer/src/store/slices/atomicAuthSlice.ts @@ -0,0 +1,275 @@ +import { createAsyncThunk, createSlice, type PayloadAction } from "@reduxjs/toolkit"; +import { + atomicBackendApi, + isUnauthorizedError, + type BalanceResponse, + type SubscriptionPlan, +} from "../../services/atomic-backend-api"; +import { + clearAtomicAuth as clearAtomicAuthStorage, + readAtomicAuth as readAtomicAuthStorage, + writeAtomicAuth as writeAtomicAuthStorage, +} from "../../services/atomic-auth-storage"; +import { patchConfig } from "../../services/api"; +import type { AppDispatch, RootState } from "../store"; + +export type AtomicAuthSliceState = { + jwt: string | null; + email: string | null; + userId: string | null; + subscriptionPlan: SubscriptionPlan | null; + balance: BalanceResponse | null; + + /** True while restoreAtomicAuth is reading JWT from local storage. */ + restoreLoading: boolean; + restoreLoaded: boolean; + + /** True while applyPaygKey thunk is running. */ + applyKeyBusy: boolean; + applyKeyError: string | null; + + /** True while fetchAtomicBalance is running. */ + balanceLoading: boolean; + balanceError: string | null; + + /** True while createPaygTopup is in flight. */ + topupBusy: boolean; + topupError: string | null; + /** True while waiting for the user to complete Stripe Checkout. */ + topupPending: boolean; +}; + +const initialState: AtomicAuthSliceState = { + jwt: null, + email: null, + userId: null, + subscriptionPlan: null, + balance: null, + + restoreLoading: false, + restoreLoaded: false, + + applyKeyBusy: false, + applyKeyError: null, + + balanceLoading: false, + balanceError: null, + + topupBusy: false, + topupError: null, + topupPending: false, +}; + +export type StoredTokenPayload = { + jwt: string; + email: string; + userId: string; + isNewUser?: boolean; +}; + +/** + * On app startup: read JWT from `window.localStorage`. Does not call the + * backend — that happens lazily in `verifyAtomicAuth` once the user enters + * a screen that needs a fresh balance / plan. + */ +export const restoreAtomicAuth = createAsyncThunk< + { jwt: string; email: string; userId: string } | null, + void +>("atomicAuth/restore", async () => { + const stored = readAtomicAuthStorage(); + if (!stored?.jwt || !stored?.userId) return null; + return { jwt: stored.jwt, email: stored.email ?? "", userId: stored.userId }; +}); + +/** + * Persist the JWT received from the OAuth deep link (atomicbot-hermes://auth?token=...). + */ +export const storeAtomicToken = createAsyncThunk< + StoredTokenPayload, + StoredTokenPayload +>("atomicAuth/store", async (payload) => { + writeAtomicAuthStorage({ + jwt: payload.jwt, + email: payload.email, + userId: payload.userId, + }); + return payload; +}); + +export const clearAtomicAuthThunk = createAsyncThunk( + "atomicAuth/clear", + async () => { + clearAtomicAuthStorage(); + }, +); + +/** + * Verify the cached JWT against /auth/me. On 401 the slice automatically + * clears local state — caller should redirect to sign-in. + */ +export const verifyAtomicAuth = createAsyncThunk< + { plan: SubscriptionPlan } | null, + void, + { state: RootState; dispatch: AppDispatch } +>("atomicAuth/verify", async (_arg, thunkApi) => { + const jwt = thunkApi.getState().atomicAuth.jwt; + if (!jwt) return null; + try { + const me = await atomicBackendApi.getMe(jwt); + return { plan: me.subscriptionPlan }; + } catch (err) { + if (isUnauthorizedError(err)) { + await thunkApi.dispatch(clearAtomicAuthThunk()).unwrap(); + return thunkApi.rejectWithValue("unauthorized") as never; + } + throw err; + } +}); + +/** + * Fetch /billing/payg/key, push the resulting OpenRouter key into the local + * Hermes gateway via PATCH /api/config (env: OPENROUTER_API_KEY + + * config: provider=openrouter). On the next chat turn the gateway picks up + * the freshly-written key from its env file. + */ +export const applyPaygKey = createAsyncThunk< + { remaining: number; limit: number }, + { port: number }, + { state: RootState; rejectValue: string } +>("atomicAuth/applyPaygKey", async ({ port }, thunkApi) => { + const jwt = thunkApi.getState().atomicAuth.jwt; + if (!jwt) { + return thunkApi.rejectWithValue("Not authenticated"); + } + + const result = await atomicBackendApi.getPaygKey(jwt); + if (!result?.key) { + return thunkApi.rejectWithValue("PAYG key missing in backend response"); + } + + await patchConfig(port, { + config: { provider: "openrouter" }, + env: { OPENROUTER_API_KEY: result.key }, + }); + + return { remaining: result.remaining, limit: result.limit }; +}); + +export const fetchAtomicBalance = createAsyncThunk< + BalanceResponse, + { sync?: boolean } | undefined, + { state: RootState; rejectValue: string } +>("atomicAuth/fetchBalance", async (arg, thunkApi) => { + const jwt = thunkApi.getState().atomicAuth.jwt; + if (!jwt) return thunkApi.rejectWithValue("Not authenticated"); + return atomicBackendApi.getBalance(jwt, { sync: arg?.sync }); +}); + +const atomicAuthSlice = createSlice({ + name: "atomicAuth", + initialState, + reducers: { + clearSliceAuthForModeSwitch(state) { + state.jwt = null; + state.email = null; + state.userId = null; + state.subscriptionPlan = null; + state.balance = null; + state.applyKeyBusy = false; + state.applyKeyError = null; + state.balanceLoading = false; + state.balanceError = null; + }, + setRestoredAuth(state, action: PayloadAction<{ jwt: string; email: string; userId: string }>) { + state.jwt = action.payload.jwt; + state.email = action.payload.email; + state.userId = action.payload.userId; + }, + setTopupPending(state, action: PayloadAction) { + state.topupPending = action.payload; + }, + setTopupError(state, action: PayloadAction) { + state.topupError = action.payload; + }, + setTopupBusy(state, action: PayloadAction) { + state.topupBusy = action.payload; + }, + }, + extraReducers: (builder) => { + builder + .addCase(restoreAtomicAuth.pending, (state) => { + state.restoreLoading = true; + }) + .addCase(restoreAtomicAuth.fulfilled, (state, action) => { + state.restoreLoading = false; + state.restoreLoaded = true; + if (action.payload) { + state.jwt = action.payload.jwt; + state.email = action.payload.email; + state.userId = action.payload.userId; + } + }) + .addCase(restoreAtomicAuth.rejected, (state) => { + state.restoreLoading = false; + state.restoreLoaded = true; + }); + + builder.addCase(storeAtomicToken.fulfilled, (state, action) => { + state.jwt = action.payload.jwt; + state.email = action.payload.email; + state.userId = action.payload.userId; + }); + + builder.addCase(clearAtomicAuthThunk.fulfilled, () => initialState); + + builder + .addCase(verifyAtomicAuth.fulfilled, (state, action) => { + if (action.payload?.plan) { + state.subscriptionPlan = action.payload.plan; + } + }); + + builder + .addCase(applyPaygKey.pending, (state) => { + state.applyKeyBusy = true; + state.applyKeyError = null; + }) + .addCase(applyPaygKey.fulfilled, (state, action) => { + state.applyKeyBusy = false; + if (state.balance?.payg) { + state.balance = { + ...state.balance, + payg: { + ...state.balance.payg, + remaining: action.payload.remaining, + limit: action.payload.limit, + }, + }; + } + }) + .addCase(applyPaygKey.rejected, (state, action) => { + state.applyKeyBusy = false; + state.applyKeyError = + (action.payload as string | undefined) ?? action.error.message ?? "Failed to apply PAYG key"; + }); + + builder + .addCase(fetchAtomicBalance.pending, (state) => { + state.balanceLoading = true; + state.balanceError = null; + }) + .addCase(fetchAtomicBalance.fulfilled, (state, action) => { + state.balanceLoading = false; + state.balance = action.payload; + state.subscriptionPlan = action.payload.subscriptionPlan; + }) + .addCase(fetchAtomicBalance.rejected, (state, action) => { + state.balanceLoading = false; + state.balanceError = + (action.payload as string | undefined) ?? action.error.message ?? "Failed to load balance"; + }); + }, +}); + +export const atomicAuthActions = atomicAuthSlice.actions; +export const atomicAuthReducer = atomicAuthSlice.reducer; diff --git a/desktop/renderer/src/store/slices/auth/auth-types.ts b/desktop/renderer/src/store/slices/auth/auth-types.ts new file mode 100644 index 000000000000..5f9641d61eaa --- /dev/null +++ b/desktop/renderer/src/store/slices/auth/auth-types.ts @@ -0,0 +1,52 @@ +import type { AppDispatch, RootState } from "../../store"; +import type { HermesSetupMode } from "../mode-persistence"; + +export type AtomicAuthToken = { + jwt: string; + email: string; + userId: string; +}; + +/** Snapshot saved when leaving atomic-payg mode (JWT + gateway routing). */ +export type AtomicPaygBackup = { + auth: AtomicAuthToken; + savedAt: string; +}; + +/** Snapshot saved when leaving self-managed (provider + model + base URL). */ +export type SelfManagedBackup = { + activeProvider: string; + activeModel: string; + baseUrl: string; + savedAt: string; +}; + +/** Snapshot saved when leaving local-model mode. */ +export type LocalModelBackup = { + activeModelId: string; + savedAt: string; +}; + +export type SwitchContext = { + dispatch: AppDispatch; + getState: () => RootState; + port: number; +}; + +export type ModeSetupResult = { + hasBackup?: boolean; + restoredAuth?: AtomicAuthToken | null; + /** When true, orchestrator dispatches `applyPaygKey` after mode + auth are finalized. */ + needsApplyPaygKey?: boolean; +}; + +export interface ModeHandler { + saveBackup(ctx: SwitchContext): Promise; + teardown(ctx: SwitchContext): Promise; + setup(ctx: SwitchContext): Promise; +} + +export type SwitchModeParams = { + port: number; + target: HermesSetupMode; +}; diff --git a/desktop/renderer/src/store/slices/auth/index.ts b/desktop/renderer/src/store/slices/auth/index.ts new file mode 100644 index 000000000000..893ccbd7e104 --- /dev/null +++ b/desktop/renderer/src/store/slices/auth/index.ts @@ -0,0 +1,2 @@ +export { switchMode } from "./mode-switch"; +export type { SwitchModeParams } from "./auth-types"; diff --git a/desktop/renderer/src/store/slices/auth/mode-backups.ts b/desktop/renderer/src/store/slices/auth/mode-backups.ts new file mode 100644 index 000000000000..d741e0e02de9 --- /dev/null +++ b/desktop/renderer/src/store/slices/auth/mode-backups.ts @@ -0,0 +1,67 @@ +import type { AtomicPaygBackup, LocalModelBackup, SelfManagedBackup } from "./auth-types"; + +const LS_ATOMIC_PAYG = "hermes-desktop-backup-atomic-payg"; +const LS_SELF_MANAGED = "hermes-desktop-backup-self-managed"; +const LS_LOCAL_MODEL = "hermes-desktop-backup-local-model"; + +function readJson(key: string): T | null { + try { + const raw = localStorage.getItem(key); + if (!raw) return null; + return JSON.parse(raw) as T; + } catch { + return null; + } +} + +function writeJson(key: string, value: unknown): void { + try { + localStorage.setItem(key, JSON.stringify(value)); + } catch { + // best effort + } +} + +function removeKey(key: string): void { + try { + localStorage.removeItem(key); + } catch { + // ignore + } +} + +export function readAtomicPaygBackup(): AtomicPaygBackup | null { + return readJson(LS_ATOMIC_PAYG); +} + +export function saveAtomicPaygBackup(backup: AtomicPaygBackup): void { + writeJson(LS_ATOMIC_PAYG, backup); +} + +export function clearAtomicPaygBackup(): void { + removeKey(LS_ATOMIC_PAYG); +} + +export function readSelfManagedBackup(): SelfManagedBackup | null { + return readJson(LS_SELF_MANAGED); +} + +export function saveSelfManagedBackup(backup: SelfManagedBackup): void { + writeJson(LS_SELF_MANAGED, backup); +} + +export function clearSelfManagedBackup(): void { + removeKey(LS_SELF_MANAGED); +} + +export function readLocalModelBackup(): LocalModelBackup | null { + return readJson(LS_LOCAL_MODEL); +} + +export function saveLocalModelBackup(backup: LocalModelBackup): void { + writeJson(LS_LOCAL_MODEL, backup); +} + +export function clearLocalModelBackup(): void { + removeKey(LS_LOCAL_MODEL); +} diff --git a/desktop/renderer/src/store/slices/auth/mode-handler-atomic-payg.ts b/desktop/renderer/src/store/slices/auth/mode-handler-atomic-payg.ts new file mode 100644 index 000000000000..22865a2fbdad --- /dev/null +++ b/desktop/renderer/src/store/slices/auth/mode-handler-atomic-payg.ts @@ -0,0 +1,58 @@ +import { patchConfig } from "../../../services/api"; +import { atomicBackendApi, isUnauthorizedError } from "../../../services/atomic-backend-api"; +import type { ModeHandler, SwitchContext, ModeSetupResult } from "./auth-types"; +import { + clearAtomicPaygBackup, + readAtomicPaygBackup, + saveAtomicPaygBackup, +} from "./mode-backups"; + +export const atomicPaygHandler: ModeHandler = { + async saveBackup(ctx: SwitchContext): Promise { + if (readAtomicPaygBackup()) return; + + const { jwt, email, userId } = ctx.getState().atomicAuth; + if (!jwt || !userId) return; + + saveAtomicPaygBackup({ + auth: { jwt, email: email ?? "", userId }, + savedAt: new Date().toISOString(), + }); + }, + + async teardown(ctx: SwitchContext): Promise { + try { + await patchConfig(ctx.port, { + env: { OPENROUTER_API_KEY: "" }, + }); + } catch (err) { + console.warn("[atomicPaygHandler] teardown patchConfig failed:", err); + } + }, + + async setup(ctx: SwitchContext): Promise { + const backup = readAtomicPaygBackup(); + if (!backup?.auth?.jwt) { + return {}; + } + + try { + await atomicBackendApi.getMe(backup.auth.jwt); + } catch (err) { + if (isUnauthorizedError(err)) { + console.warn("[atomicPaygHandler] Backup JWT invalid, discarding backup"); + } else { + console.warn("[atomicPaygHandler] getMe failed:", err); + } + clearAtomicPaygBackup(); + return {}; + } + + clearAtomicPaygBackup(); + return { + hasBackup: true, + restoredAuth: backup.auth, + needsApplyPaygKey: true, + }; + }, +}; diff --git a/desktop/renderer/src/store/slices/auth/mode-handler-local-model.ts b/desktop/renderer/src/store/slices/auth/mode-handler-local-model.ts new file mode 100644 index 000000000000..827549aaf02a --- /dev/null +++ b/desktop/renderer/src/store/slices/auth/mode-handler-local-model.ts @@ -0,0 +1,37 @@ +import type { ModeHandler, SwitchContext, ModeSetupResult } from "./auth-types"; +import { + clearLocalModelBackup, + readLocalModelBackup, + saveLocalModelBackup, +} from "./mode-backups"; +import { llamacppActions, stopLlamacppServer } from "../llamacppSlice"; + +export const localModelHandler: ModeHandler = { + async saveBackup(ctx: SwitchContext): Promise { + const activeModelId = ctx.getState().llamacpp.activeModelId; + if (!activeModelId) return; + saveLocalModelBackup({ + activeModelId, + savedAt: new Date().toISOString(), + }); + }, + + async teardown(ctx: SwitchContext): Promise { + try { + await ctx.dispatch(stopLlamacppServer()).unwrap(); + } catch (err) { + console.warn("[localModelHandler] stopLlamacppServer:", err); + } + ctx.dispatch(llamacppActions.setActiveModelId(null)); + }, + + async setup(ctx: SwitchContext): Promise { + const backup = readLocalModelBackup(); + const hasBackup = Boolean(backup?.activeModelId); + if (backup?.activeModelId) { + ctx.dispatch(llamacppActions.setActiveModelId(backup.activeModelId)); + } + clearLocalModelBackup(); + return { hasBackup }; + }, +}; diff --git a/desktop/renderer/src/store/slices/auth/mode-handler-self-managed.ts b/desktop/renderer/src/store/slices/auth/mode-handler-self-managed.ts new file mode 100644 index 000000000000..266b03707ee3 --- /dev/null +++ b/desktop/renderer/src/store/slices/auth/mode-handler-self-managed.ts @@ -0,0 +1,68 @@ +import type { ConfigResponse } from "../../../services/api"; +import { getConfig, patchConfig } from "../../../services/api"; +import type { ModeHandler, SwitchContext, ModeSetupResult } from "./auth-types"; +import { + clearSelfManagedBackup, + readSelfManagedBackup, + saveSelfManagedBackup, +} from "./mode-backups"; + +function readBaseUrlFromSnap(snap: ConfigResponse): string { + const cfg = snap.config; + if (!cfg || typeof cfg !== "object") return ""; + const root = typeof cfg.base_url === "string" ? cfg.base_url : ""; + const modelSection = cfg.model; + if (typeof modelSection === "object" && modelSection !== null) { + const nested = (modelSection as Record).base_url; + if (typeof nested === "string" && nested.trim()) return nested; + } + return root; +} + +export const selfManagedHandler: ModeHandler = { + async saveBackup(ctx: SwitchContext): Promise { + if (readSelfManagedBackup()) return; + + try { + const snap = await getConfig(ctx.port); + saveSelfManagedBackup({ + activeProvider: snap.activeProvider || "", + activeModel: snap.activeModel || "", + baseUrl: readBaseUrlFromSnap(snap), + savedAt: new Date().toISOString(), + }); + } catch (err) { + console.warn("[selfManagedHandler] saveBackup getConfig failed:", err); + } + }, + + async teardown(_ctx: SwitchContext): Promise { + // Env API keys stay on disk so the user can return to self-managed without + // re-entering secrets. Switching to atomic-payg overwrites OpenRouter only. + }, + + async setup(ctx: SwitchContext): Promise { + const backup = readSelfManagedBackup(); + if (!backup?.activeProvider) { + return { hasBackup: false }; + } + + try { + const body: { config: Record } = { + config: { + provider: backup.activeProvider, + model: backup.activeModel, + }, + }; + if (backup.baseUrl.trim()) { + body.config.base_url = backup.baseUrl.trim(); + } + await patchConfig(ctx.port, body); + } catch (err) { + console.warn("[selfManagedHandler] setup patchConfig failed:", err); + } + + clearSelfManagedBackup(); + return { hasBackup: true }; + }, +}; diff --git a/desktop/renderer/src/store/slices/auth/mode-switch.ts b/desktop/renderer/src/store/slices/auth/mode-switch.ts new file mode 100644 index 000000000000..ab21e78c3e60 --- /dev/null +++ b/desktop/renderer/src/store/slices/auth/mode-switch.ts @@ -0,0 +1,88 @@ +/** + * Unified mode-switching orchestrator (openclaw parity). + * Each mode implements symmetric saveBackup / teardown / setup phases. + */ +import { createAsyncThunk } from "@reduxjs/toolkit"; +import type { RootState, AppDispatch } from "../../store"; +import { setMode, syncConfigFromGateway } from "../configSlice"; +import { persistMode, type HermesSetupMode } from "../mode-persistence"; +import { + applyPaygKey, + atomicAuthActions, + storeAtomicToken, +} from "../atomicAuthSlice"; +import { fetchLlamacppServerStatus } from "../llamacppSlice"; +import type { SwitchModeParams, ModeSetupResult } from "./auth-types"; +import { atomicPaygHandler } from "./mode-handler-atomic-payg"; +import { selfManagedHandler } from "./mode-handler-self-managed"; +import { localModelHandler } from "./mode-handler-local-model"; + +const handlers: Record = { + "atomic-payg": atomicPaygHandler, + "self-managed": selfManagedHandler, + "local-model": localModelHandler, +}; + +export type { SwitchModeParams } from "./auth-types"; + +export const switchMode = createAsyncThunk< + ModeSetupResult | void, + SwitchModeParams, + { state: RootState; dispatch: AppDispatch } +>("mode/switch", async ({ port, target }, thunkApi) => { + const dispatch = thunkApi.dispatch; + const getState = thunkApi.getState; + const current = getState().config.mode; + + if (current === target) { + return; + } + + const ctx: SwitchContext = { dispatch, getState, port }; + + if (current) { + await handlers[current].saveBackup(ctx); + await handlers[current].teardown(ctx); + } + + dispatch(atomicAuthActions.clearSliceAuthForModeSwitch()); + + const result: ModeSetupResult = await handlers[target].setup(ctx); + + dispatch(setMode(target)); + persistMode(target); + + if (result.restoredAuth) { + dispatch(atomicAuthActions.setRestoredAuth(result.restoredAuth)); + try { + await dispatch( + storeAtomicToken({ + jwt: result.restoredAuth.jwt, + email: result.restoredAuth.email, + userId: result.restoredAuth.userId, + }), + ).unwrap(); + } catch (err) { + console.warn("[switchMode] storeAtomicToken after restore failed:", err); + } + } + + if (result.needsApplyPaygKey) { + try { + await dispatch(applyPaygKey({ port })).unwrap(); + } catch (err) { + console.warn("[switchMode] applyPaygKey failed:", err); + } + } + + try { + await dispatch(syncConfigFromGateway(port)).unwrap(); + } catch (err) { + console.warn("[switchMode] syncConfigFromGateway failed:", err); + } + + void dispatch(fetchLlamacppServerStatus()); + document.dispatchEvent(new Event("hermes-config-changed")); + + return result; +}); diff --git a/desktop/renderer/src/store/slices/configSlice.ts b/desktop/renderer/src/store/slices/configSlice.ts index 775461a526ff..9686ed0e317d 100644 --- a/desktop/renderer/src/store/slices/configSlice.ts +++ b/desktop/renderer/src/store/slices/configSlice.ts @@ -1,4 +1,5 @@ -import { createSlice, type PayloadAction } from "@reduxjs/toolkit"; +import { createAsyncThunk, createSlice, type PayloadAction } from "@reduxjs/toolkit"; +import { getConfig } from "../../services/api"; import { readPersistedMode, type HermesSetupMode } from "./mode-persistence"; type ConfigState = { @@ -15,6 +16,18 @@ const initialState: ConfigState = { mode: readPersistedMode() ?? "self-managed", }; +export const syncConfigFromGateway = createAsyncThunk( + "config/syncFromGateway", + async (port: number) => { + const snap = await getConfig(port); + return { + provider: snap.activeProvider || null, + model: snap.activeModel || null, + apiKeyConfigured: snap.hasApiKeys, + }; + }, +); + const configSlice = createSlice({ name: "config", initialState, @@ -35,6 +48,13 @@ const configSlice = createSlice({ return initialState; }, }, + extraReducers: (builder) => { + builder.addCase(syncConfigFromGateway.fulfilled, (state, action) => { + state.provider = action.payload.provider; + state.model = action.payload.model; + state.apiKeyConfigured = action.payload.apiKeyConfigured; + }); + }, }); export const { setProvider, setModel, setApiKeyConfigured, setMode, resetConfig } = diff --git a/desktop/renderer/src/store/slices/mode-persistence.ts b/desktop/renderer/src/store/slices/mode-persistence.ts index edfb9d48bb82..421b6d124b50 100644 --- a/desktop/renderer/src/store/slices/mode-persistence.ts +++ b/desktop/renderer/src/store/slices/mode-persistence.ts @@ -1,8 +1,9 @@ -export type HermesSetupMode = "self-managed" | "local-model"; +export type HermesSetupMode = "self-managed" | "local-model" | "atomic-payg"; export const MODE_LABELS: Record = { "self-managed": "API keys", "local-model": "Local Models", + "atomic-payg": "Pay as you go", }; const MODE_LS_KEY = "hermes-desktop-mode"; @@ -18,7 +19,9 @@ export function persistMode(mode: HermesSetupMode): void { export function readPersistedMode(): HermesSetupMode | null { try { const val = localStorage.getItem(MODE_LS_KEY); - if (val === "self-managed" || val === "local-model") return val; + if (val === "self-managed" || val === "local-model" || val === "atomic-payg") { + return val; + } return null; } catch { return null; diff --git a/desktop/renderer/src/store/slices/mode-switch.ts b/desktop/renderer/src/store/slices/mode-switch.ts deleted file mode 100644 index 966f04e5cb37..000000000000 --- a/desktop/renderer/src/store/slices/mode-switch.ts +++ /dev/null @@ -1,23 +0,0 @@ -import { createAsyncThunk } from "@reduxjs/toolkit"; -import { setMode } from "./configSlice"; -import type { RootState, AppDispatch } from "../store"; -import { persistMode, type HermesSetupMode } from "./mode-persistence"; - -export type SwitchModeParams = { - port: number; - target: HermesSetupMode; -}; - -export const switchMode = createAsyncThunk< - void, - SwitchModeParams, - { state: RootState; dispatch: AppDispatch } ->("mode/switch", async ({ target }, thunkApi) => { - const dispatch = thunkApi.dispatch; - const current = thunkApi.getState().config.mode; - - if (current === target) return; - - dispatch(setMode(target)); - persistMode(target); -}); diff --git a/desktop/renderer/src/store/store.ts b/desktop/renderer/src/store/store.ts index 0baad45b09a0..28b35959e490 100644 --- a/desktop/renderer/src/store/store.ts +++ b/desktop/renderer/src/store/store.ts @@ -6,6 +6,7 @@ import { chatReducer } from "./slices/chatSlice"; import { skillsReducer } from "./slices/skillsSlice"; import { llamacppReducer } from "./slices/llamacppSlice"; import { desktopWarmupReducer } from "./slices/desktopWarmupSlice"; +import { atomicAuthReducer } from "./slices/atomicAuthSlice"; export const store = configureStore({ reducer: { @@ -16,6 +17,7 @@ export const store = configureStore({ skills: skillsReducer, llamacpp: llamacppReducer, desktopWarmup: desktopWarmupReducer, + atomicAuth: atomicAuthReducer, }, }); diff --git a/desktop/renderer/src/ui/app/App.tsx b/desktop/renderer/src/ui/app/App.tsx index 01f5101f99b5..fba03773343d 100644 --- a/desktop/renderer/src/ui/app/App.tsx +++ b/desktop/renderer/src/ui/app/App.tsx @@ -1,5 +1,12 @@ import React from "react"; -import { Navigate, Outlet, Route, Routes, useNavigate, useSearchParams } from "react-router-dom"; +import { + Navigate, + Outlet, + Route, + Routes, + useNavigate, + useSearchParams, +} from "react-router-dom"; import { Brand, SpinningSplashLogo, PoweredBanner } from "@shared/kit"; import { useAppDispatch, useAppSelector } from "@store/hooks"; import { initGatewayState } from "@store/slices/gatewaySlice"; @@ -28,8 +35,20 @@ import { scheduleWarmHubSkillsCache } from "../../services/warm-hub-skills-cache import { useDesktopLocalWarmup } from "../chat/hooks/useDesktopLocalWarmup"; import { DesktopWarmupBanner } from "../chat/components/DesktopWarmupBanner"; import { useAppOpenedEvent } from "@analytics"; +import { useAtomicDeepLink } from "../../hooks/useAtomicDeepLink"; +import { useAtomicTopupPolling } from "../../hooks/useAtomicTopupPolling"; +import { + clearPostPaygSuccessNavigate, + consumePostPaygSuccessNavigate, +} from "../../services/atomic-backend-api"; +import { + atomicAuthActions, + fetchAtomicBalance, +} from "@store/slices/atomicAuthSlice"; import { UpdateBanner } from "../updates/UpdateBanner"; import { LlamacppDownloadBanner } from "../updates/LlamacppDownloadBanner"; +import { LowBalanceBanner } from "../shared/atomic/LowBalanceBanner"; +import { useAtomicAuthBootstrap } from "../shared/atomic/useAtomicAuthBootstrap"; import a from "./App.module.css"; const SIDEBAR_OPEN_LS_KEY = "hermes:sidebar-open"; @@ -50,7 +69,9 @@ function OtherTabRoute() { return ( <> {error && ( -
{error}
+
+ {error} +
)} @@ -60,7 +81,14 @@ function OtherTabRoute() { function LoadingScreen() { return (
-
+
Starting Atomic Hermes...
Initializing backend services
@@ -76,7 +104,8 @@ function ErrorScreen({ error }: { error: string }) {
Atomic Hermes failed to start
- The backend process did not become available. Check the logs for details. + The backend process did not become available. Check the logs for + details. 
{error || "No details."}
@@ -101,7 +130,9 @@ function ChatRoute() { function SidebarLayout() { useAppOpenedEvent(); - const [sidebarOpen, setSidebarOpen] = React.useState(readSidebarOpenFromStorage); + const [sidebarOpen, setSidebarOpen] = React.useState( + readSidebarOpenFromStorage, + ); React.useEffect(() => { try { @@ -141,8 +172,20 @@ export function FullscreenTopbar(props: { className={a.UiTopbarBackButton} onClick={() => void navigate(-1)} > - - + + Back @@ -152,23 +195,40 @@ export function FullscreenTopbar(props: { } export function FullscreenShell(props: { children: React.ReactNode }) { - return ( -
- {props.children} -
- ); + return
{props.children}
; } export function App() { useDesktopLocalWarmup(); + useAtomicAuthBootstrap(); + useAtomicTopupPolling(); const dispatch = useAppDispatch(); const state = useAppSelector((s) => s.gateway.state); const onboardingLoaded = useAppSelector((s) => s.onboarding.loaded); - const onboarded = useAppSelector((s) => s.onboarding.onboarded); + const onboarded = useAppSelector((s) => false); const navigate = useNavigate(); const didAutoNavRef = React.useRef(false); const wasRestartingRef = React.useRef(false); + // Stripe Checkout completes in the browser; localhost thanks page fires + // `atomicbot-hermes://stripe-success` (see getStripePaygSuccessUrl). We sync + // balance and optionally navigate (e.g. onboarding → /setup/atomic-model via + // rememberPostPaygSuccessNavigate). + useAtomicDeepLink({ + onStripeSuccess: () => { + const postPayNav = consumePostPaygSuccessNavigate(); + dispatch(atomicAuthActions.setTopupPending(false)); + void dispatch(fetchAtomicBalance({ sync: true })); + if (postPayNav) { + void navigate(postPayNav, { replace: true }); + } + }, + onStripeCancel: () => { + dispatch(atomicAuthActions.setTopupPending(false)); + clearPostPaygSuccessNavigate(); + }, + }); + React.useEffect(() => { void dispatch(initGatewayState()); void dispatch(loadOnboardingState()); @@ -215,49 +275,50 @@ export function App() { if (state?.kind === "ready") { return ( <> - - - } /> - }> - } /> - } /> - } /> - } /> - } /> - } /> - }> - } /> - } /> - } /> - } /> - } /> - } /> - - } - /> - } /> + + {onboarded ? : null} + + } /> + }> + } /> + } /> + } /> + } /> - } + path="skills" + element={} /> - } /> + } /> + }> + } /> + } + /> + } /> + } + /> + } /> + } /> + + } + /> + } /> + } /> + - - } /> - } /> - } /> - + } /> + } /> + } /> + ); } diff --git a/desktop/renderer/src/ui/settings/AiModelsInlineProvider.tsx b/desktop/renderer/src/ui/settings/AiModelsInlineProvider.tsx index d6c1d5ff8313..da4556cd2e99 100644 --- a/desktop/renderer/src/ui/settings/AiModelsInlineProvider.tsx +++ b/desktop/renderer/src/ui/settings/AiModelsInlineProvider.tsx @@ -81,14 +81,8 @@ export function AiModelsInlineProvider(props: { }, [props]); const handleSave = React.useCallback(async () => { - // #region agent log - fetch('http://127.0.0.1:7831/ingest/afbd3787-1f02-4bfb-8a9a-c6c81cb2ee48',{method:'POST',headers:{'Content-Type':'application/json','X-Debug-Session-Id':'bdc941'},body:JSON.stringify({sessionId:'bdc941',hypothesisId:'H1',location:'AiModelsInlineProvider.tsx:handleSave:entry',message:'Save click fired',data:{providerId:props.provider.id,apiKeyLen:props.apiKey.length,apiKeyTrimmedLen:props.apiKey.trim().length,isSavingProvider:props.isSavingProvider,providerConfigured:props.providerConfigured,editing,authType:props.provider.authType},timestamp:Date.now()})}).catch(()=>{}); - // #endregion setClipboardError(""); const saved = await props.onSave(); - // #region agent log - fetch('http://127.0.0.1:7831/ingest/afbd3787-1f02-4bfb-8a9a-c6c81cb2ee48',{method:'POST',headers:{'Content-Type':'application/json','X-Debug-Session-Id':'bdc941'},body:JSON.stringify({sessionId:'bdc941',hypothesisId:'H2',location:'AiModelsInlineProvider.tsx:handleSave:result',message:'onSave returned',data:{saved},timestamp:Date.now()})}).catch(()=>{}); - // #endregion if (saved) { setEditing(false); } diff --git a/desktop/renderer/src/ui/settings/AiModelsTab.module.css b/desktop/renderer/src/ui/settings/AiModelsTab.module.css index 229e606bee9d..8c4c8c4851e4 100644 --- a/desktop/renderer/src/ui/settings/AiModelsTab.module.css +++ b/desktop/renderer/src/ui/settings/AiModelsTab.module.css @@ -76,14 +76,13 @@ } .statusBarHorizontalMain { - display: flex; - flex-direction: row; - flex-wrap: wrap; - align-items: flex-start; - justify-content: space-between; - column-gap: 4px; + display: grid; + grid-template-columns: 160px minmax(240px, 1.2fr) 160px minmax(180px, 1fr); + column-gap: 24px; row-gap: 4px; + align-items: start; width: 100%; + min-width: 0; } .statusBarHorizontal .statusBarRunningModel { @@ -99,7 +98,6 @@ flex-direction: column; align-items: flex-start; gap: 2px; - flex: 0 1 auto; min-width: 0; max-width: 100%; } @@ -110,10 +108,8 @@ } .statusBarHorizontal .statusValue { - flex: 0 1 auto; - align-self: stretch; + width: 100%; min-width: 0; - min-height: 24px; } .statusBarHorizontal .statusBarRunningModel .statusValue { @@ -122,12 +118,27 @@ } .statusBarHorizontal .statusValueText { + display: block; + width: 100%; + min-width: 0; line-height: 16px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +@media (max-width: 1180px) { + .statusBarHorizontalMain { + grid-template-columns: minmax(140px, 1fr) minmax(180px, 1fr); + column-gap: 20px; + row-gap: 12px; + } + + .statusBarHorizontal .statusBarRunningModel { + grid-column: 2; + } +} + .statusSegment { display: flex; flex-direction: column; diff --git a/desktop/renderer/src/ui/settings/AiModelsTab.tsx b/desktop/renderer/src/ui/settings/AiModelsTab.tsx index 1fa8a5da4b31..94c318468a15 100644 --- a/desktop/renderer/src/ui/settings/AiModelsTab.tsx +++ b/desktop/renderer/src/ui/settings/AiModelsTab.tsx @@ -1,8 +1,14 @@ import React from "react"; import { useAppDispatch, useAppSelector } from "@store/hooks"; -import { switchMode } from "@store/slices/mode-switch"; -import { MODE_LABELS, type HermesSetupMode } from "@store/slices/mode-persistence"; -import { fetchLlamacppServerStatus, stopLlamacppServer } from "@store/slices/llamacppSlice"; +import { switchMode } from "@store/slices/auth/mode-switch"; +import { + MODE_LABELS, + type HermesSetupMode, +} from "@store/slices/mode-persistence"; +import { + fetchLlamacppServerStatus, + stopLlamacppServer, +} from "@store/slices/llamacppSlice"; import { RichSelect, type RichOption } from "../setup/RichSelect"; import { getModelTier, TIER_INFO } from "../setup/model-presentation"; import { PROVIDERS, resolveProviderIconUrl } from "../setup/providers"; @@ -16,12 +22,15 @@ import { resolveLlamacppServerUiKey, } from "./AiModelsStatusBar"; import { LocalModelsTab } from "./local-models/LocalModelsTab"; +import { AtomicAccountTab } from "./atomic/AtomicAccountTab"; +import { AtomicSignInPrompt } from "../shared/atomic/AtomicSignInPrompt"; import s from "./AiModelsTab.module.css"; -function getProviderBadge(provider: (typeof PROVIDERS)[number]): - | { text: string; variant: string } - | undefined { - if (provider.recommended) return { text: "Recommended", variant: "recommended" }; +function getProviderBadge( + provider: (typeof PROVIDERS)[number], +): { text: string; variant: string } | undefined { + if (provider.recommended) + return { text: "Recommended", variant: "recommended" }; if (provider.popular) return { text: "Popular", variant: "popular" }; if (provider.localModels) return { text: "Local", variant: "local" }; return undefined; @@ -35,14 +44,26 @@ function ConnectionToggle(props: { const active = props.activeMode; return (
-
+
+
@@ -60,11 +81,16 @@ function ConnectionToggle(props: { export function AiModelsTab() { const dispatch = useAppDispatch(); const authMode = useAppSelector((st) => st.config.mode); + const jwt = useAppSelector((st) => st.atomicAuth.jwt); const llamacpp = useAppSelector((st) => st.llamacpp); const [tabMode, setTabMode] = React.useState(authMode); const [modeSwitchBusy, setModeSwitchBusy] = React.useState(false); + React.useEffect(() => { + setTabMode(authMode); + }, [authMode]); + const { port, configSnap, @@ -119,9 +145,12 @@ export function AiModelsTab() { void loadModels(selectedProvider); }, [loadModels, selectedProvider, setSelectedModel]); - const provider = PROVIDERS.find((item) => item.id === selectedProvider) ?? null; + const provider = + PROVIDERS.find((item) => item.id === selectedProvider) ?? null; const effectiveConfiguredModel = - selectedProvider && selectedProvider === configSnap?.activeProvider ? configuredModel : ""; + selectedProvider && selectedProvider === configSnap?.activeProvider + ? configuredModel + : ""; const selectedModelValue = selectedModel || effectiveConfiguredModel || null; const providerOptions = React.useMemo[]>( @@ -146,7 +175,9 @@ export function AiModelsTab() { return { value: modelId, label: modelId, - badge: tier ? { text: TIER_INFO[tier].label, variant: tier } : undefined, + badge: tier + ? { text: TIER_INFO[tier].label, variant: tier } + : undefined, description: tier ? TIER_INFO[tier].description : undefined, }; }); @@ -185,17 +216,67 @@ export function AiModelsTab() { await loadModels(selectedProvider); } return saved; - }, [canEditConfig, loadModels, oauthStep, provider, saveProviderConfig, selectedProvider, startOAuth]); + }, [ + canEditConfig, + loadModels, + oauthStep, + provider, + saveProviderConfig, + selectedProvider, + startOAuth, + ]); React.useEffect(() => { - if (!selectedProvider || isLoadingModels || isSavingModel || modelOptions.length === 0) return; + if (tabMode === "atomic-payg") return; + if ( + !selectedProvider || + isLoadingModels || + isSavingModel || + modelOptions.length === 0 + ) + return; if (selectedModelValue?.startsWith(LLAMACPP_PRIMARY_PREFIX)) return; - if (selectedModelValue && modelOptions.some((option) => option.value === selectedModelValue)) return; + if ( + selectedModelValue && + modelOptions.some((option) => option.value === selectedModelValue) + ) + return; const fallbackModel = modelOptions[0]?.value; if (!fallbackModel) return; setSelectedModel(fallbackModel); void saveModelSelection(fallbackModel, selectedProvider); - }, [isLoadingModels, isSavingModel, modelOptions, saveModelSelection, selectedModelValue, selectedProvider, setSelectedModel]); + }, [ + isLoadingModels, + isSavingModel, + modelOptions, + saveModelSelection, + selectedModelValue, + selectedProvider, + setSelectedModel, + tabMode, + ]); + + React.useEffect(() => { + if ( + tabMode !== "atomic-payg" || + !jwt || + modeSwitchBusy || + authMode !== "atomic-payg" + ) + return; + if (selectedProvider !== "openrouter") { + setSelectedProvider("openrouter"); + } + void loadModels("openrouter"); + }, [ + authMode, + jwt, + loadModels, + modeSwitchBusy, + selectedProvider, + setSelectedProvider, + tabMode, + ]); // ── Mode switching ── @@ -207,13 +288,15 @@ export function AiModelsTab() { setModeSwitchBusy(true); try { await dispatch(switchMode({ port, target: mode })).unwrap(); + await reloadConfig(); } catch (err) { console.error("[AiModelsTab] switchMode failed:", err); + setTabMode(authMode); } finally { setModeSwitchBusy(false); } }, - [authMode, dispatch, port], + [authMode, dispatch, port, reloadConfig], ); // ── Server stop ── @@ -235,22 +318,40 @@ export function AiModelsTab() { const isLlamacppProvider = isProfileUsingLlamacppServer(configSnap); - const statusModeLabel = isLlamacppProvider ? MODE_LABELS["local-model"] : MODE_LABELS["self-managed"]; + const statusModeLabel = + authMode === "atomic-payg" + ? MODE_LABELS["atomic-payg"] + : isLlamacppProvider + ? MODE_LABELS["local-model"] + : MODE_LABELS["self-managed"]; const currentModelName = React.useMemo(() => { + if (authMode === "atomic-payg") { + return selectedModelValue || null; + } if (isLlamacppProvider) { - const localModel = llamacpp.models.find((m) => m.id === llamacpp.activeModelId); + const localModel = llamacpp.models.find( + (m) => m.id === llamacpp.activeModelId, + ); if (localModel?.name) return localModel.name; - if (llamacpp.activeModelId) return formatModelIdForStatusBar(llamacpp.activeModelId); + if (llamacpp.activeModelId) + return formatModelIdForStatusBar(llamacpp.activeModelId); return null; } return selectedModelValue || null; - }, [isLlamacppProvider, llamacpp.activeModelId, llamacpp.models, selectedModelValue]); + }, [ + authMode, + isLlamacppProvider, + llamacpp.activeModelId, + llamacpp.models, + selectedModelValue, + ]); const runningModelLabel = React.useMemo(() => { const uiKey = resolveLlamacppServerUiKey(llamacpp.serverStatus); if (uiKey === "stopped") return "None"; - const rawId = llamacpp.serverStatus?.activeModelId ?? llamacpp.activeModelId ?? null; + const rawId = + llamacpp.serverStatus?.activeModelId ?? llamacpp.activeModelId ?? null; if (!rawId) return "None"; const localModel = llamacpp.models.find((m) => m.id === rawId); if (localModel?.name) return localModel.name; @@ -283,7 +384,9 @@ export function AiModelsTab() { {modeSwitchBusy && (
)} @@ -297,7 +400,8 @@ export function AiModelsTab() {
Coming Soon
- Local models support for this platform is under development. Stay tuned! + Local models support for this platform is under development. + Stay tuned!
@@ -332,7 +436,12 @@ export function AiModelsTab() { ? "Enter API key to choose a model" : "Select model..." } - disabled={!selectedProvider || isLoadingModels || isSavingModel || modelOptions.length === 0} + disabled={ + !selectedProvider || + isLoadingModels || + isSavingModel || + modelOptions.length === 0 + } disabledStyles={!selectedProvider || modelOptions.length === 0} onlySelectedIcon /> @@ -370,6 +479,75 @@ export function AiModelsTab() { ) : null}
)} + + {tabMode === "atomic-payg" && !modeSwitchBusy && ( +
+ {!jwt ? ( + + ) : ( + <> +
+
+
Provider
+
+ Atomic +
+
+ +
+
Model
+ { + setSelectedModel(modelId); + void saveModelSelection(modelId, "openrouter"); + }} + options={modelOptions} + placeholder={ + modelOptions.length === 0 + ? "Loading models…" + : "Select model..." + } + disabled={ + selectedProvider !== "openrouter" || + isLoadingModels || + isSavingModel || + modelOptions.length === 0 + } + disabledStyles={modelOptions.length === 0} + onlySelectedIcon + /> +
+
+ + {selectedProvider === "openrouter" && + modelOptions.length === 0 && + !isLoadingModels ? ( +
+ Models could not be loaded. Check your connection or try + reloading settings. +
+ ) : null} + +
+ +
+ + )} +
+ )}
); } diff --git a/desktop/renderer/src/ui/settings/OtherTab.module.css b/desktop/renderer/src/ui/settings/OtherTab.module.css index 6df10cb28448..6539ff0afdbe 100644 --- a/desktop/renderer/src/ui/settings/OtherTab.module.css +++ b/desktop/renderer/src/ui/settings/OtherTab.module.css @@ -319,7 +319,7 @@ color: #f2f2f7; } -/* Subscription mode confirmation popup */ +/* Pay-as-you-go mode confirmation popup */ .UiConfirmDescription { font-size: 14px; diff --git a/desktop/renderer/src/ui/settings/atomic/AtomicAccountTab.module.css b/desktop/renderer/src/ui/settings/atomic/AtomicAccountTab.module.css new file mode 100644 index 000000000000..cd7ce9822837 --- /dev/null +++ b/desktop/renderer/src/ui/settings/atomic/AtomicAccountTab.module.css @@ -0,0 +1,312 @@ +/* ── AtomicAccountTab ── */ + +.root { + display: flex; + flex-direction: column; + gap: 20px; + animation: fadeIn ease 0.2s; +} + +@keyframes fadeIn { + 0% { + opacity: 0; + } + 100% { + opacity: 1; + } +} + +/* ── Balance card ── */ + +.balanceCard { + background: rgba(255, 255, 255, 0.04); + border-radius: var(--radius-md, 12px); + padding: 20px; +} + +.balanceHeader { + display: flex; + align-items: center; + justify-content: space-between; + margin-bottom: 16px; +} + +.balanceTitle { + font-size: 15px; + font-weight: 600; + margin: 0; + color: var(--text, #fff); +} + +.balanceHero { + display: flex; + flex-direction: column; + align-items: flex-start; + gap: 6px; +} + +.balanceRow { + display: flex; + gap: 4px; + align-items: center; +} + +.balanceAmount { + font-size: 36px; + font-weight: 700; + color: var(--text, #fff); + line-height: 1.1; + letter-spacing: -0.5px; +} + +.balanceAmount--depleted { + color: var(--error, #ef4444); +} + +.balanceLabel { + font-size: 14px; + color: rgba(255, 255, 255, 0.5); +} + +.balancePollingHint { + display: inline-flex; + align-items: center; + gap: 6px; + font-size: 13px; + color: rgba(255, 255, 255, 0.5); +} + +.balancePollingSpinner { + display: inline-block; + width: 14px; + height: 14px; + border: 2px solid rgba(255, 255, 255, 0.15); + border-top-color: rgba(255, 255, 255, 0.6); + border-radius: 50%; + animation: balanceSpin 0.7s linear infinite; +} + +@keyframes balanceSpin { + to { + transform: rotate(360deg); + } +} + +.depletedCard { + display: flex; + align-items: center; + gap: 12px; + padding: 14px 16px; + border-radius: var(--radius-xl, 16px); + background: rgba(239, 68, 68, 0.08); + border: 1px solid rgba(239, 68, 68, 0.18); + margin-top: 16px; +} + +.depletedBody { + flex: 1; + min-width: 0; + display: flex; + flex-direction: column; + gap: 2px; +} + +.depletedTitle { + font-size: 14px; + font-weight: 600; + color: var(--text, #fff); +} + +.depletedSubtitle { + font-size: 13px; + color: var(--muted2, rgba(230, 237, 243, 0.65)); +} + +.depletedAction { + appearance: none; + border: none; + border-radius: var(--radius-full, 999px); + padding: 8px 18px; + font-size: 13px; + font-weight: 650; + cursor: pointer; + white-space: nowrap; + flex-shrink: 0; + background: var(--lime, #aeff00); + color: var(--surface-primary, #121212); + transition: opacity 150ms ease; +} + +.depletedAction:hover { + opacity: 0.85; +} + +.depletedAction:active { + opacity: 0.7; +} + +.depletedAction:disabled { + opacity: 0.55; + cursor: default; +} + +.errorRow { + margin-top: 12px; + font-size: 13px; + color: var(--error, #ef4444); +} + +/* ── One-Time Top-Up ── */ + +.topUpSection { + display: flex; + flex-direction: column; + gap: 12px; +} + +.topUpTitle { + font-size: 15px; + font-weight: 600; + margin: 0; + color: var(--text, #fff); +} + +.topUpRow { + display: flex; + align-items: center; + gap: 10px; +} + +.topUpInputWrap { + flex: 1; + display: flex; + align-items: center; + gap: 6px; + background: rgba(255, 255, 255, 0.06); + border: 1px solid rgba(255, 255, 255, 0.1); + border-radius: var(--radius-md, 10px); + padding: 10px 14px; + transition: border-color 140ms ease; +} + +.topUpInputWrap:focus-within { + border-color: rgba(255, 255, 255, 0.25); +} + +.topUpCurrency { + font-size: 15px; + color: rgba(255, 255, 255, 0.5); + font-weight: 500; +} + +.topUpInput { + flex: 1; + border: none; + background: none; + color: var(--text, #fff); + font-size: 15px; + font-weight: 500; + outline: none; + min-width: 0; + -moz-appearance: textfield; +} + +.topUpInput::-webkit-inner-spin-button, +.topUpInput::-webkit-outer-spin-button { + -webkit-appearance: none; + margin: 0; +} + +.topUpButton { + appearance: none; + border: none; + border-radius: var(--radius-full, 999px); + background: #ffffff; + color: var(--surface-secondary, #292929); + font-weight: 600; + font-size: 14px; + padding: 8px 18px; + cursor: pointer; + white-space: nowrap; + transition: opacity 150ms ease; + width: 105px; +} + +.topUpButton:hover { + opacity: 0.85; +} + +.topUpButton:disabled { + opacity: 0.55; + cursor: default; +} + +.topUpPendingRow { + font-size: 13px; + color: var(--muted2, rgba(230, 237, 243, 0.65)); +} + +.topUpPendingAction { + margin-left: 8px; + background: none; + border: none; + text-decoration: underline; + cursor: pointer; + color: inherit; + font: inherit; + padding: 0; +} + +/* ── Account footer (email + logout) ── */ + +.accountFooter { + display: flex; + align-items: center; + gap: 10px; + padding: 8px 0; +} + +.accountAvatar { + width: 32px; + height: 32px; + border-radius: 50%; + background: rgba(255, 255, 255, 0.1); + display: flex; + align-items: center; + justify-content: center; + font-size: 14px; + font-weight: 600; + color: rgba(255, 255, 255, 0.7); + flex-shrink: 0; +} + +.accountEmail { + flex: 1; + font-size: 14px; + color: rgba(255, 255, 255, 0.6); + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.logoutBtn { + appearance: none; + border: none; + background: none; + color: rgba(255, 255, 255, 0.4); + cursor: pointer; + padding: 4px; + border-radius: 6px; + display: flex; + align-items: center; + justify-content: center; + transition: + color 140ms ease, + background 140ms ease; + flex-shrink: 0; +} + +.logoutBtn:hover { + color: var(--text, #fff); + background: rgba(255, 255, 255, 0.08); +} diff --git a/desktop/renderer/src/ui/settings/atomic/AtomicAccountTab.tsx b/desktop/renderer/src/ui/settings/atomic/AtomicAccountTab.tsx new file mode 100644 index 000000000000..e039a35b9c13 --- /dev/null +++ b/desktop/renderer/src/ui/settings/atomic/AtomicAccountTab.tsx @@ -0,0 +1,236 @@ +import React from "react"; +import { useAppDispatch, useAppSelector } from "@store/hooks"; +import { + atomicAuthActions, + clearAtomicAuthThunk, + fetchAtomicBalance, +} from "@store/slices/atomicAuthSlice"; +import { clearAtomicPaygBackup } from "@store/slices/auth/mode-backups"; +import { patchConfig } from "../../../services/api"; +import { + STRIPE_PAYG_CANCEL_URL, + atomicBackendApi, + getStripePaygSuccessUrl, +} from "../../../services/atomic-backend-api"; +import s from "./AtomicAccountTab.module.css"; + +const DEPLETED_THRESHOLD_USD = 0.05; +const DEFAULT_TOPUP_USD = "10.00"; + +function openExternal(url: string): void { + const api = (window as { hermesAPI?: { openExternal?: (u: string) => void } }) + .hermesAPI; + if (api?.openExternal) { + void api.openExternal(url); + return; + } + window.open(url, "_blank"); +} + +function formatDollars(value: number | null | undefined): string { + if (value == null || !Number.isFinite(value)) return "$0.00"; + return `$${value.toFixed(2)}`; +} + +function LogOutIcon() { + return ( + + + + ); +} + +export function AtomicAccountTab(props: { port: number }) { + const dispatch = useAppDispatch(); + + const jwt = useAppSelector((state) => state.atomicAuth.jwt); + const email = useAppSelector((state) => state.atomicAuth.email); + const balance = useAppSelector((state) => state.atomicAuth.balance); + const balanceLoading = useAppSelector( + (state) => state.atomicAuth.balanceLoading, + ); + const balanceError = useAppSelector((state) => state.atomicAuth.balanceError); + const topupBusy = useAppSelector((state) => state.atomicAuth.topupBusy); + const topupError = useAppSelector((state) => state.atomicAuth.topupError); + const topupPending = useAppSelector((state) => state.atomicAuth.topupPending); + + const [topUpAmount, setTopUpAmount] = React.useState(DEFAULT_TOPUP_USD); + + // Refresh balance on tab open. Single-shot per mount; matches openclaw behavior. + React.useEffect(() => { + if (jwt) { + void dispatch(fetchAtomicBalance({ sync: true })); + } + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); + + const remaining = balance?.payg?.remaining; + const balanceDepleted = + balance !== null && + remaining != null && + remaining <= DEPLETED_THRESHOLD_USD; + + const handleTopUp = React.useCallback(async () => { + if (!jwt) return; + const amount = Number.parseFloat(topUpAmount); + if (!Number.isFinite(amount) || amount < 1) { + dispatch(atomicAuthActions.setTopupError("Enter at least $1")); + return; + } + dispatch(atomicAuthActions.setTopupBusy(true)); + dispatch(atomicAuthActions.setTopupError(null)); + try { + const { checkoutUrl } = await atomicBackendApi.createPaygTopup(jwt, { + amountUsd: amount, + successUrl: getStripePaygSuccessUrl(), + cancelUrl: STRIPE_PAYG_CANCEL_URL, + }); + openExternal(checkoutUrl); + dispatch(atomicAuthActions.setTopupPending(true)); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + dispatch(atomicAuthActions.setTopupError(msg)); + } finally { + dispatch(atomicAuthActions.setTopupBusy(false)); + } + }, [jwt, topUpAmount, dispatch]); + + const refreshBalance = React.useCallback(() => { + void dispatch(fetchAtomicBalance({ sync: true })); + }, [dispatch]); + + const signOut = React.useCallback(async () => { + await dispatch(clearAtomicAuthThunk()).unwrap(); + clearAtomicPaygBackup(); + try { + await patchConfig(props.port, { + env: { OPENROUTER_API_KEY: "" }, + }); + } catch (err) { + console.warn( + "[AtomicAccountTab] clearing OPENROUTER_API_KEY failed:", + err, + ); + } + }, [dispatch, props.port]); + + const heroAmount = balanceDepleted ? "$0" : formatDollars(remaining); + const errorMessage = balanceError ?? topupError; + + return ( +
+
+
+

Balance

+
+ +
+ + {heroAmount} + + {/*
+ {balanceLoading ? ( + + + ) : ( + Remaining credits + )} +
*/} +
+ + {balanceDepleted && ( +
+
+
No credits left
+
+ Top up to continue using AI. +
+
+ +
+ )} + + {errorMessage &&
{errorMessage}
} +
+ +
+

One-Time Top-Up

+
+
+ $ + setTopUpAmount(e.target.value)} + min={1} + step={1} + aria-label="Top-up amount in USD" + /> +
+ +
+ {topupPending && ( +
+ Waiting for payment to complete in your browser… + +
+ )} +
+ + {jwt && ( +
+
+ {email ? email.charAt(0).toUpperCase() : "?"} +
+ {email || "—"} + +
+ )} +
+ ); +} diff --git a/desktop/renderer/src/ui/settings/settings-context.tsx b/desktop/renderer/src/ui/settings/settings-context.tsx index 63d2e823b3df..6a7b1b58556e 100644 --- a/desktop/renderer/src/ui/settings/settings-context.tsx +++ b/desktop/renderer/src/ui/settings/settings-context.tsx @@ -70,9 +70,6 @@ export function SettingsStateProvider(props: { checkCapabilities(port).catch(() => null), getConfig(port), ]); - // #region agent log - fetch('http://127.0.0.1:7831/ingest/afbd3787-1f02-4bfb-8a9a-c6c81cb2ee48',{method:'POST',headers:{'Content-Type':'application/json','X-Debug-Session-Id':'bdc941'},body:JSON.stringify({sessionId:'bdc941',hypothesisId:'H4H5',location:'settings-context.tsx:reloadConfig:configLoaded',message:'Config reloaded from server',data:{activeProvider:nextConfig.activeProvider,activeModel:nextConfig.activeModel,hasApiKeys:nextConfig.hasApiKeys,providersCount:nextConfig.providers?.length??0,providersEnvVars:(nextConfig.providers??[]).map(p=>({envVar:p.envVar,configured:p.configured})),hermesHome:nextConfig.hermesHome},timestamp:Date.now()})}).catch(()=>{}); - // #endregion setCapabilities(nextCapabilities); setConfigSnap(nextConfig); setSelectedProvider(nextConfig.activeProvider || PROVIDERS[0]?.id || null); @@ -119,9 +116,6 @@ export function SettingsStateProvider(props: { const saveProviderConfig = React.useCallback(async () => { const provider = getProviderById(selectedProvider); - // #region agent log - fetch('http://127.0.0.1:7831/ingest/afbd3787-1f02-4bfb-8a9a-c6c81cb2ee48',{method:'POST',headers:{'Content-Type':'application/json','X-Debug-Session-Id':'bdc941'},body:JSON.stringify({sessionId:'bdc941',hypothesisId:'H2',location:'settings-context.tsx:saveProviderConfig:entry',message:'saveProviderConfig called',data:{selectedProvider,providerFound:Boolean(provider),providerId:provider?.id,envKey:provider?.envKey,apiKeyLen:apiKey.length,apiKeyTrimmedLen:apiKey.trim().length,baseUrlLen:baseUrl.length,port},timestamp:Date.now()})}).catch(()=>{}); - // #endregion if (!provider) { return false; } @@ -137,20 +131,11 @@ export function SettingsStateProvider(props: { if (provider.envKey && apiKey.trim()) { body.env = { [provider.envKey]: apiKey.trim() }; } - // #region agent log - fetch('http://127.0.0.1:7831/ingest/afbd3787-1f02-4bfb-8a9a-c6c81cb2ee48',{method:'POST',headers:{'Content-Type':'application/json','X-Debug-Session-Id':'bdc941'},body:JSON.stringify({sessionId:'bdc941',hypothesisId:'H3',location:'settings-context.tsx:saveProviderConfig:preRequest',message:'About to PATCH /api/config',data:{bodyConfigKeys:Object.keys(body.config),bodyHasEnv:Boolean(body.env),bodyEnvKeys:body.env?Object.keys(body.env):[],bodyEnvValueLens:body.env?Object.fromEntries(Object.entries(body.env).map(([k,v])=>[k,(v as string).length])):{}},timestamp:Date.now()})}).catch(()=>{}); - // #endregion - const patchResp = await patchConfig(port, body); - // #region agent log - fetch('http://127.0.0.1:7831/ingest/afbd3787-1f02-4bfb-8a9a-c6c81cb2ee48',{method:'POST',headers:{'Content-Type':'application/json','X-Debug-Session-Id':'bdc941'},body:JSON.stringify({sessionId:'bdc941',hypothesisId:'H3',location:'settings-context.tsx:saveProviderConfig:postRequest',message:'PATCH /api/config response',data:{patchResp},timestamp:Date.now()})}).catch(()=>{}); - // #endregion + await patchConfig(port, body); await reloadConfig(); document.dispatchEvent(new Event("hermes-config-changed")); return true; } catch (error) { - // #region agent log - fetch('http://127.0.0.1:7831/ingest/afbd3787-1f02-4bfb-8a9a-c6c81cb2ee48',{method:'POST',headers:{'Content-Type':'application/json','X-Debug-Session-Id':'bdc941'},body:JSON.stringify({sessionId:'bdc941',hypothesisId:'H3',location:'settings-context.tsx:saveProviderConfig:catch',message:'saveProviderConfig caught error',data:{errorMsg:error instanceof Error?error.message:String(error)},timestamp:Date.now()})}).catch(()=>{}); - // #endregion setSaveError(error instanceof Error ? error.message : "Failed to save provider settings."); return false; } finally { diff --git a/desktop/renderer/src/ui/setup/FinishStep.tsx b/desktop/renderer/src/ui/setup/FinishStep.tsx index eb6f5fdc53ee..af15cbe00305 100644 --- a/desktop/renderer/src/ui/setup/FinishStep.tsx +++ b/desktop/renderer/src/ui/setup/FinishStep.tsx @@ -2,7 +2,7 @@ import { PrimaryButton } from "@shared/kit"; import { useOnboardingStepEvent } from "@analytics"; import { OnboardingHeader } from "./OnboardingHeader"; import { useSetup } from "./setup-context"; -import { API_KEYS_FLOW, LOCAL_MODEL_FLOW } from "./onboarding-steps"; +import { API_KEYS_FLOW, ATOMIC_PAYG_FLOW, LOCAL_MODEL_FLOW } from "./onboarding-steps"; import { useNavigate } from "react-router-dom"; import s from "./FinishStep.module.css"; @@ -20,8 +20,20 @@ const CONFETTI_COLORS = [ export function FinishStep() { const navigate = useNavigate(); const { complete, setupFlow } = useSetup(); - useOnboardingStepEvent("finished", setupFlow === "local-model" ? "local-model" : "api-keys"); - const flow = setupFlow === "local-model" ? LOCAL_MODEL_FLOW : API_KEYS_FLOW; + useOnboardingStepEvent( + "finished", + setupFlow === "local-model" + ? "local-model" + : setupFlow === "atomic-payg" + ? "atomic-payg" + : "api-keys", + ); + const flow = + setupFlow === "local-model" + ? LOCAL_MODEL_FLOW + : setupFlow === "atomic-payg" + ? ATOMIC_PAYG_FLOW + : API_KEYS_FLOW; return ( <> @@ -30,7 +42,11 @@ export function FinishStep() { activeStep={flow.steps.finish} onBack={() => void navigate( - setupFlow === "local-model" ? "../local-model-select" : "../model", + setupFlow === "local-model" + ? "../local-model-select" + : setupFlow === "atomic-payg" + ? "../atomic-model" + : "../model", { relative: "path" }, ) } diff --git a/desktop/renderer/src/ui/setup/OnboardingHeader.module.css b/desktop/renderer/src/ui/setup/OnboardingHeader.module.css index ed0d724386f9..f92b3551e112 100644 --- a/desktop/renderer/src/ui/setup/OnboardingHeader.module.css +++ b/desktop/renderer/src/ui/setup/OnboardingHeader.module.css @@ -1,10 +1,17 @@ .header { - width: min(520px, 92vw); display: flex; align-items: center; margin-bottom: 0px; } +.headerWide { + width: min(800px, 92vw); +} + +.headerNarrow { + width: min(520px, 92vw); +} + .side { width: 50px; min-width: 50px; diff --git a/desktop/renderer/src/ui/setup/OnboardingHeader.tsx b/desktop/renderer/src/ui/setup/OnboardingHeader.tsx index 907d39c7cb90..e5881911f1b2 100644 --- a/desktop/renderer/src/ui/setup/OnboardingHeader.tsx +++ b/desktop/renderer/src/ui/setup/OnboardingHeader.tsx @@ -1,5 +1,6 @@ import { OnboardingDots } from "@shared/kit"; import s from "./OnboardingHeader.module.css"; +import { ATOMIC_PAYG_FLOW } from "./onboarding-steps"; export type OnboardingHeaderProps = { totalSteps: number; @@ -16,8 +17,12 @@ export function OnboardingHeader({ onSkip, backDisabled, }: OnboardingHeaderProps) { + const isFirstStep = activeStep === ATOMIC_PAYG_FLOW.steps.setupMode; + return ( -
+
{onBack ? (
+
+ ) : ( + <> + + Secure payment via Stripe + + )} +
+
+
+
+ + {!jwt && ( +
+ + Not signed in — go back and sign in with Google. + +
+ )} + {payError && {payError}} + + + + ); +} diff --git a/desktop/renderer/src/ui/setup/onboarding-steps.ts b/desktop/renderer/src/ui/setup/onboarding-steps.ts index 2a52be82f46a..23c5813fe957 100644 --- a/desktop/renderer/src/ui/setup/onboarding-steps.ts +++ b/desktop/renderer/src/ui/setup/onboarding-steps.ts @@ -1,24 +1,32 @@ /** Step indices for onboarding progress (OnboardingDots), per branch after setup-mode. */ export const API_KEYS_FLOW = { - totalSteps: 6, + totalSteps: 5, steps: { - welcome: 0, - setupMode: 1, - provider: 2, - apiKey: 3, - model: 4, - finish: 5, + setupMode: 0, + provider: 1, + apiKey: 2, + model: 3, + finish: 4, }, } as const; export const LOCAL_MODEL_FLOW = { - totalSteps: 5, + totalSteps: 4, steps: { - welcome: 0, - setupMode: 1, - backendDownload: 2, - modelSelect: 3, - finish: 4, + setupMode: 0, + backendDownload: 1, + modelSelect: 2, + finish: 3, + }, +} as const; + +export const ATOMIC_PAYG_FLOW = { + totalSteps: 4, + steps: { + setupMode: 0, + topup: 1, + model: 2, + finish: 3, }, } as const; diff --git a/desktop/renderer/src/ui/setup/setup-context.ts b/desktop/renderer/src/ui/setup/setup-context.ts index 3fb0683d23d3..21babc9282a1 100644 --- a/desktop/renderer/src/ui/setup/setup-context.ts +++ b/desktop/renderer/src/ui/setup/setup-context.ts @@ -4,7 +4,7 @@ import type { CapabilitiesResponse, DeviceCodeResponse } from "../../services/ap export type OAuthStep = "idle" | "loading" | "waiting" | "success" | "error"; -export type SetupFlowKind = "unset" | "api-keys" | "local-model"; +export type SetupFlowKind = "unset" | "api-keys" | "local-model" | "atomic-payg"; export type SetupState = { port: number; diff --git a/desktop/renderer/src/ui/shared/atomic/AtomicBalanceWidget.module.css b/desktop/renderer/src/ui/shared/atomic/AtomicBalanceWidget.module.css new file mode 100644 index 000000000000..f293112de678 --- /dev/null +++ b/desktop/renderer/src/ui/shared/atomic/AtomicBalanceWidget.module.css @@ -0,0 +1,57 @@ +.widget { + display: flex; + flex-direction: column; + gap: 8px; + padding: 14px 16px; + border: 1px solid var(--border, rgba(255, 255, 255, 0.08)); + border-radius: 10px; + background: var(--surface-overlay, rgba(255, 255, 255, 0.03)); +} + +.row { + display: flex; + justify-content: space-between; + align-items: baseline; + gap: 8px; +} + +.label { + font-size: 12px; + color: var(--muted2, #8e8e8e); +} + +.value { + font-size: 18px; + font-weight: 600; + font-variant-numeric: tabular-nums; +} + +.actions { + display: flex; + align-items: center; + gap: 12px; + margin-top: 4px; +} + +.refresh { + background: none; + border: none; + color: var(--muted2, #8e8e8e); + cursor: pointer; + font-size: 12px; + padding: 0; +} + +.refresh:hover:not(:disabled) { + color: var(--text, #fff); +} + +.refresh:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.error { + font-size: 12px; + color: var(--danger, #ff6b6b); +} diff --git a/desktop/renderer/src/ui/shared/atomic/AtomicBalanceWidget.tsx b/desktop/renderer/src/ui/shared/atomic/AtomicBalanceWidget.tsx new file mode 100644 index 000000000000..e50471f14c23 --- /dev/null +++ b/desktop/renderer/src/ui/shared/atomic/AtomicBalanceWidget.tsx @@ -0,0 +1,68 @@ +import React from "react"; +import { PrimaryButton } from "@shared/kit"; +import { useAppDispatch, useAppSelector } from "@store/hooks"; +import { fetchAtomicBalance } from "@store/slices/atomicAuthSlice"; +import { PaygTopUpDialog } from "./PaygTopUpDialog"; +import s from "./AtomicBalanceWidget.module.css"; + +function formatUsd(value: number | null | undefined): string { + if (value == null || !Number.isFinite(value)) return "—"; + return `$${value.toFixed(2)}`; +} + +export function AtomicBalanceWidget(props: { onTopUpClick?: () => void }) { + const dispatch = useAppDispatch(); + const balance = useAppSelector((state) => state.atomicAuth.balance); + const loading = useAppSelector((state) => state.atomicAuth.balanceLoading); + const error = useAppSelector((state) => state.atomicAuth.balanceError); + const jwt = useAppSelector((state) => state.atomicAuth.jwt); + + const [topupOpen, setTopupOpen] = React.useState(false); + + React.useEffect(() => { + if (jwt && !balance && !loading) { + void dispatch(fetchAtomicBalance({})); + } + }, [jwt, balance, loading, dispatch]); + + const handleTopUp = React.useCallback(() => { + if (props.onTopUpClick) { + props.onTopUpClick(); + return; + } + setTopupOpen(true); + }, [props]); + + if (!jwt) return null; + + return ( +
+
+ PAYG balance + {formatUsd(balance?.payg?.remaining ?? null)} +
+ {balance?.subscription && ( +
+ Pay as you go credits + {formatUsd(balance.subscription.remaining)} +
+ )} + {error &&
{error}
} +
+ + Top up + + +
+ + setTopupOpen(false)} /> +
+ ); +} diff --git a/desktop/renderer/src/ui/shared/atomic/AtomicSignInPrompt.module.css b/desktop/renderer/src/ui/shared/atomic/AtomicSignInPrompt.module.css new file mode 100644 index 000000000000..97454b49f9de --- /dev/null +++ b/desktop/renderer/src/ui/shared/atomic/AtomicSignInPrompt.module.css @@ -0,0 +1,125 @@ +/* ── AtomicSignInPrompt ── */ + +.root { + display: flex; + flex-direction: column; + gap: 20px; + animation: fadeIn ease 0.2s; +} + +@keyframes fadeIn { + 0% { + opacity: 0; + } + 100% { + opacity: 1; + } +} + +.signUpCard { + display: flex; + flex-direction: row; + align-items: center; + gap: 14px; + padding: 16px; + border-radius: var(--radius-lg, 14px); + background: rgba(255, 255, 255, 0.04); + border: 1px solid rgba(255, 255, 255, 0.08); +} + +.signUpIcon { + display: flex; + align-items: center; + justify-content: center; + width: 40px; + height: 40px; + border-radius: 10px; + background: rgba(255, 255, 255, 0.06); + flex-shrink: 0; +} + +.signUpIcon img { + object-fit: contain; +} + +.signUpBody { + flex: 1; + min-width: 0; + display: flex; + flex-direction: column; + gap: 4px; +} + +.signUpTitle { + font-size: 14px; + font-weight: 650; + line-height: 20px; + margin: 0; + color: var(--text, #fff); +} + +.signUpHint { + font-size: 12px; + color: rgba(255, 255, 255, 0.4); + line-height: 16px; + margin: 0; +} + +.signUpButton { + display: inline-flex; + align-items: center; + justify-content: center; + gap: 8px; + align-self: center; + padding: 9px 20px; + border: none; + border-radius: var(--radius-full, 999px); + background: var(--text, #fff); + color: var(--surface-primary, #121212); + font-size: 13px; + font-weight: 600; + cursor: pointer; + transition: opacity 140ms ease; + white-space: nowrap; + flex-shrink: 0; +} + +.signUpButton:hover { + opacity: 0.9; +} + +.signUpButton:active { + opacity: 0.75; +} + +.signUpButton:disabled { + opacity: 0.7; + cursor: default; +} + +.googleIcon { + flex-shrink: 0; + width: 16px; + height: 16px; +} + +.buttonSpinner { + display: inline-block; + width: 14px; + height: 14px; + border: 2px solid rgba(18, 18, 18, 0.2); + border-top-color: var(--surface-primary, #121212); + border-radius: 50%; + animation: signInSpin 0.7s linear infinite; + flex-shrink: 0; +} + +@keyframes signInSpin { + to { + transform: rotate(360deg); + } +} + +.errorRow { + margin-top: 6px; +} diff --git a/desktop/renderer/src/ui/shared/atomic/AtomicSignInPrompt.tsx b/desktop/renderer/src/ui/shared/atomic/AtomicSignInPrompt.tsx new file mode 100644 index 000000000000..b00cef48bdcf --- /dev/null +++ b/desktop/renderer/src/ui/shared/atomic/AtomicSignInPrompt.tsx @@ -0,0 +1,166 @@ +import React from "react"; +import { InlineError, SplashLogo } from "@shared/kit"; +import { useAppDispatch, useAppSelector } from "@store/hooks"; +import { + applyPaygKey, + storeAtomicToken, +} from "@store/slices/atomicAuthSlice"; +import { googleAuthDesktopUrl } from "../../../services/atomic-backend-api"; +import { useAtomicDeepLink } from "../../../hooks/useAtomicDeepLink"; +import s from "./AtomicSignInPrompt.module.css"; + +const GOOGLE_ICON = new URL( + "../../../../../assets/set-up-skills/Google.svg", + import.meta.url, +).toString(); + +type Phase = "idle" | "waiting" | "authenticated" | "applying" | "error"; + +function openExternal(url: string): void { + const api = (window as { hermesAPI?: { openExternal?: (u: string) => void } }) + .hermesAPI; + if (api?.openExternal) { + void api.openExternal(url); + return; + } + window.open(url, "_blank"); +} + +// `createAsyncThunk` rethrows non-`rejectWithValue` errors as a plain +// `SerializedError` object, not an `Error` instance — collapse to a string. +function describeError(err: unknown): string { + if (err instanceof Error) return err.message; + if (typeof err === "string") return err; + if (err && typeof err === "object") { + const candidate = (err as { message?: unknown }).message; + if (typeof candidate === "string" && candidate.length > 0) return candidate; + } + return "Unexpected error"; +} + +export function AtomicSignInPrompt(props: { + port: number; + title?: string; + hint?: string; +}) { + const dispatch = useAppDispatch(); + const jwt = useAppSelector((state) => state.atomicAuth.jwt); + const applyKeyBusy = useAppSelector((state) => state.atomicAuth.applyKeyBusy); + const applyKeyError = useAppSelector( + (state) => state.atomicAuth.applyKeyError, + ); + + const [phase, setPhase] = React.useState(jwt ? "authenticated" : "idle"); + const [localError, setLocalError] = React.useState(null); + + const title = props.title ?? "Sign in with Atomic"; + const hint = props.hint ?? "Use OpenRouter on a pay-as-you-go plan."; + + const startSignIn = React.useCallback(() => { + setLocalError(null); + setPhase("waiting"); + openExternal(googleAuthDesktopUrl()); + }, []); + + useAtomicDeepLink({ + onAuth: (params) => { + void (async () => { + try { + await dispatch( + storeAtomicToken({ + jwt: params.jwt, + email: params.email, + userId: params.userId, + isNewUser: params.isNewUser, + }), + ).unwrap(); + setPhase("authenticated"); + } catch (err) { + console.error("[AtomicSignInPrompt] storeAtomicToken failed:", err); + setLocalError(describeError(err)); + setPhase("error"); + } + })(); + }, + onAuthError: () => { + setLocalError("Authentication failed — missing token data"); + setPhase("error"); + }, + }); + + // Once we have a JWT, push the PAYG key into the local gateway. Runs once + // per `jwt` value to avoid loops. + const appliedRef = React.useRef(null); + React.useEffect(() => { + if (!jwt || phase !== "authenticated") return; + if (appliedRef.current === jwt) return; + appliedRef.current = jwt; + + setPhase("applying"); + void (async () => { + try { + await dispatch(applyPaygKey({ port: props.port })).unwrap(); + setPhase("idle"); + } catch (err) { + console.error("[AtomicSignInPrompt] applyPaygKey failed:", err); + setLocalError(describeError(err)); + setPhase("error"); + } + })(); + }, [jwt, phase, dispatch, props.port]); + + const errorMessage = localError ?? applyKeyError ?? null; + const showSpinner = + phase === "waiting" || phase === "applying" || applyKeyBusy; + + return ( +
+
+
+ +
+ +
+

{title}

+

{hint}

+ {errorMessage ? ( +
+ {errorMessage} +
+ ) : null} +
+ + +
+
+ ); +} diff --git a/desktop/renderer/src/ui/shared/atomic/LowBalanceBanner.module.css b/desktop/renderer/src/ui/shared/atomic/LowBalanceBanner.module.css new file mode 100644 index 000000000000..14487174b32b --- /dev/null +++ b/desktop/renderer/src/ui/shared/atomic/LowBalanceBanner.module.css @@ -0,0 +1,134 @@ +/* ── LowBalanceBanner ── */ + +.banner { + position: fixed; + top: 12px; + left: 50%; + transform: translateX(-50%); + z-index: 9998; + display: flex; + align-items: center; + gap: 12px; + padding: 12px 14px; + border-radius: var(--radius-xl); + background: + linear-gradient(var(--bg-card), var(--bg-card)) padding-box, + linear-gradient( + 150deg, + rgba(255, 255, 255, 0.18) 0%, + rgba(255, 255, 255, 0) 50%, + rgba(255, 255, 255, 0.18) 100% + ) + border-box; + border: 1px solid transparent; + backdrop-filter: blur(16px); + box-shadow: 0 12px 40px rgba(0, 0, 0, 0.5); + min-width: 360px; + max-width: 480px; + color: var(--text); + font-size: 13px; + overflow: hidden; + user-select: none; + animation: bannerSlideIn 300ms cubic-bezier(0.2, 0.9, 0.2, 1); + pointer-events: auto; +} + +@keyframes bannerSlideIn { + from { + opacity: 0; + transform: translateX(-50%) translateY(-16px) scale(0.96); + } + to { + opacity: 1; + transform: translateX(-50%); + } +} + +.icon { + display: flex; + align-items: center; + justify-content: center; + flex-shrink: 0; + width: 38px; + height: 38px; + border-radius: 10px; + background: rgba(239, 68, 68, 0.12); + color: var(--error); +} + +.body { + flex: 1; + min-width: 0; + display: flex; + flex-direction: column; + gap: 2px; +} + +.title { + font-size: 13px; + font-weight: 600; + color: var(--text); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.subtitle { + font-size: 12px; + color: var(--muted2); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.action { + appearance: none; + border: none; + border-radius: var(--radius-full); + padding: 7px 16px; + font-size: 12px; + font-weight: 650; + cursor: pointer; + white-space: nowrap; + flex-shrink: 0; + background: var(--lime); + color: var(--surface-primary); + transition: opacity 150ms ease; +} + +.action:hover { + opacity: 0.85; +} + +.action:active { + opacity: 0.7; +} + +.dismiss { + width: 26px; + height: 26px; + display: flex; + align-items: center; + justify-content: center; + appearance: none; + border: none; + background: transparent; + color: var(--muted2); + border-radius: 8px; + cursor: pointer; + flex-shrink: 0; + transition: + color 150ms ease, + background 150ms ease; +} + +.dismiss:hover { + color: var(--text); + background: rgba(255, 255, 255, 0.06); +} + +@media (prefers-reduced-motion: reduce) { + .banner { + animation: none; + } +} diff --git a/desktop/renderer/src/ui/shared/atomic/LowBalanceBanner.tsx b/desktop/renderer/src/ui/shared/atomic/LowBalanceBanner.tsx new file mode 100644 index 000000000000..e0a05d67adbf --- /dev/null +++ b/desktop/renderer/src/ui/shared/atomic/LowBalanceBanner.tsx @@ -0,0 +1,141 @@ +import React from "react"; +import { useAppDispatch, useAppSelector } from "@store/hooks"; +import { + atomicAuthActions, + fetchAtomicBalance, +} from "@store/slices/atomicAuthSlice"; +import { + STRIPE_PAYG_CANCEL_URL, + atomicBackendApi, + getStripePaygSuccessUrl, +} from "../../../services/atomic-backend-api"; +import s from "./LowBalanceBanner.module.css"; + +const DEFAULT_THRESHOLD_USD = 1; +const SESSION_DISMISS_KEY = "hermes:low-balance-banner-dismissed"; +const DEFAULT_TOPUP_USD = 10; + +function openExternal(url: string): void { + const api = (window as { hermesAPI?: { openExternal?: (u: string) => void } }).hermesAPI; + if (api?.openExternal) { + void api.openExternal(url); + return; + } + window.open(url, "_blank"); +} + +function EmptyWalletIcon() { + return ( + + + + ); +} + +function CloseIcon() { + return ( + + + + ); +} + +/** + * Floating pill-shaped banner shown when the user's PAYG pool drops below + * `thresholdUsd`. Only visible in atomic-payg mode. Dismiss is per-session + * (sessionStorage), reappears on next app launch if balance is still low. + */ +export function LowBalanceBanner(props: { thresholdUsd?: number }) { + const dispatch = useAppDispatch(); + const jwt = useAppSelector((state) => state.atomicAuth.jwt); + const mode = useAppSelector((state) => state.config.mode); + const balance = useAppSelector((state) => state.atomicAuth.balance); + + const [dismissed, setDismissed] = React.useState(() => { + try { + return sessionStorage.getItem(SESSION_DISMISS_KEY) === "1"; + } catch { + return false; + } + }); + + React.useEffect(() => { + if (jwt && mode === "atomic-payg" && !balance) { + void dispatch(fetchAtomicBalance({})); + } + }, [jwt, mode, balance, dispatch]); + + const handleTopUp = React.useCallback(async () => { + if (!jwt) return; + dispatch(atomicAuthActions.setTopupBusy(true)); + dispatch(atomicAuthActions.setTopupError(null)); + try { + const { checkoutUrl } = await atomicBackendApi.createPaygTopup(jwt, { + amountUsd: DEFAULT_TOPUP_USD, + successUrl: getStripePaygSuccessUrl(), + cancelUrl: STRIPE_PAYG_CANCEL_URL, + }); + openExternal(checkoutUrl); + dispatch(atomicAuthActions.setTopupPending(true)); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + dispatch(atomicAuthActions.setTopupError(msg)); + } finally { + dispatch(atomicAuthActions.setTopupBusy(false)); + } + }, [jwt, dispatch]); + + const handleDismiss = React.useCallback(() => { + try { + sessionStorage.setItem(SESSION_DISMISS_KEY, "1"); + } catch { + /* sessionStorage unavailable */ + } + setDismissed(true); + }, []); + + if (mode !== "atomic-payg" || !jwt || dismissed) return null; + const remaining = balance?.payg?.remaining; + if (remaining == null) return null; + + const threshold = props.thresholdUsd ?? DEFAULT_THRESHOLD_USD; + if (remaining > threshold) return null; + + return ( +
+
+ +
+
+
No credits left
+
Top up to continue using AI.
+
+ + +
+ ); +} diff --git a/desktop/renderer/src/ui/shared/atomic/PaygTopUpDialog.tsx b/desktop/renderer/src/ui/shared/atomic/PaygTopUpDialog.tsx new file mode 100644 index 000000000000..02e5664c167c --- /dev/null +++ b/desktop/renderer/src/ui/shared/atomic/PaygTopUpDialog.tsx @@ -0,0 +1,156 @@ +import React from "react"; +import { + InlineError, + Modal, + PrimaryButton, + SecondaryButton, + TextInput, +} from "@shared/kit"; +import { useAppDispatch, useAppSelector } from "@store/hooks"; +import { + atomicAuthActions, + fetchAtomicBalance, +} from "@store/slices/atomicAuthSlice"; +import { + STRIPE_PAYG_CANCEL_URL, + atomicBackendApi, + getStripePaygSuccessUrl, +} from "../../../services/atomic-backend-api"; + +const PRESET_AMOUNTS_USD = [5, 10, 25, 50]; + +function openExternal(url: string): void { + const api = (window as { hermesAPI?: { openExternal?: (u: string) => void } }) + .hermesAPI; + if (api?.openExternal) { + void api.openExternal(url); + return; + } + window.open(url, "_blank"); +} + +export function PaygTopUpDialog(props: { + open: boolean; + onClose: () => void; + initialAmountUsd?: number; +}) { + const dispatch = useAppDispatch(); + const jwt = useAppSelector((state) => state.atomicAuth.jwt); + const topupBusy = useAppSelector((state) => state.atomicAuth.topupBusy); + const topupError = useAppSelector((state) => state.atomicAuth.topupError); + const topupPending = useAppSelector((state) => state.atomicAuth.topupPending); + + const [amount, setAmount] = React.useState( + String(props.initialAmountUsd ?? 10), + ); + const [localError, setLocalError] = React.useState(null); + + React.useEffect(() => { + if (props.open) { + setAmount(String(props.initialAmountUsd ?? 10)); + setLocalError(null); + dispatch(atomicAuthActions.setTopupError(null)); + } + }, [props.open, props.initialAmountUsd, dispatch]); + + const parsedAmount = Number.parseFloat(amount); + const amountValid = Number.isFinite(parsedAmount) && parsedAmount >= 1; + + const startCheckout = React.useCallback(async () => { + if (!jwt) { + setLocalError("Not signed in"); + return; + } + if (!amountValid) { + setLocalError("Enter at least $1"); + return; + } + setLocalError(null); + dispatch(atomicAuthActions.setTopupBusy(true)); + dispatch(atomicAuthActions.setTopupError(null)); + try { + const { checkoutUrl } = await atomicBackendApi.createPaygTopup(jwt, { + amountUsd: parsedAmount, + successUrl: getStripePaygSuccessUrl(), + cancelUrl: STRIPE_PAYG_CANCEL_URL, + }); + openExternal(checkoutUrl); + dispatch(atomicAuthActions.setTopupPending(true)); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + dispatch(atomicAuthActions.setTopupError(msg)); + } finally { + dispatch(atomicAuthActions.setTopupBusy(false)); + } + }, [jwt, amount, amountValid, parsedAmount, dispatch]); + + const refreshBalance = React.useCallback(() => { + void dispatch(fetchAtomicBalance({ sync: true })); + }, [dispatch]); + + const errorMessage = localError ?? topupError; + + return ( + +

+ Add credits to your pay-as-you-go pool. We’ll open Stripe Checkout + in your browser; come back here when payment is complete. +

+ +
+ {PRESET_AMOUNTS_USD.map((preset) => ( + setAmount(String(preset))} + > + ${preset} + + ))} +
+ + setAmount(v.replace(/[^0-9.]/g, ""))} + placeholder="10" + /> + + {errorMessage && {errorMessage}} + + {topupPending && ( +
+ Waiting for payment to complete in your browser… + +
+ )} + +
+ + Close + + void startCheckout()} + > + Open Stripe Checkout + +
+ + ); +} diff --git a/desktop/renderer/src/ui/shared/atomic/useAtomicAuthBootstrap.ts b/desktop/renderer/src/ui/shared/atomic/useAtomicAuthBootstrap.ts new file mode 100644 index 000000000000..cb4a374ece1e --- /dev/null +++ b/desktop/renderer/src/ui/shared/atomic/useAtomicAuthBootstrap.ts @@ -0,0 +1,28 @@ +import React from "react"; +import { useAppDispatch, useAppSelector } from "@store/hooks"; +import { + fetchAtomicBalance, + restoreAtomicAuth, + verifyAtomicAuth, +} from "@store/slices/atomicAuthSlice"; + +/** + * App-level bootstrap for the Atomic auth slice. Runs once: restores the + * persisted JWT from the main process, then verifies it and fetches the + * current balance. Called from the root component. + */ +export function useAtomicAuthBootstrap(): void { + const dispatch = useAppDispatch(); + const restoreLoaded = useAppSelector((state) => state.atomicAuth.restoreLoaded); + const jwt = useAppSelector((state) => state.atomicAuth.jwt); + + React.useEffect(() => { + void dispatch(restoreAtomicAuth()); + }, [dispatch]); + + React.useEffect(() => { + if (!restoreLoaded || !jwt) return; + void dispatch(verifyAtomicAuth()); + void dispatch(fetchAtomicBalance({})); + }, [restoreLoaded, jwt, dispatch]); +} diff --git a/desktop/renderer/vite.config.ts b/desktop/renderer/vite.config.ts index 1f0560165245..66aa3198b6c7 100644 --- a/desktop/renderer/vite.config.ts +++ b/desktop/renderer/vite.config.ts @@ -3,25 +3,37 @@ import react from "@vitejs/plugin-react"; import path from "node:path"; import fs from "node:fs"; -function resolvePosthogKey(): string { - if (process.env.POSTHOG_API_KEY) return process.env.POSTHOG_API_KEY; +function readDotEnvValue(name: string): string { try { const envFile = path.resolve(__dirname, "..", ".env"); if (!fs.existsSync(envFile)) return ""; for (const line of fs.readFileSync(envFile, "utf-8").split("\n")) { - const m = line.match(/^\s*(?:VITE_)?POSTHOG_API_KEY=(.+)$/); + const m = line.match(new RegExp(`^\\s*(?:VITE_)?${name}=(.+)$`)); if (m) return m[1].trim().replace(/^["']|["']$/g, ""); } } catch { /* best-effort */ } return ""; } +function resolvePosthogKey(): string { + if (process.env.POSTHOG_API_KEY) return process.env.POSTHOG_API_KEY; + return readDotEnvValue("POSTHOG_API_KEY"); +} + +function resolveAtomicBackendUrl(): string { + if (process.env.VITE_ATOMIC_BACKEND_URL) return process.env.VITE_ATOMIC_BACKEND_URL; + return readDotEnvValue("ATOMIC_BACKEND_URL"); +} + export default defineConfig({ root: path.resolve(__dirname), plugins: [react()], base: "./", define: { __POSTHOG_API_KEY__: JSON.stringify(resolvePosthogKey()), + "import.meta.env.VITE_ATOMIC_BACKEND_URL": JSON.stringify( + resolveAtomicBackendUrl(), + ), }, css: { modules: { diff --git a/desktop/scripts/patch-electron-bundle-id.cjs b/desktop/scripts/patch-electron-bundle-id.cjs new file mode 100644 index 000000000000..89f26fae45b8 --- /dev/null +++ b/desktop/scripts/patch-electron-bundle-id.cjs @@ -0,0 +1,99 @@ +#!/usr/bin/env node +/* eslint-disable no-console */ +/** + * Patch Electron.app's CFBundleIdentifier so macOS LaunchServices can + * uniquely associate `atomicbot-hermes://` deep-links with our dev binary + * instead of routing them to any other Electron-based app installed on the + * machine (e.g. openclaw). All Electron dev binaries ship with the same + * bundle id `com.github.electron`, which makes deep-link routing + * non-deterministic across multiple Electron projects on the same OS. + * + * The script is idempotent and a no-op on non-macOS platforms. + * + * After patching the Info.plist, we ad-hoc re-sign Electron.app (otherwise + * macOS rejects the modified bundle) and force LaunchServices to refresh + * its registration so the next `setAsDefaultProtocolClient` call points to + * the new bundle id. + */ + +const fs = require("fs"); +const path = require("path"); +const { execSync } = require("child_process"); + +const NEW_BUNDLE_ID = "ai.atomicbot.hermes.dev"; +const ORIGINAL_BUNDLE_ID = "com.github.electron"; + +const ELECTRON_APP = path.resolve( + __dirname, + "..", + "node_modules", + "electron", + "dist", + "Electron.app", +); +const INFO_PLIST = path.join(ELECTRON_APP, "Contents", "Info.plist"); + +function log(message) { + console.log(`[patch-electron-bundle-id] ${message}`); +} + +function warn(message) { + console.warn(`[patch-electron-bundle-id] ${message}`); +} + +function main() { + if (process.platform !== "darwin") { + log("Skipping: macOS only"); + return; + } + + if (!fs.existsSync(INFO_PLIST)) { + log(`Skipping: ${INFO_PLIST} not found (electron not installed yet?)`); + return; + } + + const original = fs.readFileSync(INFO_PLIST, "utf8"); + + if (original.includes(`${NEW_BUNDLE_ID}`)) { + log(`Already patched (CFBundleIdentifier = ${NEW_BUNDLE_ID})`); + return; + } + + const patched = original.replace( + /(CFBundleIdentifier<\/key>\s*)[^<]+(<\/string>)/, + `$1${NEW_BUNDLE_ID}$2`, + ); + + if (patched === original) { + console.error( + "[patch-electron-bundle-id] FAILED: CFBundleIdentifier key not found in Info.plist", + ); + process.exit(1); + } + + fs.writeFileSync(INFO_PLIST, patched); + log(`Patched CFBundleIdentifier: ${ORIGINAL_BUNDLE_ID} -> ${NEW_BUNDLE_ID}`); + + try { + execSync(`codesign --force --deep --sign - "${ELECTRON_APP}"`, { + stdio: "inherit", + }); + log("Ad-hoc re-signed Electron.app"); + } catch (e) { + warn(`codesign failed: ${e.message}. Electron may refuse to launch.`); + } + + try { + execSync( + `/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -f "${ELECTRON_APP}"`, + { stdio: "inherit" }, + ); + log("Re-registered Electron.app with LaunchServices"); + } catch (e) { + warn(`lsregister refresh failed: ${e.message}`); + } + + log("Done. macOS will now route atomicbot-hermes:// only to our Electron binary."); +} + +main(); diff --git a/desktop/src/main/atomic-auth/deep-link.ts b/desktop/src/main/atomic-auth/deep-link.ts new file mode 100644 index 000000000000..fe5577c8e7a1 --- /dev/null +++ b/desktop/src/main/atomic-auth/deep-link.ts @@ -0,0 +1,33 @@ +import type { BrowserWindow } from "electron"; + +export type DeepLinkPayload = { + host: string; + pathname: string; + params: Record; +}; + +export function parseDeepLinkUrl(url: string): DeepLinkPayload | null { + try { + const parsed = new URL(url); + return { + host: parsed.host, + pathname: parsed.pathname, + params: Object.fromEntries(parsed.searchParams.entries()), + }; + } catch { + return null; + } +} + +export const DEEP_LINK_IPC_CHANNEL = "atomic:deep-link"; + +export function handleDeepLink(url: string, win: BrowserWindow | null): void { + const payload = parseDeepLinkUrl(url); + if (!payload) { + console.warn("[main/deep-link] failed to parse URL:", url); + return; + } + if (win && !win.isDestroyed()) { + win.webContents.send(DEEP_LINK_IPC_CHANNEL, payload); + } +} diff --git a/desktop/src/main/atomic-auth/stripe-thanks-server.ts b/desktop/src/main/atomic-auth/stripe-thanks-server.ts new file mode 100644 index 000000000000..4cb4a21fcc95 --- /dev/null +++ b/desktop/src/main/atomic-auth/stripe-thanks-server.ts @@ -0,0 +1,197 @@ +import * as http from "node:http"; + +/** + * Fixed local port for the post-Stripe-Checkout "thank-you" page. Stripe + * requires a valid HTTP(S) `success_url` and silently rewrites unknown + * URI schemes (e.g. `atomicbot-hermes://`) to `https://`, breaking the + * deep-link return path. Routing through `http://localhost:` lets + * us serve a small landing page that triggers the proper deep link + * (atomicbot-hermes://stripe-success?session_id=…) and brings focus back + * to the Electron app. + * + * The port is fixed (not random) so that a checkout session created in + * one process lifetime still resolves after an app restart. + */ +export const STRIPE_THANKS_PORT = 27871; + +export const STRIPE_THANKS_PATH = "/stripe-thanks"; + +const DEEP_LINK_SCHEME = "atomicbot-hermes"; + +/** Public URL we hand to Stripe as `success_url`. */ +export function getStripeThanksUrl(): string { + return `http://localhost:${STRIPE_THANKS_PORT}${STRIPE_THANKS_PATH}?session_id={CHECKOUT_SESSION_ID}`; +} + +function escapeAttr(value: string): string { + return value + .replace(/&/g, "&") + .replace(/"/g, """) + .replace(/'/g, "'") + .replace(//g, ">"); +} + +function renderThanksPage(deepLinkUrl: string): string { + const safeDeepLink = escapeAttr(deepLinkUrl); + return ` + + + + + Payment successful + + + +
+ +

Payment successful

+

Returning you to Atomic Hermes…

+
You can close this tab.
+
+ + +`; +} + +export type StripeThanksServer = { + stop: () => Promise; +}; + +/** + * Starts a localhost-only HTTP server that serves the post-checkout + * landing page on `STRIPE_THANKS_PORT`. Idempotent: returns null if the + * port is busy (another Hermes instance, port collision) — callers should + * fall back to the renderer's polling-only flow. + */ +export function startStripeThanksServer(): Promise { + return new Promise((resolve) => { + const server = http.createServer((req, res) => { + try { + if (!req.url) { + res.writeHead(404).end(); + return; + } + const url = new URL(req.url, `http://localhost:${STRIPE_THANKS_PORT}`); + if (url.pathname !== STRIPE_THANKS_PATH) { + res.writeHead(404, { "Content-Type": "text/plain" }); + res.end("Not Found"); + return; + } + const sessionId = url.searchParams.get("session_id") ?? ""; + const deepLinkUrl = sessionId + ? `${DEEP_LINK_SCHEME}://stripe-success?session_id=${encodeURIComponent(sessionId)}` + : `${DEEP_LINK_SCHEME}://stripe-success`; + const html = renderThanksPage(deepLinkUrl); + res.writeHead(200, { + "Content-Type": "text/html; charset=utf-8", + "Cache-Control": "no-store", + "X-Content-Type-Options": "nosniff", + }); + res.end(html); + } catch (err) { + console.warn("[stripe-thanks-server] handler error:", err); + if (!res.headersSent) { + res.writeHead(500, { "Content-Type": "text/plain" }); + } + res.end(); + } + }); + + let resolved = false; + + server.once("error", (err) => { + console.warn( + `[stripe-thanks-server] failed to bind :${STRIPE_THANKS_PORT}:`, + err, + ); + if (!resolved) { + resolved = true; + resolve(null); + } + }); + + server.listen(STRIPE_THANKS_PORT, "127.0.0.1", () => { + if (resolved) return; + resolved = true; + console.log( + `[stripe-thanks-server] listening on http://127.0.0.1:${STRIPE_THANKS_PORT}${STRIPE_THANKS_PATH}`, + ); + resolve({ + stop: () => + new Promise((stopResolve) => { + server.close(() => stopResolve()); + }), + }); + }); + }); +} diff --git a/desktop/src/main/main.ts b/desktop/src/main/main.ts index f112a0cae6ae..e7c61c9f7e86 100644 --- a/desktop/src/main/main.ts +++ b/desktop/src/main/main.ts @@ -1,4 +1,11 @@ -import { app, BrowserWindow, ipcMain, Notification, session, shell } from "electron"; +import { + app, + BrowserWindow, + ipcMain, + Notification, + session, + shell, +} from "electron"; import * as fs from "fs"; import * as path from "path"; import { startPythonBackend, PythonBridge } from "./python-bridge"; @@ -22,27 +29,87 @@ import { getAppVersion, } from "./updater"; import { registerLlamacppIpcHandlers } from "./llamacpp/ipc"; -import { stopLlamacppServer, killOrphanedServer, startLlamacppServer } from "./llamacpp/server"; +import { + stopLlamacppServer, + killOrphanedServer, + startLlamacppServer, +} from "./llamacpp/server"; import { readActiveModelId } from "./llamacpp/model-state"; -import { getLlamacppModelDef, resolveLlamacppModelPath, resolveChatTemplatePath, type LlamacppModelId } from "./llamacpp/models"; -import { isBackendDownloaded, resolveServerBinPath } from "./llamacpp/backend-download"; +import { + getLlamacppModelDef, + resolveLlamacppModelPath, + resolveChatTemplatePath, + type LlamacppModelId, +} from "./llamacpp/models"; +import { + isBackendDownloaded, + resolveServerBinPath, +} from "./llamacpp/backend-download"; import { getSystemInfo, computeContextLength } from "./llamacpp/hardware"; import { isAnyProfileUsingLlamacpp } from "./llamacpp/profile-usage"; import { killUpdateSplash } from "./update-splash"; -import { readAnalyticsState, writeAnalyticsState } from "./analytics/analytics-state"; -import { initPosthogMain, captureMain, shutdownPosthogMain } from "./analytics/posthog-main"; +import { + readAnalyticsState, + writeAnalyticsState, +} from "./analytics/analytics-state"; +import { + initPosthogMain, + captureMain, + shutdownPosthogMain, +} from "./analytics/posthog-main"; import { registerAnalyticsHandlers } from "./analytics/analytics-ipc"; import { registerNotificationsHandlers, isNotificationsEnabled, } from "./notifications"; +import { handleDeepLink } from "./atomic-auth/deep-link"; +import { + startStripeThanksServer, + type StripeThanksServer, +} from "./atomic-auth/stripe-thanks-server"; + +const DEEP_LINK_PROTOCOL = "atomicbot-hermes"; + +app.setPath( + "userData", + path.join(app.getPath("appData"), "ai.atomicbot.hermes"), +); -app.setPath("userData", path.join(app.getPath("appData"), "ai.atomicbot.hermes")); +// Single-instance lock + deep-link protocol registration must happen before +// `app.whenReady()`. When a second instance is launched (e.g. by macOS opening +// an `atomicbot-hermes://...` URL while the app is already running), we forward +// the URL to the existing main window via `second-instance` / `open-url` +// events. Note: the scheme is `atomicbot-hermes://` (not `atomicbot://`) to +// avoid colliding with the openclaw desktop client which uses `atomicbot://`. +// The backend allowlist (`ALLOWED_DESKTOP_SCHEMES`) must include this exact +// value, otherwise OAuth/Stripe deep links fall back to `atomicbot://`. +const gotSingleInstanceLock = app.requestSingleInstanceLock(); +if (!gotSingleInstanceLock) { + app.quit(); +} + +if (process.defaultApp) { + if (process.argv.length >= 2) { + app.setAsDefaultProtocolClient(DEEP_LINK_PROTOCOL, process.execPath, [ + path.resolve(process.argv[1]!), + ]); + } +} else { + app.setAsDefaultProtocolClient(DEEP_LINK_PROTOCOL); +} let mainWindow: BrowserWindow | null = null; let pythonBridge: PythonBridge | null = null; let backendPort: number | null = null; let snapshotWatcher: SnapshotWatcher | null = null; +let stripeThanksServer: StripeThanksServer | null = null; + +function focusMainWindow(): void { + if (!mainWindow || mainWindow.isDestroyed()) return; + if (mainWindow.isMinimized()) mainWindow.restore(); + mainWindow.show(); + mainWindow.focus(); +} type DashboardState = | { kind: "starting" } @@ -57,8 +124,8 @@ function createWindow(): void { mainWindow = new BrowserWindow({ width: 960, height: 720, - minWidth: 480, - minHeight: 400, + minWidth: 900, + minHeight: 600, title: windowTitle, backgroundColor: "#1a1a2e", webPreferences: { @@ -73,7 +140,14 @@ function createWindow(): void { event.preventDefault(); }); - const rendererPath = path.join(__dirname, "..", "..", "renderer", "dist", "index.html"); + const rendererPath = path.join( + __dirname, + "..", + "..", + "renderer", + "dist", + "index.html", + ); mainWindow.loadFile(rendererPath); @@ -90,7 +164,8 @@ const analyticsState = readAnalyticsState(stateDir); if (!analyticsState.prompted) { analyticsState.enabled = true; analyticsState.prompted = true; - analyticsState.enabledAt = analyticsState.enabledAt ?? new Date().toISOString(); + analyticsState.enabledAt = + analyticsState.enabledAt ?? new Date().toISOString(); writeAnalyticsState(stateDir, analyticsState); } initPosthogMain(analyticsState.userId, analyticsState.enabled); @@ -102,6 +177,28 @@ captureMain("app_launched", { registerAnalyticsHandlers({ stateDir }); registerNotificationsHandlers({ stateDir }); +// Deep link delivery (atomicbot-hermes://...) — macOS uses `open-url`, +// Windows/Linux receive the URL as an argv entry on a second-instance launch. +app.on("open-url", (event, url) => { + event.preventDefault(); + handleDeepLink(url, mainWindow); + // macOS delivers deep links via `open-url` while the app is already + // running; bring the window forward so the user lands back in Hermes + // instead of Stripe's tab. + focusMainWindow(); +}); + +app.on("second-instance", (_event, argv) => { + const url = argv.find( + (arg) => + typeof arg === "string" && arg.startsWith(`${DEEP_LINK_PROTOCOL}://`), + ); + if (url) { + handleDeepLink(url, mainWindow); + } + focusMainWindow(); +}); + ipcMain.handle("get-port", () => backendPort); ipcMain.handle("get-hermes-home", () => stateDir); ipcMain.handle("get-dashboard-state", () => dashboardState); @@ -119,23 +216,30 @@ ipcMain.handle("get-launch-at-login", () => { if (process.platform !== "darwin" && process.platform !== "win32") { return { enabled: false }; } - const opts = process.platform === "win32" ? { path: app.getPath("exe") } : undefined; + const opts = + process.platform === "win32" ? { path: app.getPath("exe") } : undefined; const s = app.getLoginItemSettings(opts); return { enabled: Boolean(s.openAtLogin) }; }); -ipcMain.handle("set-launch-at-login", (_evt, payload: { enabled?: boolean }) => { - if (process.platform !== "darwin" && process.platform !== "win32") { - throw new Error("Launch at login is only available on macOS and Windows"); - } - const enabled = Boolean(payload?.enabled); - if (process.platform === "win32") { - app.setLoginItemSettings({ openAtLogin: enabled, path: app.getPath("exe") }); - } else { - app.setLoginItemSettings({ openAtLogin: enabled }); - } - return { ok: true }; -}); +ipcMain.handle( + "set-launch-at-login", + (_evt, payload: { enabled?: boolean }) => { + if (process.platform !== "darwin" && process.platform !== "win32") { + throw new Error("Launch at login is only available on macOS and Windows"); + } + const enabled = Boolean(payload?.enabled); + if (process.platform === "win32") { + app.setLoginItemSettings({ + openAtLogin: enabled, + path: app.getPath("exe"), + }); + } else { + app.setLoginItemSettings({ openAtLogin: enabled }); + } + return { ok: true }; + }, +); ipcMain.handle("onboarding-get-state", () => { return { onboarded: readOnboardedState(stateDir) }; @@ -168,7 +272,12 @@ ipcMain.handle("reset-and-close", async () => { killAllTerminals(); - const rmOpts = { recursive: true, force: true, maxRetries: 3, retryDelay: 200 } as const; + const rmOpts = { + recursive: true, + force: true, + maxRetries: 3, + retryDelay: 200, + } as const; fs.rmSync(stateDir, rmOpts); await session.defaultSession.clearStorageData(); @@ -291,7 +400,10 @@ async function startDesktopBackend(): Promise { } catch (err: any) { console.error("Failed to start Python backend:", err); mainWindow?.webContents.send("python-error", err.message || String(err)); - dashboardState = { kind: "failed", error: "Gateway process failed to start" }; + dashboardState = { + kind: "failed", + error: "Gateway process failed to start", + }; mainWindow?.webContents.send("dashboard-error", dashboardState.error); } } @@ -302,7 +414,9 @@ async function autoStartLlamacppIfNeeded(): Promise { killOrphanedServer(stateDir); if (!isAnyProfileUsingLlamacpp(stateDir)) { - console.log("[llamacpp] auto-start skipped: no profile is configured to use llama.cpp"); + console.log( + "[llamacpp] auto-start skipped: no profile is configured to use llama.cpp", + ); return; } @@ -343,6 +457,10 @@ app.whenReady().then(async () => { } await startDesktopBackend(); void autoStartLlamacppIfNeeded(); + // Localhost landing page for post-Stripe-Checkout returns. Failure to bind + // (port collision with another Hermes instance) is non-fatal: the renderer + // still polls the balance after top-up. + stripeThanksServer = await startStripeThanksServer(); }); app.on("window-all-closed", () => { @@ -358,6 +476,8 @@ app.on("before-quit", () => { killAllTerminals(); void stopLlamacppServer().catch(() => {}); pythonBridge?.kill(); + void stripeThanksServer?.stop().catch(() => {}); + stripeThanksServer = null; void shutdownPosthogMain(); }); diff --git a/desktop/src/main/preload.ts b/desktop/src/main/preload.ts index f7f78ef29d18..ec55459e3be6 100644 --- a/desktop/src/main/preload.ts +++ b/desktop/src/main/preload.ts @@ -5,6 +5,12 @@ type DashboardState = | { kind: "ready"; port: number; url: string } | { kind: "failed"; error: string }; +type DeepLinkPayload = { + host: string; + pathname: string; + params: Record; +}; + function onIpc(channel: string, cb: (payload: T) => void): () => void { const handler = (_event: Electron.IpcRendererEvent, payload: T) => cb(payload); ipcRenderer.on(channel, handler); @@ -193,4 +199,10 @@ contextBridge.exposeInMainWorld("hermesAPI", { ipcRenderer.invoke("sidebar:set-favorites", { entries }), seedProfileProvider: async (source: string, target: string): Promise => ipcRenderer.invoke("seed-profile-provider", { source, target }), + + // ── Atomic deep-link (atomicbot-hermes://...) ────────────────────── + // JWT persistence lives in window.localStorage on the renderer side; only + // the deep-link delivery channel crosses the IPC boundary. + onAtomicDeepLink: (cb: (payload: DeepLinkPayload) => void): (() => void) => + onIpc("atomic:deep-link", cb), }); diff --git a/desktop/src/main/python-bridge.ts b/desktop/src/main/python-bridge.ts index e1fe8f9fac02..c729bd44344b 100644 --- a/desktop/src/main/python-bridge.ts +++ b/desktop/src/main/python-bridge.ts @@ -150,21 +150,29 @@ function createBridge( const kill = () => { restartCb = null; - if (!child.killed) { - child.kill("SIGTERM"); - setTimeout(() => { - if (!child.killed) child.kill("SIGKILL"); - }, 5_000); + if (child.exitCode !== null || child.signalCode !== null) return; + try { + child.kill("SIGKILL"); + } catch { + /* already dead */ } }; const killAndWait = (): Promise => { return new Promise((resolve) => { - if (child.killed || child.exitCode !== null) { + if (child.exitCode !== null || child.signalCode !== null) { resolve(); return; } - child.once("exit", () => resolve()); + let settled = false; + const done = () => { + if (settled) return; + settled = true; + resolve(); + }; + child.once("exit", done); + const safety = setTimeout(done, 1_500); + safety.unref?.(); kill(); }); }; diff --git a/desktop/tests/main/deep-link.test.ts b/desktop/tests/main/deep-link.test.ts new file mode 100644 index 000000000000..a76ed414db6a --- /dev/null +++ b/desktop/tests/main/deep-link.test.ts @@ -0,0 +1,38 @@ +import { describe, expect, it } from "vitest"; +import { parseDeepLinkUrl } from "../../src/main/atomic-auth/deep-link"; + +describe("parseDeepLinkUrl", () => { + it("parses atomicbot-hermes://auth with token, email, userId, isNewUser", () => { + const url = + "atomicbot-hermes://auth?token=jwt-abc&email=alice%40example.com&userId=u_123&isNewUser=true"; + const payload = parseDeepLinkUrl(url); + + expect(payload).not.toBeNull(); + expect(payload!.host).toBe("auth"); + expect(payload!.params.token).toBe("jwt-abc"); + expect(payload!.params.email).toBe("alice@example.com"); + expect(payload!.params.userId).toBe("u_123"); + expect(payload!.params.isNewUser).toBe("true"); + }); + + it("parses atomicbot-hermes://stripe-success with session_id", () => { + const payload = parseDeepLinkUrl( + "atomicbot-hermes://stripe-success?session_id=cs_test_xyz", + ); + expect(payload).not.toBeNull(); + expect(payload!.host).toBe("stripe-success"); + expect(payload!.params.session_id).toBe("cs_test_xyz"); + }); + + it("parses atomicbot-hermes://stripe-cancel with no params", () => { + const payload = parseDeepLinkUrl("atomicbot-hermes://stripe-cancel"); + expect(payload).not.toBeNull(); + expect(payload!.host).toBe("stripe-cancel"); + expect(payload!.params).toEqual({}); + }); + + it("returns null for malformed URLs", () => { + expect(parseDeepLinkUrl("not a url")).toBeNull(); + expect(parseDeepLinkUrl("")).toBeNull(); + }); +}); diff --git a/desktop/tests/renderer/atomic-backend-api.test.ts b/desktop/tests/renderer/atomic-backend-api.test.ts new file mode 100644 index 000000000000..652ff1e1e9b3 --- /dev/null +++ b/desktop/tests/renderer/atomic-backend-api.test.ts @@ -0,0 +1,115 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { + atomicBackendApi, + isUnauthorizedError, + getAtomicBackendUrl, + googleAuthDesktopUrl, +} from "../../renderer/src/services/atomic-backend-api"; + +const fetchMock = vi.fn(); +const originalFetch = globalThis.fetch; + +beforeEach(() => { + fetchMock.mockReset(); + globalThis.fetch = fetchMock as unknown as typeof fetch; +}); + +afterEach(() => { + globalThis.fetch = originalFetch; +}); + +function jsonResponse(body: unknown, init: ResponseInit = {}): Response { + return new Response(JSON.stringify(body), { + status: 200, + headers: { "Content-Type": "application/json" }, + ...init, + }); +} + +describe("atomicBackendApi", () => { + it("getAtomicBackendUrl falls back to https://api.atomicbot.ai", () => { + expect(getAtomicBackendUrl()).toBe("https://api.atomicbot.ai"); + }); + + it("googleAuthDesktopUrl appends scheme=atomicbot-hermes (backend reads query.scheme)", () => { + const url = googleAuthDesktopUrl(); + expect(url).toContain("/auth/google/desktop"); + expect(url).toContain("scheme=atomicbot-hermes"); + expect(url).not.toContain("redirect_scheme="); + }); + + it("getMe attaches Bearer token", async () => { + fetchMock.mockResolvedValueOnce( + jsonResponse({ userId: "u1", email: "a@b", subscriptionPlan: "free" }), + ); + + const me = await atomicBackendApi.getMe("jwt-x"); + + expect(me).toEqual({ userId: "u1", email: "a@b", subscriptionPlan: "free" }); + const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]; + expect(url).toContain("/auth/me"); + expect((init.headers as Headers).get("Authorization")).toBe("Bearer jwt-x"); + }); + + it("getBalance passes ?sync=true when requested", async () => { + fetchMock.mockResolvedValueOnce( + jsonResponse({ + total: 10, + subscriptionPlan: "free", + subscription: null, + payg: { limit: 10, remaining: 10 }, + }), + ); + + await atomicBackendApi.getBalance("jwt", { sync: true }); + + expect(fetchMock.mock.calls[0]?.[0]).toContain("/billing/balance?sync=true"); + }); + + it("createPaygTopup posts the JSON body", async () => { + fetchMock.mockResolvedValueOnce( + jsonResponse({ checkoutUrl: "https://stripe.test/cs_xyz" }), + ); + + const res = await atomicBackendApi.createPaygTopup("jwt", { + amountUsd: 25, + successUrl: "atomicbot-hermes://stripe-success", + cancelUrl: "atomicbot-hermes://stripe-cancel", + }); + + expect(res.checkoutUrl).toBe("https://stripe.test/cs_xyz"); + const init = fetchMock.mock.calls[0]?.[1] as RequestInit; + expect(init.method).toBe("POST"); + expect(JSON.parse(init.body as string)).toEqual({ + amountUsd: 25, + successUrl: "atomicbot-hermes://stripe-success", + cancelUrl: "atomicbot-hermes://stripe-cancel", + }); + }); + + it("getPortalUrl appends mode=payg when requested", async () => { + fetchMock.mockResolvedValueOnce( + jsonResponse({ portalUrl: "https://stripe.test/portal" }), + ); + + await atomicBackendApi.getPortalUrl("jwt", { mode: "payg" }); + + expect(fetchMock.mock.calls[0]?.[0]).toContain("/billing/portal?mode=payg"); + }); + + it("throws an error with status when the backend returns 401", async () => { + fetchMock.mockResolvedValueOnce( + jsonResponse({ error: "Invalid token" }, { status: 401 }), + ); + + let caught: unknown = null; + try { + await atomicBackendApi.getMe("jwt-bad"); + } catch (err) { + caught = err; + } + expect(caught).toBeInstanceOf(Error); + expect(isUnauthorizedError(caught)).toBe(true); + expect((caught as Error).message).toBe("Invalid token"); + }); +}); diff --git a/desktop/tests/renderer/atomicAuthSlice.test.ts b/desktop/tests/renderer/atomicAuthSlice.test.ts new file mode 100644 index 000000000000..ca650f57fcc2 --- /dev/null +++ b/desktop/tests/renderer/atomicAuthSlice.test.ts @@ -0,0 +1,179 @@ +import { configureStore } from "@reduxjs/toolkit"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const patchConfigMock = vi.fn(); +vi.mock("../../renderer/src/services/api", () => ({ + patchConfig: (...args: unknown[]) => patchConfigMock(...args), +})); + +const getPaygKeyMock = vi.fn(); +const getBalanceMock = vi.fn(); +const getMeMock = vi.fn(); +vi.mock("../../renderer/src/services/atomic-backend-api", async () => { + const actual = await vi.importActual< + typeof import("../../renderer/src/services/atomic-backend-api") + >("../../renderer/src/services/atomic-backend-api"); + return { + ...actual, + atomicBackendApi: { + getMe: (jwt: string) => getMeMock(jwt), + getBalance: (jwt: string, opts?: unknown) => getBalanceMock(jwt, opts), + getPaygKey: (jwt: string) => getPaygKeyMock(jwt), + createPaygTopup: vi.fn(), + getPortalUrl: vi.fn(), + getHistory: vi.fn(), + }, + }; +}); + +import { + applyPaygKey, + atomicAuthActions, + atomicAuthReducer, + clearAtomicAuthThunk, + fetchAtomicBalance, + restoreAtomicAuth, + storeAtomicToken, +} from "../../renderer/src/store/slices/atomicAuthSlice"; + +function makeStore() { + return configureStore({ reducer: { atomicAuth: atomicAuthReducer } }); +} + +beforeEach(() => { + window.localStorage.clear(); + patchConfigMock.mockReset(); + getPaygKeyMock.mockReset(); + getBalanceMock.mockReset(); + getMeMock.mockReset(); +}); + +afterEach(() => { + window.localStorage.clear(); + vi.clearAllMocks(); +}); + +describe("atomicAuthSlice", () => { + it("storeAtomicToken populates jwt/email/userId and persists to localStorage", async () => { + const store = makeStore(); + + await store + .dispatch( + storeAtomicToken({ jwt: "j", email: "e@x", userId: "u_1" }) as never, + ); + + const state = store.getState().atomicAuth; + expect(state.jwt).toBe("j"); + expect(state.email).toBe("e@x"); + expect(state.userId).toBe("u_1"); + + const persisted = window.localStorage.getItem("atomic-auth"); + expect(persisted).not.toBeNull(); + expect(JSON.parse(persisted as string)).toEqual({ + jwt: "j", + email: "e@x", + userId: "u_1", + }); + }); + + it("restoreAtomicAuth reads JWT back from localStorage", async () => { + window.localStorage.setItem( + "atomic-auth", + JSON.stringify({ jwt: "jwt-restored", email: "r@x", userId: "u_42" }), + ); + + const store = makeStore(); + await store.dispatch(restoreAtomicAuth() as never); + + const state = store.getState().atomicAuth; + expect(state.jwt).toBe("jwt-restored"); + expect(state.email).toBe("r@x"); + expect(state.userId).toBe("u_42"); + expect(state.restoreLoaded).toBe(true); + }); + + it("restoreAtomicAuth tolerates missing or malformed entries", async () => { + window.localStorage.setItem("atomic-auth", "not json"); + + const store = makeStore(); + await store.dispatch(restoreAtomicAuth() as never); + + const state = store.getState().atomicAuth; + expect(state.jwt).toBeNull(); + expect(state.restoreLoaded).toBe(true); + }); + + it("clearAtomicAuthThunk wipes the slice and localStorage entry", async () => { + const store = makeStore(); + await store.dispatch( + storeAtomicToken({ jwt: "j", email: "e@x", userId: "u_1" }) as never, + ); + expect(window.localStorage.getItem("atomic-auth")).not.toBeNull(); + + await store.dispatch(clearAtomicAuthThunk() as never); + + const state = store.getState().atomicAuth; + expect(state.jwt).toBeNull(); + expect(window.localStorage.getItem("atomic-auth")).toBeNull(); + }); + + it("applyPaygKey rejects when no JWT is present", async () => { + const store = makeStore(); + const action = await store.dispatch(applyPaygKey({ port: 8080 }) as never); + expect((action as { type: string }).type).toBe("atomicAuth/applyPaygKey/rejected"); + expect(store.getState().atomicAuth.applyKeyError).toMatch(/Not authenticated/); + }); + + it("applyPaygKey calls patchConfig with OPENROUTER_API_KEY env", async () => { + const store = makeStore(); + await store.dispatch( + storeAtomicToken({ jwt: "jwt-x", email: "e", userId: "u" }) as never, + ); + + getPaygKeyMock.mockResolvedValueOnce({ + key: "sk-or-fake", + keyHash: "hash", + remaining: 10, + limit: 10, + }); + patchConfigMock.mockResolvedValueOnce({ ok: true }); + + const result = await store.dispatch(applyPaygKey({ port: 8123 }) as never); + + expect((result as { type: string }).type).toBe("atomicAuth/applyPaygKey/fulfilled"); + expect(patchConfigMock).toHaveBeenCalledWith(8123, { + config: { provider: "openrouter" }, + env: { OPENROUTER_API_KEY: "sk-or-fake" }, + }); + }); + + it("fetchAtomicBalance stores the result and the subscription plan", async () => { + const store = makeStore(); + await store.dispatch( + storeAtomicToken({ jwt: "jwt", email: "e", userId: "u" }) as never, + ); + + getBalanceMock.mockResolvedValueOnce({ + total: 5, + subscriptionPlan: "pro", + subscription: { limit: 100, remaining: 80, expiresAt: null }, + payg: { limit: 5, remaining: 5 }, + }); + + await store.dispatch(fetchAtomicBalance({}) as never); + + const state = store.getState().atomicAuth; + expect(state.subscriptionPlan).toBe("pro"); + expect(state.balance?.payg?.remaining).toBe(5); + }); + + it("setTopupPending and setTopupError reducers update state", () => { + const store = makeStore(); + + store.dispatch(atomicAuthActions.setTopupPending(true)); + expect(store.getState().atomicAuth.topupPending).toBe(true); + + store.dispatch(atomicAuthActions.setTopupError("oops")); + expect(store.getState().atomicAuth.topupError).toBe("oops"); + }); +}); diff --git a/desktop/tests/setup.ts b/desktop/tests/setup.ts new file mode 100644 index 000000000000..d0de870dc558 --- /dev/null +++ b/desktop/tests/setup.ts @@ -0,0 +1 @@ +import "@testing-library/jest-dom"; diff --git a/desktop/vitest.config.ts b/desktop/vitest.config.ts new file mode 100644 index 000000000000..6418520bcada --- /dev/null +++ b/desktop/vitest.config.ts @@ -0,0 +1,24 @@ +import { defineConfig } from "vitest/config"; +import react from "@vitejs/plugin-react"; +import path from "node:path"; + +export default defineConfig({ + plugins: [react()], + resolve: { + alias: { + "@store": path.resolve(__dirname, "renderer/src/store"), + "@shared": path.resolve(__dirname, "renderer/src/ui/shared"), + "@styles": path.resolve(__dirname, "renderer/src/ui/styles"), + "@ui": path.resolve(__dirname, "renderer/src/ui"), + "@ipc": path.resolve(__dirname, "renderer/src/ipc"), + "@lib": path.resolve(__dirname, "renderer/src/lib"), + "@analytics": path.resolve(__dirname, "renderer/src/analytics"), + }, + }, + test: { + environment: "jsdom", + globals: true, + include: ["tests/**/*.test.ts", "tests/**/*.test.tsx"], + setupFiles: ["./tests/setup.ts"], + }, +}); diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh old mode 100644 new mode 100755 index dc1edd32c24f..67d193f13b7e --- a/docker/entrypoint.sh +++ b/docker/entrypoint.sh @@ -1,13 +1,14 @@ #!/bin/bash -# Docker entrypoint: bootstrap config files into the mounted volume, then run hermes. +# Docker/Podman entrypoint: bootstrap config files into the mounted volume, then run hermes. set -e -HERMES_HOME="/opt/data" +HERMES_HOME="${HERMES_HOME:-/opt/data}" INSTALL_DIR="/opt/hermes" # --- Privilege dropping via gosu --- -# When started as root (the default), optionally remap the hermes user/group -# to match host-side ownership, fix volume permissions, then re-exec as hermes. +# When started as root (the default for Docker, or fakeroot in rootless Podman), +# optionally remap the hermes user/group to match host-side ownership, fix volume +# permissions, then re-exec as hermes. if [ "$(id -u)" = "0" ]; then if [ -n "$HERMES_UID" ] && [ "$HERMES_UID" != "$(id -u hermes)" ]; then echo "Changing hermes UID to $HERMES_UID" @@ -16,13 +17,19 @@ if [ "$(id -u)" = "0" ]; then if [ -n "$HERMES_GID" ] && [ "$HERMES_GID" != "$(id -g hermes)" ]; then echo "Changing hermes GID to $HERMES_GID" - groupmod -g "$HERMES_GID" hermes + # -o allows non-unique GID (e.g. macOS GID 20 "staff" may already exist + # as "dialout" in the Debian-based container image) + groupmod -o -g "$HERMES_GID" hermes 2>/dev/null || true fi actual_hermes_uid=$(id -u hermes) if [ "$(stat -c %u "$HERMES_HOME" 2>/dev/null)" != "$actual_hermes_uid" ]; then echo "$HERMES_HOME is not owned by $actual_hermes_uid, fixing" - chown -R hermes:hermes "$HERMES_HOME" + # In rootless Podman the container's "root" is mapped to an unprivileged + # host UID — chown will fail. That's fine: the volume is already owned + # by the mapped user on the host side. + chown -R hermes:hermes "$HERMES_HOME" 2>/dev/null || \ + echo "Warning: chown failed (rootless container?) — continuing anyway" fi echo "Dropping root privileges" @@ -51,6 +58,13 @@ if [ ! -f "$HERMES_HOME/config.yaml" ]; then cp "$INSTALL_DIR/cli-config.yaml.example" "$HERMES_HOME/config.yaml" fi +# Ensure the main config file remains accessible to the hermes runtime user +# even if it was edited on the host after initial ownership setup. +if [ -f "$HERMES_HOME/config.yaml" ]; then + chown hermes:hermes "$HERMES_HOME/config.yaml" + chmod 640 "$HERMES_HOME/config.yaml" +fi + # SOUL.md if [ ! -f "$HERMES_HOME/SOUL.md" ]; then cp "$INSTALL_DIR/docker/SOUL.md" "$HERMES_HOME/SOUL.md" @@ -61,4 +75,19 @@ if [ -d "$INSTALL_DIR/skills" ]; then python3 "$INSTALL_DIR/tools/skills_sync.py" fi +# Final exec: two supported invocation patterns. +# +# docker run -> exec `hermes` with no args (legacy default) +# docker run chat -q "..." -> exec `hermes chat -q "..."` (legacy wrap) +# docker run sleep infinity -> exec `sleep infinity` directly +# docker run bash -> exec `bash` directly +# +# If the first positional arg resolves to an executable on PATH, we assume the +# caller wants to run it directly (needed by the launcher which runs long-lived +# `sleep infinity` sandbox containers — see tools/environments/docker.py). +# Otherwise we treat the args as a hermes subcommand and wrap with `hermes`, +# preserving the documented `docker run ` behavior. +if [ $# -gt 0 ] && command -v "$1" >/dev/null 2>&1; then + exec "$@" +fi exec hermes "$@" diff --git a/docs/acp-setup.md b/docs/acp-setup.md deleted file mode 100644 index 8da4e2a21509..000000000000 --- a/docs/acp-setup.md +++ /dev/null @@ -1,228 +0,0 @@ -# Hermes Agent — ACP (Agent Client Protocol) Setup Guide - -Hermes Agent supports the **Agent Client Protocol (ACP)**, allowing it to run as -a coding agent inside your editor. ACP lets your IDE send tasks to Hermes, and -Hermes responds with file edits, terminal commands, and explanations — all shown -natively in the editor UI. - ---- - -## Prerequisites - -- Hermes Agent installed and configured (`hermes setup` completed) -- An API key / provider set up in `~/.hermes/.env` or via `hermes login` -- Python 3.11+ - -Install the ACP extra: - -```bash -pip install -e ".[acp]" -``` - ---- - -## VS Code Setup - -### 1. Install the ACP Client extension - -Open VS Code and install **ACP Client** from the marketplace: - -- Press `Ctrl+Shift+X` (or `Cmd+Shift+X` on macOS) -- Search for **"ACP Client"** -- Click **Install** - -Or install from the command line: - -```bash -code --install-extension anysphere.acp-client -``` - -### 2. Configure settings.json - -Open your VS Code settings (`Ctrl+,` → click the `{}` icon for JSON) and add: - -```json -{ - "acpClient.agents": [ - { - "name": "hermes-agent", - "registryDir": "/path/to/hermes-agent/acp_registry" - } - ] -} -``` - -Replace `/path/to/hermes-agent` with the actual path to your Hermes Agent -installation (e.g. `~/.hermes/hermes-agent`). - -Alternatively, if `hermes` is on your PATH, the ACP Client can discover it -automatically via the registry directory. - -### 3. Restart VS Code - -After configuring, restart VS Code. You should see **Hermes Agent** appear in -the ACP agent picker in the chat/agent panel. - ---- - -## Zed Setup - -Zed has built-in ACP support. - -### 1. Configure Zed settings - -Open Zed settings (`Cmd+,` on macOS or `Ctrl+,` on Linux) and add to your -`settings.json`: - -```json -{ - "agent_servers": { - "hermes-agent": { - "type": "custom", - "command": "hermes", - "args": ["acp"], - }, - }, -} -``` - -### 2. Restart Zed - -Hermes Agent will appear in the agent panel. Select it and start a conversation. - ---- - -## JetBrains Setup (IntelliJ, PyCharm, WebStorm, etc.) - -### 1. Install the ACP plugin - -- Open **Settings** → **Plugins** → **Marketplace** -- Search for **"ACP"** or **"Agent Client Protocol"** -- Install and restart the IDE - -### 2. Configure the agent - -- Open **Settings** → **Tools** → **ACP Agents** -- Click **+** to add a new agent -- Set the registry directory to your `acp_registry/` folder: - `/path/to/hermes-agent/acp_registry` -- Click **OK** - -### 3. Use the agent - -Open the ACP panel (usually in the right sidebar) and select **Hermes Agent**. - ---- - -## What You Will See - -Once connected, your editor provides a native interface to Hermes Agent: - -### Chat Panel -A conversational interface where you can describe tasks, ask questions, and -give instructions. Hermes responds with explanations and actions. - -### File Diffs -When Hermes edits files, you see standard diffs in the editor. You can: -- **Accept** individual changes -- **Reject** changes you don't want -- **Review** the full diff before applying - -### Terminal Commands -When Hermes needs to run shell commands (builds, tests, installs), the editor -shows them in an integrated terminal. Depending on your settings: -- Commands may run automatically -- Or you may be prompted to **approve** each command - -### Approval Flow -For potentially destructive operations, the editor will prompt you for -approval before Hermes proceeds. This includes: -- File deletions -- Shell commands -- Git operations - ---- - -## Configuration - -Hermes Agent under ACP uses the **same configuration** as the CLI: - -- **API keys / providers**: `~/.hermes/.env` -- **Agent config**: `~/.hermes/config.yaml` -- **Skills**: `~/.hermes/skills/` -- **Sessions**: `~/.hermes/state.db` - -You can run `hermes setup` to configure providers, or edit `~/.hermes/.env` -directly. - -### Changing the model - -Edit `~/.hermes/config.yaml`: - -```yaml -model: openrouter/nous/hermes-3-llama-3.1-70b -``` - -Or set the `HERMES_MODEL` environment variable. - -### Toolsets - -ACP sessions use the curated `hermes-acp` toolset by default. It is designed for editor workflows and intentionally excludes things like messaging delivery, cronjob management, and audio-first UX features. - ---- - -## Troubleshooting - -### Agent doesn't appear in the editor - -1. **Check the registry path** — make sure the `acp_registry/` directory path - in your editor settings is correct and contains `agent.json`. -2. **Check `hermes` is on PATH** — run `which hermes` in a terminal. If not - found, you may need to activate your virtualenv or add it to PATH. -3. **Restart the editor** after changing settings. - -### Agent starts but errors immediately - -1. Run `hermes doctor` to check your configuration. -2. Check that you have a valid API key: `hermes status` -3. Try running `hermes acp` directly in a terminal to see error output. - -### "Module not found" errors - -Make sure you installed the ACP extra: - -```bash -pip install -e ".[acp]" -``` - -### Slow responses - -- ACP streams responses, so you should see incremental output. If the agent - appears stuck, check your network connection and API provider status. -- Some providers have rate limits. Try switching to a different model/provider. - -### Permission denied for terminal commands - -If the editor blocks terminal commands, check your ACP Client extension -settings for auto-approval or manual-approval preferences. - -### Logs - -Hermes logs are written to stderr when running in ACP mode. Check: -- VS Code: **Output** panel → select **ACP Client** or **Hermes Agent** -- Zed: **View** → **Toggle Terminal** and check the process output -- JetBrains: **Event Log** or the ACP tool window - -You can also enable verbose logging: - -```bash -HERMES_LOG_LEVEL=DEBUG hermes acp -``` - ---- - -## Further Reading - -- [ACP Specification](https://github.com/anysphere/acp) -- [Hermes Agent Documentation](https://github.com/NousResearch/hermes-agent) -- Run `hermes --help` for all CLI options diff --git a/docs/honcho-integration-spec.html b/docs/honcho-integration-spec.html deleted file mode 100644 index 455fb84f237c..000000000000 --- a/docs/honcho-integration-spec.html +++ /dev/null @@ -1,698 +0,0 @@ - - - - - -honcho-integration-spec - - - - - - - -
- -
- -
-

honcho-integration-spec

-

Comparison of Hermes Agent vs. openclaw-honcho — and a porting spec for bringing Hermes patterns into other Honcho integrations.

-
- hermes-agent / openclaw-honcho - Python + TypeScript - 2026-03-09 -
-
- - - - -
-

Overview

- -

Two independent Honcho integrations have been built for two different agent runtimes: Hermes Agent (Python, baked into the runner) and openclaw-honcho (TypeScript plugin via hook/tool API). Both use the same Honcho peer paradigm — dual peer model, session.context(), peer.chat() — but they made different tradeoffs at every layer.

- -

This document maps those tradeoffs and defines a porting spec: a set of Hermes-originated patterns, each stated as an integration-agnostic interface, that any Honcho integration can adopt regardless of runtime or language.

- -
- Scope Both integrations work correctly today. This spec is about the delta — patterns in Hermes that are worth propagating and patterns in openclaw-honcho that Hermes should eventually adopt. The spec is additive, not prescriptive. -
-
- - -
-

Architecture comparison

- -

Hermes: baked-in runner

-

Honcho is initialised directly inside AIAgent.__init__. There is no plugin boundary. Session management, context injection, async prefetch, and CLI surface are all first-class concerns of the runner. Context is injected once per session (baked into _cached_system_prompt) and never re-fetched mid-session — this maximises prefix cache hits at the LLM provider.

- -
-%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1f3150', 'primaryTextColor': '#c9d1d9', 'primaryBorderColor': '#3d6ea5', 'lineColor': '#3d6ea5', 'secondaryColor': '#162030', 'tertiaryColor': '#11151c' }}}%% -flowchart TD - U["user message"] --> P["_honcho_prefetch()
(reads cache — no HTTP)"] - P --> SP["_build_system_prompt()
(first turn only, cached)"] - SP --> LLM["LLM call"] - LLM --> R["response"] - R --> FP["_honcho_fire_prefetch()
(daemon threads, turn end)"] - FP --> C1["prefetch_context() thread"] - FP --> C2["prefetch_dialectic() thread"] - C1 --> CACHE["_context_cache / _dialectic_cache"] - C2 --> CACHE - - style U fill:#162030,stroke:#3d6ea5,color:#c9d1d9 - style P fill:#1f3150,stroke:#3d6ea5,color:#c9d1d9 - style SP fill:#1f3150,stroke:#3d6ea5,color:#c9d1d9 - style LLM fill:#162030,stroke:#3d6ea5,color:#c9d1d9 - style R fill:#162030,stroke:#3d6ea5,color:#c9d1d9 - style FP fill:#2a1a40,stroke:#bc8cff,color:#c9d1d9 - style C1 fill:#2a1a40,stroke:#bc8cff,color:#c9d1d9 - style C2 fill:#2a1a40,stroke:#bc8cff,color:#c9d1d9 - style CACHE fill:#11151c,stroke:#484f58,color:#6e7681 -
- -

openclaw-honcho: hook-based plugin

-

The plugin registers hooks against OpenClaw's event bus. Context is fetched synchronously inside before_prompt_build on every turn. Message capture happens in agent_end. The multi-agent hierarchy is tracked via subagent_spawned. This model is correct but every turn pays a blocking Honcho round-trip before the LLM call can begin.

- -
-%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1f3150', 'primaryTextColor': '#c9d1d9', 'primaryBorderColor': '#3d6ea5', 'lineColor': '#3d6ea5', 'secondaryColor': '#162030', 'tertiaryColor': '#11151c' }}}%% -flowchart TD - U2["user message"] --> BPB["before_prompt_build
(BLOCKING HTTP — every turn)"] - BPB --> CTX["session.context()"] - CTX --> SP2["system prompt assembled"] - SP2 --> LLM2["LLM call"] - LLM2 --> R2["response"] - R2 --> AE["agent_end hook"] - AE --> SAVE["session.addMessages()
session.setMetadata()"] - - style U2 fill:#162030,stroke:#3d6ea5,color:#c9d1d9 - style BPB fill:#3a1515,stroke:#f47067,color:#c9d1d9 - style CTX fill:#3a1515,stroke:#f47067,color:#c9d1d9 - style SP2 fill:#1f3150,stroke:#3d6ea5,color:#c9d1d9 - style LLM2 fill:#162030,stroke:#3d6ea5,color:#c9d1d9 - style R2 fill:#162030,stroke:#3d6ea5,color:#c9d1d9 - style AE fill:#162030,stroke:#3d6ea5,color:#c9d1d9 - style SAVE fill:#11151c,stroke:#484f58,color:#6e7681 -
-
- - -
-

Diff table

- -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
DimensionHermes Agentopenclaw-honcho
Context injection timingOnce per session (cached). Zero HTTP on response path after turn 1.Every turn, blocking. Fresh context per turn but adds latency.
Prefetch strategyDaemon threads fire at turn end; consumed next turn from cache.None. Blocking call at prompt-build time.
Dialectic (peer.chat)Prefetched async; result injected into system prompt next turn.On-demand via honcho_recall / honcho_analyze tools.
Reasoning levelDynamic: scales with message length. Floor = config default. Cap = "high".Fixed per tool: recall=minimal, analyze=medium.
Memory modesuser_memory_mode / agent_memory_mode: hybrid / honcho / local.None. Always writes to Honcho.
Write frequencyasync (background queue), turn, session, N turns.After every agent_end (no control).
AI peer identityobserve_me=True, seed_ai_identity(), get_ai_representation(), SOUL.md → AI peer.Agent files uploaded to agent peer at setup. No ongoing self-observation seeding.
Context scopeUser peer + AI peer representation, both injected.User peer (owner) representation + conversation summary. peerPerspective on context call.
Session namingper-directory / global / manual map / title-based.Derived from platform session key.
Multi-agentSingle-agent only.Parent observer hierarchy via subagent_spawned.
Tool surfaceSingle query_user_context tool (on-demand dialectic).6 tools: session, profile, search, context (fast) + recall, analyze (LLM).
Platform metadataNot stripped.Explicitly stripped before Honcho storage.
Message dedupNone (sends on every save cycle).lastSavedIndex in session metadata prevents re-sending.
CLI surface in promptManagement commands injected into system prompt. Agent knows its own CLI.Not injected.
AI peer name in identityReplaces "Hermes Agent" in DEFAULT_AGENT_IDENTITY when configured.Not implemented.
QMD / local file searchNot implemented.Passthrough tools when QMD backend configured.
Workspace metadataNot implemented.agentPeerMap in workspace metadata tracks agent→peer ID.
-
-
- - -
-

Hermes patterns to port

- -

Six patterns from Hermes are worth adopting in any Honcho integration. They are described below as integration-agnostic interfaces — the implementation will differ per runtime, but the contract is the same.

- -
-
-

Patterns Hermes contributes

-
    -
  • Async prefetch (zero-latency)
    • -
  • Dynamic reasoning level
    • -
  • Per-peer memory modes
    • -
  • AI peer identity formation
    • -
  • Session naming strategies
    • -
  • CLI surface injection
    • -
-
-
-

Patterns openclaw contributes back

-
    -
  • lastSavedIndex dedup
    • -
  • Platform metadata stripping
    • -
  • Multi-agent observer hierarchy
    • -
  • peerPerspective on context()
    • -
  • Tiered tool surface (fast/LLM)
    • -
  • Workspace agentPeerMap
    • -
-
-
-
- - -
-

Spec: async prefetch

- -

Problem

-

Calling session.context() and peer.chat() synchronously before each LLM call adds 200–800ms of Honcho round-trip latency to every turn. Users experience this as the agent "thinking slowly."

- -

Pattern

-

Fire both calls as non-blocking background work at the end of each turn. Store results in a per-session cache keyed by session ID. At the start of the next turn, pop from cache — the HTTP is already done. First turn is cold (empty cache); all subsequent turns are zero-latency on the response path.

- -

Interface contract

- 
// TypeScript (openclaw / nanobot plugin shape)
-
-interface AsyncPrefetch {
-  // Fire context + dialectic fetches at turn end. Non-blocking.
-  firePrefetch(sessionId: string, userMessage: string): void;
-
-  // Pop cached results at turn start. Returns empty if cache is cold.
-  popContextResult(sessionId: string): ContextResult | null;
-  popDialecticResult(sessionId: string): string | null;
-}
-
-type ContextResult = {
-  representation: string;
-  card: string[];
-  aiRepresentation?: string;  // AI peer context if enabled
-  summary?: string;            // conversation summary if fetched
-};
- -

Implementation notes

-
    -
  • Python: threading.Thread(daemon=True). Write to dict[session_id, result] — GIL makes this safe for simple writes.
    • -
  • TypeScript: Promise stored in Map<string, Promise<ContextResult>>. Await at pop time. If not resolved yet, skip (return null) — do not block.
    • -
  • The pop is destructive: clears the cache entry after reading so stale data never accumulates.
    • -
  • Prefetch should also fire on first turn (even though it won't be consumed until turn 2) — this ensures turn 2 is never cold.
    • -
- -

openclaw-honcho adoption

-

Move session.context() from before_prompt_build to a post-agent_end background task. Store result in state.contextCache. In before_prompt_build, read from cache instead of calling Honcho. If cache is empty (turn 1), inject nothing — the prompt is still valid without Honcho context on the first turn.

-
- - -
-

Spec: dynamic reasoning level

- -

Problem

-

Honcho's dialectic endpoint supports reasoning levels from minimal to max. A fixed level per tool wastes budget on simple queries and under-serves complex ones.

- -

Pattern

-

Select the reasoning level dynamically based on the user's message. Use the configured default as a floor. Bump by message length. Cap auto-selection at high — never select max automatically.

- -

Interface contract

- 
// Shared helper — identical logic in any language
-
-const LEVELS = ["minimal", "low", "medium", "high", "max"];
-
-function dynamicReasoningLevel(
-  query: string,
-  configDefault: string = "low"
-): string {
-  const baseIdx = Math.max(0, LEVELS.indexOf(configDefault));
-  const n = query.length;
-  const bump = n < 120 ? 0 : n < 400 ? 1 : 2;
-  return LEVELS[Math.min(baseIdx + bump, 3)]; // cap at "high" (idx 3)
-}
- -

Config key

-

Add a dialecticReasoningLevel config field (string, default "low"). This sets the floor. Users can raise or lower it. The dynamic bump always applies on top.

- -

openclaw-honcho adoption

-

Apply in honcho_recall and honcho_analyze: replace the fixed reasoningLevel with the dynamic selector. honcho_recall should use floor "minimal" and honcho_analyze floor "medium" — both still bump with message length.

-
- - -
-

Spec: per-peer memory modes

- -

Problem

-

Users want independent control over whether user context and agent context are written locally, to Honcho, or both. A single memoryMode shorthand is not granular enough.

- -

Pattern

-

Three modes per peer: hybrid (write both local + Honcho), honcho (Honcho only, disable local files), local (local files only, skip Honcho sync for this peer). Two orthogonal axes: user peer and agent peer.

- -

Config schema

- 
// ~/.openclaw/openclaw.json  (or ~/.nanobot/config.json)
-{
-  "plugins": {
-    "openclaw-honcho": {
-      "config": {
-        "apiKey": "...",
-        "memoryMode": "hybrid",          // shorthand: both peers
-        "userMemoryMode": "honcho",       // override for user peer
-        "agentMemoryMode": "hybrid"       // override for agent peer
-      }
-    }
-  }
-}
- -

Resolution order

-
    -
  1. Per-peer field (userMemoryMode / agentMemoryMode) — wins if present.
    2. -
  2. Shorthand memoryMode — applies to both peers as default.
    3. -
  3. Hardcoded default: "hybrid".
    4. -
- -

Effect on Honcho sync

-
    -
  • userMemoryMode=local: skip adding user peer messages to Honcho.
    • -
  • agentMemoryMode=local: skip adding assistant peer messages to Honcho.
    • -
  • Both local: skip session.addMessages() entirely.
    • -
  • userMemoryMode=honcho: disable local USER.md writes.
    • -
  • agentMemoryMode=honcho: disable local MEMORY.md / SOUL.md writes.
    • -
-
- - -
-

Spec: AI peer identity formation

- -

Problem

-

Honcho builds the user's representation organically by observing what the user says. The same mechanism exists for the AI peer — but only if observe_me=True is set for the agent peer. Without it, the agent peer accumulates nothing and Honcho's AI-side model never forms.

- -

Additionally, existing persona files (SOUL.md, IDENTITY.md) should seed the AI peer's Honcho representation at first activation, rather than waiting for it to emerge from scratch.

- -

Part A: observe_me=True for agent peer

- 
// TypeScript — in session.addPeers() call
-await session.addPeers([
-  [ownerPeer.id, { observeMe: true,  observeOthers: false }],
-  [agentPeer.id, { observeMe: true,  observeOthers: true  }], // was false
-]);
- -

This is a one-line change but foundational. Without it, Honcho's AI peer representation stays empty regardless of what the agent says.

- -

Part B: seedAiIdentity()

- 
async function seedAiIdentity(
-  session: HonchoSession,
-  agentPeer: Peer,
-  content: string,
-  source: string
-): Promise<boolean> {
-  const wrapped = [
-    `<ai_identity_seed>`,
-    `<source>${source}</source>`,
-    ``,
-    content.trim(),
-    `</ai_identity_seed>`,
-  ].join("\n");
-
-  await agentPeer.addMessage("assistant", wrapped);
-  return true;
-}
- -

Part C: migrate agent files at setup

-

During openclaw honcho setup, upload agent-self files (SOUL.md, IDENTITY.md, AGENTS.md, BOOTSTRAP.md) to the agent peer using seedAiIdentity() instead of session.uploadFile(). This routes the content through Honcho's observation pipeline rather than the file store.

- -

Part D: AI peer name in identity

-

When the agent has a configured name (non-default), inject it into the agent's self-identity prefix. In OpenClaw this means adding to the injected system prompt section:

- 
// In context hook return value
-return {
-  systemPrompt: [
-    agentName ? `You are ${agentName}.` : "",
-    "## User Memory Context",
-    ...sections,
-  ].filter(Boolean).join("\n\n")
-};
- -

CLI surface: honcho identity subcommand

- 
openclaw honcho identity <file>    # seed from file
-openclaw honcho identity --show    # show current AI peer representation
-
- - -
-

Spec: session naming strategies

- -

Problem

-

When Honcho is used across multiple projects or directories, a single global session means every project shares the same context. Per-directory sessions provide isolation without requiring users to name sessions manually.

- -

Strategies

-
- - - - - - - - -
StrategySession keyWhen to use
per-directorybasename of CWDDefault. Each project gets its own session.
globalfixed string "global"Single cross-project session.
manual mapuser-configured per pathsessions config map overrides directory basename.
title-basedsanitized session titleWhen agent supports named sessions; title set mid-conversation.
-
- -

Config schema

- 
{
-  "sessionStrategy": "per-directory",   // "per-directory" | "global"
-  "sessionPeerPrefix": false,            // prepend peer name to session key
-  "sessions": {                            // manual overrides
-    "/home/user/projects/foo": "foo-project"
-  }
-}
- -

CLI surface

- 
openclaw honcho sessions              # list all mappings
-openclaw honcho map <name>           # map cwd to session name
-openclaw honcho map                   # no-arg = list mappings
- -

Resolution order: manual map wins → session title → directory basename → platform key.

-
- - -
-

Spec: CLI surface injection

- -

Problem

-

When a user asks "how do I change my memory settings?" or "what Honcho commands are available?" the agent either hallucinates or says it doesn't know. The agent should know its own management interface.

- -

Pattern

-

When Honcho is active, append a compact command reference to the system prompt. The agent can cite these commands directly instead of guessing.

- - 
// In context hook, append to systemPrompt
-const honchoSection = [
-  "# Honcho memory integration",
-  `Active. Session: ${sessionKey}. Mode: ${mode}.`,
-  "Management commands:",
-  "  openclaw honcho status                    — show config + connection",
-  "  openclaw honcho mode [hybrid|honcho|local] — show or set memory mode",
-  "  openclaw honcho sessions                  — list session mappings",
-  "  openclaw honcho map <name>                — map directory to session",
-  "  openclaw honcho identity [file] [--show]  — seed or show AI identity",
-  "  openclaw honcho setup                     — full interactive wizard",
-].join("\n");
- -
- Keep it compact. This section is injected every turn. Keep it under 300 chars of context. List commands, not explanations — the agent can explain them on request. -
-
- - -
-

openclaw-honcho checklist

- -

Ordered by impact. Each item maps to a spec section above.

- -
    -
  • Async prefetch — move session.context() out of before_prompt_build into post-agent_end background Promise. Pop from cache at prompt build. (spec)
    • -
  • observe_me=True for agent peer — one-line change in session.addPeers() config for agent peer. (spec)
    • -
  • Dynamic reasoning level — add dynamicReasoningLevel() helper; apply in honcho_recall and honcho_analyze. Add dialecticReasoningLevel to config schema. (spec)
    • -
  • Per-peer memory modes — add userMemoryMode / agentMemoryMode to config; gate Honcho sync and local writes accordingly. (spec)
    • -
  • seedAiIdentity() — add helper; apply during setup migration for SOUL.md / IDENTITY.md instead of session.uploadFile(). (spec)
    • -
  • Session naming strategies — add sessionStrategy, sessions map, sessionPeerPrefix to config; implement resolution function. (spec)
    • -
  • CLI surface injection — append command reference to before_prompt_build return value when Honcho is active. (spec)
    • -
  • honcho identity subcommand — add openclaw honcho identity CLI command. (spec)
    • -
  • AI peer name injection — if aiPeer name configured, prepend to injected system prompt. (spec)
    • -
  • honcho mode / honcho sessions / honcho map — CLI parity with Hermes. (spec)
    • -
- -
- Already done in openclaw-honcho (do not re-implement): lastSavedIndex dedup, platform metadata stripping, multi-agent parent observer hierarchy, peerPerspective on context(), tiered tool surface (fast/LLM), workspace agentPeerMap, QMD passthrough, self-hosted Honcho support. -
-
- - -
-

nanobot-honcho checklist

- -

nanobot-honcho is a greenfield integration. Start from openclaw-honcho's architecture (hook-based, dual peer) and apply all Hermes patterns from day one rather than retrofitting. Priority order:

- -

Phase 1 — core correctness

-
    -
  • Dual peer model (owner + agent peer), both with observe_me=True
    • -
  • Message capture at turn end with lastSavedIndex dedup
    • -
  • Platform metadata stripping before Honcho storage
    • -
  • Async prefetch from day one — do not implement blocking context injection
    • -
  • Legacy file migration at first activation (USER.md → owner peer, SOUL.md → seedAiIdentity())
    • -
- -

Phase 2 — configuration

-
    -
  • Config schema: apiKey, workspaceId, baseUrl, memoryMode, userMemoryMode, agentMemoryMode, dialecticReasoningLevel, sessionStrategy, sessions
    • -
  • Per-peer memory mode gating
    • -
  • Dynamic reasoning level
    • -
  • Session naming strategies
    • -
- -

Phase 3 — tools and CLI

-
    -
  • Tool surface: honcho_profile, honcho_recall, honcho_analyze, honcho_search, honcho_context
    • -
  • CLI: setup, status, sessions, map, mode, identity
    • -
  • CLI surface injection into system prompt
    • -
  • AI peer name wired into agent identity
    • -
-
- -
- - - - - diff --git a/docs/honcho-integration-spec.md b/docs/honcho-integration-spec.md deleted file mode 100644 index 7731a262d908..000000000000 --- a/docs/honcho-integration-spec.md +++ /dev/null @@ -1,377 +0,0 @@ -# honcho-integration-spec - -Comparison of Hermes Agent vs. openclaw-honcho — and a porting spec for bringing Hermes patterns into other Honcho integrations. - ---- - -## Overview - -Two independent Honcho integrations have been built for two different agent runtimes: **Hermes Agent** (Python, baked into the runner) and **openclaw-honcho** (TypeScript plugin via hook/tool API). Both use the same Honcho peer paradigm — dual peer model, `session.context()`, `peer.chat()` — but they made different tradeoffs at every layer. - -This document maps those tradeoffs and defines a porting spec: a set of Hermes-originated patterns, each stated as an integration-agnostic interface, that any Honcho integration can adopt regardless of runtime or language. - -> **Scope** Both integrations work correctly today. This spec is about the delta — patterns in Hermes that are worth propagating and patterns in openclaw-honcho that Hermes should eventually adopt. The spec is additive, not prescriptive. - ---- - -## Architecture comparison - -### Hermes: baked-in runner - -Honcho is initialised directly inside `AIAgent.__init__`. There is no plugin boundary. Session management, context injection, async prefetch, and CLI surface are all first-class concerns of the runner. Context is injected once per session (baked into `_cached_system_prompt`) and never re-fetched mid-session — this maximises prefix cache hits at the LLM provider. - -Turn flow: - -``` -user message - → _honcho_prefetch() (reads cache — no HTTP) - → _build_system_prompt() (first turn only, cached) - → LLM call - → response - → _honcho_fire_prefetch() (daemon threads, turn end) - → prefetch_context() thread ──┐ - → prefetch_dialectic() thread ─┴→ _context_cache / _dialectic_cache -``` - -### openclaw-honcho: hook-based plugin - -The plugin registers hooks against OpenClaw's event bus. Context is fetched synchronously inside `before_prompt_build` on every turn. Message capture happens in `agent_end`. The multi-agent hierarchy is tracked via `subagent_spawned`. This model is correct but every turn pays a blocking Honcho round-trip before the LLM call can begin. - -Turn flow: - -``` -user message - → before_prompt_build (BLOCKING HTTP — every turn) - → session.context() - → system prompt assembled - → LLM call - → response - → agent_end hook - → session.addMessages() - → session.setMetadata() -``` - ---- - -## Diff table - -| Dimension | Hermes Agent | openclaw-honcho | -|---|---|---| -| **Context injection timing** | Once per session (cached). Zero HTTP on response path after turn 1. | Every turn, blocking. Fresh context per turn but adds latency. | -| **Prefetch strategy** | Daemon threads fire at turn end; consumed next turn from cache. | None. Blocking call at prompt-build time. | -| **Dialectic (peer.chat)** | Prefetched async; result injected into system prompt next turn. | On-demand via `honcho_recall` / `honcho_analyze` tools. | -| **Reasoning level** | Dynamic: scales with message length. Floor = config default. Cap = "high". | Fixed per tool: recall=minimal, analyze=medium. | -| **Memory modes** | `user_memory_mode` / `agent_memory_mode`: hybrid / honcho / local. | None. Always writes to Honcho. | -| **Write frequency** | async (background queue), turn, session, N turns. | After every agent_end (no control). | -| **AI peer identity** | `observe_me=True`, `seed_ai_identity()`, `get_ai_representation()`, SOUL.md → AI peer. | Agent files uploaded to agent peer at setup. No ongoing self-observation. | -| **Context scope** | User peer + AI peer representation, both injected. | User peer (owner) representation + conversation summary. `peerPerspective` on context call. | -| **Session naming** | per-directory / global / manual map / title-based. | Derived from platform session key. | -| **Multi-agent** | Single-agent only. | Parent observer hierarchy via `subagent_spawned`. | -| **Tool surface** | Single `query_user_context` tool (on-demand dialectic). | 6 tools: session, profile, search, context (fast) + recall, analyze (LLM). | -| **Platform metadata** | Not stripped. | Explicitly stripped before Honcho storage. | -| **Message dedup** | None. | `lastSavedIndex` in session metadata prevents re-sending. | -| **CLI surface in prompt** | Management commands injected into system prompt. Agent knows its own CLI. | Not injected. | -| **AI peer name in identity** | Replaces "Hermes Agent" in DEFAULT_AGENT_IDENTITY when configured. | Not implemented. | -| **QMD / local file search** | Not implemented. | Passthrough tools when QMD backend configured. | -| **Workspace metadata** | Not implemented. | `agentPeerMap` in workspace metadata tracks agent→peer ID. | - ---- - -## Patterns - -Six patterns from Hermes are worth adopting in any Honcho integration. Each is described as an integration-agnostic interface. - -**Hermes contributes:** -- Async prefetch (zero-latency) -- Dynamic reasoning level -- Per-peer memory modes -- AI peer identity formation -- Session naming strategies -- CLI surface injection - -**openclaw-honcho contributes back (Hermes should adopt):** -- `lastSavedIndex` dedup -- Platform metadata stripping -- Multi-agent observer hierarchy -- `peerPerspective` on `context()` -- Tiered tool surface (fast/LLM) -- Workspace `agentPeerMap` - ---- - -## Spec: async prefetch - -### Problem - -Calling `session.context()` and `peer.chat()` synchronously before each LLM call adds 200–800ms of Honcho round-trip latency to every turn. - -### Pattern - -Fire both calls as non-blocking background work at the **end** of each turn. Store results in a per-session cache keyed by session ID. At the **start** of the next turn, pop from cache — the HTTP is already done. First turn is cold (empty cache); all subsequent turns are zero-latency on the response path. - -### Interface contract - -```typescript -interface AsyncPrefetch { - // Fire context + dialectic fetches at turn end. Non-blocking. - firePrefetch(sessionId: string, userMessage: string): void; - - // Pop cached results at turn start. Returns empty if cache is cold. - popContextResult(sessionId: string): ContextResult | null; - popDialecticResult(sessionId: string): string | null; -} - -type ContextResult = { - representation: string; - card: string[]; - aiRepresentation?: string; // AI peer context if enabled - summary?: string; // conversation summary if fetched -}; -``` - -### Implementation notes - -- **Python:** `threading.Thread(daemon=True)`. Write to `dict[session_id, result]` — GIL makes this safe for simple writes. -- **TypeScript:** `Promise` stored in `Map>`. Await at pop time. If not resolved yet, return null — do not block. -- The pop is destructive: clears the cache entry after reading so stale data never accumulates. -- Prefetch should also fire on first turn (even though it won't be consumed until turn 2). - -### openclaw-honcho adoption - -Move `session.context()` from `before_prompt_build` to a post-`agent_end` background task. Store result in `state.contextCache`. In `before_prompt_build`, read from cache instead of calling Honcho. If cache is empty (turn 1), inject nothing — the prompt is still valid without Honcho context on the first turn. - ---- - -## Spec: dynamic reasoning level - -### Problem - -Honcho's dialectic endpoint supports reasoning levels from `minimal` to `max`. A fixed level per tool wastes budget on simple queries and under-serves complex ones. - -### Pattern - -Select the reasoning level dynamically based on the user's message. Use the configured default as a floor. Bump by message length. Cap auto-selection at `high` — never select `max` automatically. - -### Logic - -``` -< 120 chars → default (typically "low") -120–400 chars → one level above default (cap at "high") -> 400 chars → two levels above default (cap at "high") -``` - -### Config key - -Add `dialecticReasoningLevel` (string, default `"low"`). This sets the floor. The dynamic bump always applies on top. - -### openclaw-honcho adoption - -Apply in `honcho_recall` and `honcho_analyze`: replace fixed `reasoningLevel` with the dynamic selector. `honcho_recall` uses floor `"minimal"`, `honcho_analyze` uses floor `"medium"` — both still bump with message length. - ---- - -## Spec: per-peer memory modes - -### Problem - -Users want independent control over whether user context and agent context are written locally, to Honcho, or both. - -### Modes - -| Mode | Effect | -|---|---| -| `hybrid` | Write to both local files and Honcho (default) | -| `honcho` | Honcho only — disable corresponding local file writes | -| `local` | Local files only — skip Honcho sync for this peer | - -### Config schema - -```json -{ - "memoryMode": "hybrid", - "userMemoryMode": "honcho", - "agentMemoryMode": "hybrid" -} -``` - -Resolution order: per-peer field wins → shorthand `memoryMode` → default `"hybrid"`. - -### Effect on Honcho sync - -- `userMemoryMode=local`: skip adding user peer messages to Honcho -- `agentMemoryMode=local`: skip adding assistant peer messages to Honcho -- Both local: skip `session.addMessages()` entirely -- `userMemoryMode=honcho`: disable local USER.md writes -- `agentMemoryMode=honcho`: disable local MEMORY.md / SOUL.md writes - ---- - -## Spec: AI peer identity formation - -### Problem - -Honcho builds the user's representation organically by observing what the user says. The same mechanism exists for the AI peer — but only if `observe_me=True` is set for the agent peer. Without it, the agent peer accumulates nothing. - -Additionally, existing persona files (SOUL.md, IDENTITY.md) should seed the AI peer's Honcho representation at first activation. - -### Part A: observe_me=True for agent peer - -```typescript -await session.addPeers([ - [ownerPeer.id, { observeMe: true, observeOthers: false }], - [agentPeer.id, { observeMe: true, observeOthers: true }], // was false -]); -``` - -One-line change. Foundational. Without it, the AI peer representation stays empty regardless of what the agent says. - -### Part B: seedAiIdentity() - -```typescript -async function seedAiIdentity( - agentPeer: Peer, - content: string, - source: string -): Promise { - const wrapped = [ - ``, - `${source}`, - ``, - content.trim(), - ``, - ].join("\n"); - - await agentPeer.addMessage("assistant", wrapped); - return true; -} -``` - -### Part C: migrate agent files at setup - -During `honcho setup`, upload agent-self files (SOUL.md, IDENTITY.md, AGENTS.md) to the agent peer via `seedAiIdentity()` instead of `session.uploadFile()`. This routes content through Honcho's observation pipeline. - -### Part D: AI peer name in identity - -When the agent has a configured name, prepend it to the injected system prompt: - -```typescript -const namePrefix = agentName ? `You are ${agentName}.\n\n` : ""; -return { systemPrompt: namePrefix + "## User Memory Context\n\n" + sections }; -``` - -### CLI surface - -``` -honcho identity # seed from file -honcho identity --show # show current AI peer representation -``` - ---- - -## Spec: session naming strategies - -### Problem - -A single global session means every project shares the same Honcho context. Per-directory sessions provide isolation without requiring users to name sessions manually. - -### Strategies - -| Strategy | Session key | When to use | -|---|---|---| -| `per-directory` | basename of CWD | Default. Each project gets its own session. | -| `global` | fixed string `"global"` | Single cross-project session. | -| manual map | user-configured per path | `sessions` config map overrides directory basename. | -| title-based | sanitized session title | When agent supports named sessions set mid-conversation. | - -### Config schema - -```json -{ - "sessionStrategy": "per-directory", - "sessionPeerPrefix": false, - "sessions": { - "/home/user/projects/foo": "foo-project" - } -} -``` - -### CLI surface - -``` -honcho sessions # list all mappings -honcho map # map cwd to session name -honcho map # no-arg = list mappings -``` - -Resolution order: manual map → session title → directory basename → platform key. - ---- - -## Spec: CLI surface injection - -### Problem - -When a user asks "how do I change my memory settings?" the agent either hallucinates or says it doesn't know. The agent should know its own management interface. - -### Pattern - -When Honcho is active, append a compact command reference to the system prompt. Keep it under 300 chars. - -``` -# Honcho memory integration -Active. Session: {sessionKey}. Mode: {mode}. -Management commands: - honcho status — show config + connection - honcho mode [hybrid|honcho|local] — show or set memory mode - honcho sessions — list session mappings - honcho map — map directory to session - honcho identity [file] [--show] — seed or show AI identity - honcho setup — full interactive wizard -``` - ---- - -## openclaw-honcho checklist - -Ordered by impact: - -- [ ] **Async prefetch** — move `session.context()` out of `before_prompt_build` into post-`agent_end` background Promise -- [ ] **observe_me=True for agent peer** — one-line change in `session.addPeers()` -- [ ] **Dynamic reasoning level** — add helper; apply in `honcho_recall` and `honcho_analyze`; add `dialecticReasoningLevel` to config -- [ ] **Per-peer memory modes** — add `userMemoryMode` / `agentMemoryMode` to config; gate Honcho sync and local writes -- [ ] **seedAiIdentity()** — add helper; use during setup migration for SOUL.md / IDENTITY.md -- [ ] **Session naming strategies** — add `sessionStrategy`, `sessions` map, `sessionPeerPrefix` -- [ ] **CLI surface injection** — append command reference to `before_prompt_build` return value -- [ ] **honcho identity subcommand** — seed from file or `--show` current representation -- [ ] **AI peer name injection** — if `aiPeer` name configured, prepend to injected system prompt -- [ ] **honcho mode / sessions / map** — CLI parity with Hermes - -Already done in openclaw-honcho (do not re-implement): `lastSavedIndex` dedup, platform metadata stripping, multi-agent parent observer, `peerPerspective` on `context()`, tiered tool surface, workspace `agentPeerMap`, QMD passthrough, self-hosted Honcho. - ---- - -## nanobot-honcho checklist - -Greenfield integration. Start from openclaw-honcho's architecture and apply all Hermes patterns from day one. - -### Phase 1 — core correctness - -- [ ] Dual peer model (owner + agent peer), both with `observe_me=True` -- [ ] Message capture at turn end with `lastSavedIndex` dedup -- [ ] Platform metadata stripping before Honcho storage -- [ ] Async prefetch from day one — do not implement blocking context injection -- [ ] Legacy file migration at first activation (USER.md → owner peer, SOUL.md → `seedAiIdentity()`) - -### Phase 2 — configuration - -- [ ] Config schema: `apiKey`, `workspaceId`, `baseUrl`, `memoryMode`, `userMemoryMode`, `agentMemoryMode`, `dialecticReasoningLevel`, `sessionStrategy`, `sessions` -- [ ] Per-peer memory mode gating -- [ ] Dynamic reasoning level -- [ ] Session naming strategies - -### Phase 3 — tools and CLI - -- [ ] Tool surface: `honcho_profile`, `honcho_recall`, `honcho_analyze`, `honcho_search`, `honcho_context` -- [ ] CLI: `setup`, `status`, `sessions`, `map`, `mode`, `identity` -- [ ] CLI surface injection into system prompt -- [ ] AI peer name wired into agent identity diff --git a/docs/migration/openclaw.md b/docs/migration/openclaw.md deleted file mode 100644 index 30f2f97e4d6e..000000000000 --- a/docs/migration/openclaw.md +++ /dev/null @@ -1,142 +0,0 @@ -# Migrating from OpenClaw to Hermes Agent - -This guide covers how to import your OpenClaw settings, memories, skills, and API keys into Hermes Agent. - -## Three Ways to Migrate - -### 1. Automatic (during first-time setup) - -When you run `hermes setup` for the first time and Hermes detects `~/.openclaw`, it automatically offers to import your OpenClaw data before configuration begins. Just accept the prompt and everything is handled for you. - -### 2. CLI Command (quick, scriptable) - -```bash -hermes claw migrate # Preview then migrate (always shows preview first) -hermes claw migrate --dry-run # Preview only, no changes -hermes claw migrate --preset user-data # Migrate without API keys/secrets -hermes claw migrate --yes # Skip confirmation prompt -``` - -The migration always shows a full preview of what will be imported before making any changes. You review the preview and confirm before anything is written. - -**All options:** - -| Flag | Description | -|------|-------------| -| `--source PATH` | Path to OpenClaw directory (default: `~/.openclaw`) | -| `--dry-run` | Preview only — no files are modified | -| `--preset {user-data,full}` | Migration preset (default: `full`). `user-data` excludes secrets | -| `--overwrite` | Overwrite existing files (default: skip conflicts) | -| `--migrate-secrets` | Include allowlisted secrets (auto-enabled with `full` preset) | -| `--workspace-target PATH` | Copy workspace instructions (AGENTS.md) to this absolute path | -| `--skill-conflict {skip,overwrite,rename}` | How to handle skill name conflicts (default: `skip`) | -| `--yes`, `-y` | Skip confirmation prompts | - -### 3. Agent-Guided (interactive, with previews) - -Ask the agent to run the migration for you: - -``` -> Migrate my OpenClaw setup to Hermes -``` - -The agent will use the `openclaw-migration` skill to: -1. Run a preview first to show what would change -2. Ask about conflict resolution (SOUL.md, skills, etc.) -3. Let you choose between `user-data` and `full` presets -4. Execute the migration with your choices -5. Print a detailed summary of what was migrated - -## What Gets Migrated - -### `user-data` preset -| Item | Source | Destination | -|------|--------|-------------| -| SOUL.md | `~/.openclaw/workspace/SOUL.md` | `~/.hermes/SOUL.md` | -| Memory entries | `~/.openclaw/workspace/MEMORY.md` | `~/.hermes/memories/MEMORY.md` | -| User profile | `~/.openclaw/workspace/USER.md` | `~/.hermes/memories/USER.md` | -| Skills | `~/.openclaw/workspace/skills/` | `~/.hermes/skills/openclaw-imports/` | -| Command allowlist | `~/.openclaw/workspace/exec_approval_patterns.yaml` | Merged into `~/.hermes/config.yaml` | -| Messaging settings | `~/.openclaw/config.yaml` (TELEGRAM_ALLOWED_USERS, MESSAGING_CWD) | `~/.hermes/.env` | -| TTS assets | `~/.openclaw/workspace/tts/` | `~/.hermes/tts/` | - -Workspace files are also checked at `workspace.default/` and `workspace-main/` as fallback paths (OpenClaw renamed `workspace/` to `workspace-main/` in recent versions). - -### `full` preset (adds to `user-data`) -| Item | Source | Destination | -|------|--------|-------------| -| Telegram bot token | `openclaw.json` channels config | `~/.hermes/.env` | -| OpenRouter API key | `.env`, `openclaw.json`, or `openclaw.json["env"]` | `~/.hermes/.env` | -| OpenAI API key | `.env`, `openclaw.json`, or `openclaw.json["env"]` | `~/.hermes/.env` | -| Anthropic API key | `.env`, `openclaw.json`, or `openclaw.json["env"]` | `~/.hermes/.env` | -| ElevenLabs API key | `.env`, `openclaw.json`, or `openclaw.json["env"]` | `~/.hermes/.env` | - -API keys are searched across four sources: inline config values, `~/.openclaw/.env`, the `openclaw.json` `"env"` sub-object, and per-agent auth profiles. - -Only allowlisted secrets are ever imported. Other credentials are skipped and reported. - -## OpenClaw Schema Compatibility - -The migration handles both old and current OpenClaw config layouts: - -- **Channel tokens**: Reads from flat paths (`channels.telegram.botToken`) and the newer `accounts.default` layout (`channels.telegram.accounts.default.botToken`) -- **TTS provider**: OpenClaw renamed "edge" to "microsoft" — both are recognized and mapped to Hermes' "edge" -- **Provider API types**: Both short (`openai`, `anthropic`) and hyphenated (`openai-completions`, `anthropic-messages`, `google-generative-ai`) values are mapped correctly -- **thinkingDefault**: All enum values are handled including newer ones (`minimal`, `xhigh`, `adaptive`) -- **Matrix**: Uses `accessToken` field (not `botToken`) -- **SecretRef formats**: Plain strings, env templates (`${VAR}`), and `source: "env"` SecretRefs are resolved. `source: "file"` and `source: "exec"` SecretRefs produce a warning — add those keys manually after migration. - -## Conflict Handling - -By default, the migration **will not overwrite** existing Hermes data: - -- **SOUL.md** — skipped if one already exists in `~/.hermes/` -- **Memory entries** — skipped if memories already exist (to avoid duplicates) -- **Skills** — skipped if a skill with the same name already exists -- **API keys** — skipped if the key is already set in `~/.hermes/.env` - -To overwrite conflicts, use `--overwrite`. The migration creates backups before overwriting. - -For skills, you can also use `--skill-conflict rename` to import conflicting skills under a new name (e.g., `skill-name-imported`). - -## Migration Report - -Every migration produces a report showing: -- **Migrated items** — what was successfully imported -- **Conflicts** — items skipped because they already exist -- **Skipped items** — items not found in the source -- **Errors** — items that failed to import - -For executed migrations, the full report is saved to `~/.hermes/migration/openclaw//`. - -## Post-Migration Notes - -- **Skills require a new session** — imported skills take effect after restarting your agent or starting a new chat. -- **WhatsApp requires re-pairing** — WhatsApp uses QR-code pairing, not token-based auth. Run `hermes whatsapp` to pair. -- **Archive cleanup** — after migration, you'll be offered to rename `~/.openclaw/` to `.openclaw.pre-migration/` to prevent state confusion. You can also run `hermes claw cleanup` later. - -## Troubleshooting - -### "OpenClaw directory not found" -The migration looks for `~/.openclaw` by default, then tries `~/.clawdbot` and `~/.moltbot`. If your OpenClaw is installed elsewhere, use `--source`: -```bash -hermes claw migrate --source /path/to/.openclaw -``` - -### "Migration script not found" -The migration script ships with Hermes Agent. If you installed via pip (not git clone), the `optional-skills/` directory may not be present. Install the skill from the Skills Hub: -```bash -hermes skills install openclaw-migration -``` - -### Memory overflow -If your OpenClaw MEMORY.md or USER.md exceeds Hermes' character limits, excess entries are exported to an overflow file in the migration report directory. You can manually review and add the most important ones. - -### API keys not found -Keys might be stored in different places depending on your OpenClaw setup: -- `~/.openclaw/.env` file -- Inline in `openclaw.json` under `models.providers.*.apiKey` -- In `openclaw.json` under the `"env"` or `"env.vars"` sub-objects -- In `~/.openclaw/agents/main/agent/auth-profiles.json` - -The migration checks all four. If keys use `source: "file"` or `source: "exec"` SecretRefs, they can't be resolved automatically — add them via `hermes config set`. diff --git a/docs/plans/2026-03-16-pricing-accuracy-architecture-design.md b/docs/plans/2026-03-16-pricing-accuracy-architecture-design.md deleted file mode 100644 index a75f14ff5aac..000000000000 --- a/docs/plans/2026-03-16-pricing-accuracy-architecture-design.md +++ /dev/null @@ -1,608 +0,0 @@ -# Pricing Accuracy Architecture - -Date: 2026-03-16 - -## Goal - -Hermes should only show dollar costs when they are backed by an official source for the user's actual billing path. - -This design replaces the current static, heuristic pricing flow in: - -- `run_agent.py` -- `agent/usage_pricing.py` -- `agent/insights.py` -- `cli.py` - -with a provider-aware pricing system that: - -- handles cache billing correctly -- distinguishes `actual` vs `estimated` vs `included` vs `unknown` -- reconciles post-hoc costs when providers expose authoritative billing data -- supports direct providers, OpenRouter, subscriptions, enterprise pricing, and custom endpoints - -## Problems In The Current Design - -Current Hermes behavior has four structural issues: - -1. It stores only `prompt_tokens` and `completion_tokens`, which is insufficient for providers that bill cache reads and cache writes separately. -2. It uses a static model price table and fuzzy heuristics, which can drift from current official pricing. -3. It assumes public API list pricing matches the user's real billing path. -4. It has no distinction between live estimates and reconciled billed cost. - -## Design Principles - -1. Normalize usage before pricing. -2. Never fold cached tokens into plain input cost. -3. Track certainty explicitly. -4. Treat the billing path as part of the model identity. -5. Prefer official machine-readable sources over scraped docs. -6. Use post-hoc provider cost APIs when available. -7. Show `n/a` rather than inventing precision. - -## High-Level Architecture - -The new system has four layers: - -1. `usage_normalization` - Converts raw provider usage into a canonical usage record. -2. `pricing_source_resolution` - Determines the billing path, source of truth, and applicable pricing source. -3. `cost_estimation_and_reconciliation` - Produces an immediate estimate when possible, then replaces or annotates it with actual billed cost later. -4. `presentation` - `/usage`, `/insights`, and the status bar display cost with certainty metadata. - -## Canonical Usage Record - -Add a canonical usage model that every provider path maps into before any pricing math happens. - -Suggested structure: - -```python -@dataclass -class CanonicalUsage: - provider: str - billing_provider: str - model: str - billing_route: str - - input_tokens: int = 0 - output_tokens: int = 0 - cache_read_tokens: int = 0 - cache_write_tokens: int = 0 - reasoning_tokens: int = 0 - request_count: int = 1 - - raw_usage: dict[str, Any] | None = None - raw_usage_fields: dict[str, str] | None = None - computed_fields: set[str] | None = None - - provider_request_id: str | None = None - provider_generation_id: str | None = None - provider_response_id: str | None = None -``` - -Rules: - -- `input_tokens` means non-cached input only. -- `cache_read_tokens` and `cache_write_tokens` are never merged into `input_tokens`. -- `output_tokens` excludes cache metrics. -- `reasoning_tokens` is telemetry unless a provider officially bills it separately. - -This is the same normalization pattern used by `opencode`, extended with provenance and reconciliation ids. - -## Provider Normalization Rules - -### OpenAI Direct - -Source usage fields: - -- `prompt_tokens` -- `completion_tokens` -- `prompt_tokens_details.cached_tokens` - -Normalization: - -- `cache_read_tokens = cached_tokens` -- `input_tokens = prompt_tokens - cached_tokens` -- `cache_write_tokens = 0` unless OpenAI exposes it in the relevant route -- `output_tokens = completion_tokens` - -### Anthropic Direct - -Source usage fields: - -- `input_tokens` -- `output_tokens` -- `cache_read_input_tokens` -- `cache_creation_input_tokens` - -Normalization: - -- `input_tokens = input_tokens` -- `output_tokens = output_tokens` -- `cache_read_tokens = cache_read_input_tokens` -- `cache_write_tokens = cache_creation_input_tokens` - -### OpenRouter - -Estimate-time usage normalization should use the response usage payload with the same rules as the underlying provider when possible. - -Reconciliation-time records should also store: - -- OpenRouter generation id -- native token fields when available -- `total_cost` -- `cache_discount` -- `upstream_inference_cost` -- `is_byok` - -### Gemini / Vertex - -Use official Gemini or Vertex usage fields where available. - -If cached content tokens are exposed: - -- map them to `cache_read_tokens` - -If a route exposes no cache creation metric: - -- store `cache_write_tokens = 0` -- preserve the raw usage payload for later extension - -### DeepSeek And Other Direct Providers - -Normalize only the fields that are officially exposed. - -If a provider does not expose cache buckets: - -- do not infer them unless the provider explicitly documents how to derive them - -### Subscription / Included-Cost Routes - -These still use the canonical usage model. - -Tokens are tracked normally. Cost depends on billing mode, not on whether usage exists. - -## Billing Route Model - -Hermes must stop keying pricing solely by `model`. - -Introduce a billing route descriptor: - -```python -@dataclass -class BillingRoute: - provider: str - base_url: str | None - model: str - billing_mode: str - organization_hint: str | None = None -``` - -`billing_mode` values: - -- `official_cost_api` -- `official_generation_api` -- `official_models_api` -- `official_docs_snapshot` -- `subscription_included` -- `user_override` -- `custom_contract` -- `unknown` - -Examples: - -- OpenAI direct API with Costs API access: `official_cost_api` -- Anthropic direct API with Usage & Cost API access: `official_cost_api` -- OpenRouter request before reconciliation: `official_models_api` -- OpenRouter request after generation lookup: `official_generation_api` -- GitHub Copilot style subscription route: `subscription_included` -- local OpenAI-compatible server: `unknown` -- enterprise contract with configured rates: `custom_contract` - -## Cost Status Model - -Every displayed cost should have: - -```python -@dataclass -class CostResult: - amount_usd: Decimal | None - status: Literal["actual", "estimated", "included", "unknown"] - source: Literal[ - "provider_cost_api", - "provider_generation_api", - "provider_models_api", - "official_docs_snapshot", - "user_override", - "custom_contract", - "none", - ] - label: str - fetched_at: datetime | None - pricing_version: str | None - notes: list[str] -``` - -Presentation rules: - -- `actual`: show dollar amount as final -- `estimated`: show dollar amount with estimate labeling -- `included`: show `included` or `$0.00 (included)` depending on UX choice -- `unknown`: show `n/a` - -## Official Source Hierarchy - -Resolve cost using this order: - -1. Request-level or account-level official billed cost -2. Official machine-readable model pricing -3. Official docs snapshot -4. User override or custom contract -5. Unknown - -The system must never skip to a lower level if a higher-confidence source exists for the current billing route. - -## Provider-Specific Truth Rules - -### OpenAI Direct - -Preferred truth: - -1. Costs API for reconciled spend -2. Official pricing page for live estimate - -### Anthropic Direct - -Preferred truth: - -1. Usage & Cost API for reconciled spend -2. Official pricing docs for live estimate - -### OpenRouter - -Preferred truth: - -1. `GET /api/v1/generation` for reconciled `total_cost` -2. `GET /api/v1/models` pricing for live estimate - -Do not use underlying provider public pricing as the source of truth for OpenRouter billing. - -### Gemini / Vertex - -Preferred truth: - -1. official billing export or billing API for reconciled spend when available for the route -2. official pricing docs for estimate - -### DeepSeek - -Preferred truth: - -1. official machine-readable cost source if available in the future -2. official pricing docs snapshot today - -### Subscription-Included Routes - -Preferred truth: - -1. explicit route config marking the model as included in subscription - -These should display `included`, not an API list-price estimate. - -### Custom Endpoint / Local Model - -Preferred truth: - -1. user override -2. custom contract config -3. unknown - -These should default to `unknown`. - -## Pricing Catalog - -Replace the current `MODEL_PRICING` dict with a richer pricing catalog. - -Suggested record: - -```python -@dataclass -class PricingEntry: - provider: str - route_pattern: str - model_pattern: str - - input_cost_per_million: Decimal | None = None - output_cost_per_million: Decimal | None = None - cache_read_cost_per_million: Decimal | None = None - cache_write_cost_per_million: Decimal | None = None - request_cost: Decimal | None = None - image_cost: Decimal | None = None - - source: str = "official_docs_snapshot" - source_url: str | None = None - fetched_at: datetime | None = None - pricing_version: str | None = None -``` - -The catalog should be route-aware: - -- `openai:gpt-5` -- `anthropic:claude-opus-4-6` -- `openrouter:anthropic/claude-opus-4.6` -- `copilot:gpt-4o` - -This avoids conflating direct-provider billing with aggregator billing. - -## Pricing Sync Architecture - -Introduce a pricing sync subsystem instead of manually maintaining a single hardcoded table. - -Suggested modules: - -- `agent/pricing/catalog.py` -- `agent/pricing/sources.py` -- `agent/pricing/sync.py` -- `agent/pricing/reconcile.py` -- `agent/pricing/types.py` - -### Sync Sources - -- OpenRouter models API -- official provider docs snapshots where no API exists -- user overrides from config - -### Sync Output - -Cache pricing entries locally with: - -- source URL -- fetch timestamp -- version/hash -- confidence/source type - -### Sync Frequency - -- startup warm cache -- background refresh every 6 to 24 hours depending on source -- manual `hermes pricing sync` - -## Reconciliation Architecture - -Live requests may produce only an estimate initially. Hermes should reconcile them later when a provider exposes actual billed cost. - -Suggested flow: - -1. Agent call completes. -2. Hermes stores canonical usage plus reconciliation ids. -3. Hermes computes an immediate estimate if a pricing source exists. -4. A reconciliation worker fetches actual cost when supported. -5. Session and message records are updated with `actual` cost. - -This can run: - -- inline for cheap lookups -- asynchronously for delayed provider accounting - -## Persistence Changes - -Session storage should stop storing only aggregate prompt/completion totals. - -Add fields for both usage and cost certainty: - -- `input_tokens` -- `output_tokens` -- `cache_read_tokens` -- `cache_write_tokens` -- `reasoning_tokens` -- `estimated_cost_usd` -- `actual_cost_usd` -- `cost_status` -- `cost_source` -- `pricing_version` -- `billing_provider` -- `billing_mode` - -If schema expansion is too large for one PR, add a new pricing events table: - -```text -session_cost_events - id - session_id - request_id - provider - model - billing_mode - input_tokens - output_tokens - cache_read_tokens - cache_write_tokens - estimated_cost_usd - actual_cost_usd - cost_status - cost_source - pricing_version - created_at - updated_at -``` - -## Hermes Touchpoints - -### `run_agent.py` - -Current responsibility: - -- parse raw provider usage -- update session token counters - -New responsibility: - -- build `CanonicalUsage` -- update canonical counters -- store reconciliation ids -- emit usage event to pricing subsystem - -### `agent/usage_pricing.py` - -Current responsibility: - -- static lookup table -- direct cost arithmetic - -New responsibility: - -- move or replace with pricing catalog facade -- no fuzzy model-family heuristics -- no direct pricing without billing-route context - -### `cli.py` - -Current responsibility: - -- compute session cost directly from prompt/completion totals - -New responsibility: - -- display `CostResult` -- show status badges: - - `actual` - - `estimated` - - `included` - - `n/a` - -### `agent/insights.py` - -Current responsibility: - -- recompute historical estimates from static pricing - -New responsibility: - -- aggregate stored pricing events -- prefer actual cost over estimate -- surface estimates only when reconciliation is unavailable - -## UX Rules - -### Status Bar - -Show one of: - -- `$1.42` -- `~$1.42` -- `included` -- `cost n/a` - -Where: - -- `$1.42` means `actual` -- `~$1.42` means `estimated` -- `included` means subscription-backed or explicitly zero-cost route -- `cost n/a` means unknown - -### `/usage` - -Show: - -- token buckets -- estimated cost -- actual cost if available -- cost status -- pricing source - -### `/insights` - -Aggregate: - -- actual cost totals -- estimated-only totals -- unknown-cost sessions count -- included-cost sessions count - -## Config And Overrides - -Add user-configurable pricing overrides in config: - -```yaml -pricing: - mode: hybrid - sync_on_startup: true - sync_interval_hours: 12 - overrides: - - provider: openrouter - model: anthropic/claude-opus-4.6 - billing_mode: custom_contract - input_cost_per_million: 4.25 - output_cost_per_million: 22.0 - cache_read_cost_per_million: 0.5 - cache_write_cost_per_million: 6.0 - included_routes: - - provider: copilot - model: "*" - - provider: codex-subscription - model: "*" -``` - -Overrides must win over catalog defaults for the matching billing route. - -## Rollout Plan - -### Phase 1 - -- add canonical usage model -- split cache token buckets in `run_agent.py` -- stop pricing cache-inflated prompt totals -- preserve current UI with improved backend math - -### Phase 2 - -- add route-aware pricing catalog -- integrate OpenRouter models API sync -- add `estimated` vs `included` vs `unknown` - -### Phase 3 - -- add reconciliation for OpenRouter generation cost -- add actual cost persistence -- update `/insights` to prefer actual cost - -### Phase 4 - -- add direct OpenAI and Anthropic reconciliation paths -- add user overrides and contract pricing -- add pricing sync CLI command - -## Testing Strategy - -Add tests for: - -- OpenAI cached token subtraction -- Anthropic cache read/write separation -- OpenRouter estimated vs actual reconciliation -- subscription-backed models showing `included` -- custom endpoints showing `n/a` -- override precedence -- stale catalog fallback behavior - -Current tests that assume heuristic pricing should be replaced with route-aware expectations. - -## Non-Goals - -- exact enterprise billing reconstruction without an official source or user override -- backfilling perfect historical cost for old sessions that lack cache bucket data -- scraping arbitrary provider web pages at request time - -## Recommendation - -Do not expand the existing `MODEL_PRICING` dict. - -That path cannot satisfy the product requirement. Hermes should instead migrate to: - -- canonical usage normalization -- route-aware pricing sources -- estimate-then-reconcile cost lifecycle -- explicit certainty states in the UI - -This is the minimum architecture that makes the statement "Hermes pricing is backed by official sources where possible, and otherwise clearly labeled" defensible. diff --git a/docs/skins/example-skin.yaml b/docs/skins/example-skin.yaml deleted file mode 100644 index 612c841eb33c..000000000000 --- a/docs/skins/example-skin.yaml +++ /dev/null @@ -1,89 +0,0 @@ -# ============================================================================ -# Hermes Agent — Example Skin Template -# ============================================================================ -# -# Copy this file to ~/.hermes/skins/.yaml to create a custom skin. -# All fields are optional — missing values inherit from the default skin. -# Activate with: /skin or display.skin: in config.yaml -# -# See hermes_cli/skin_engine.py for the full schema reference. -# ============================================================================ - -# Required: unique skin name (used in /skin command and config) -name: example -description: An example custom skin — copy and modify this template - -# ── Colors ────────────────────────────────────────────────────────────────── -# Hex color values for Rich markup. These control the CLI's visual palette. -colors: - # Banner panel (the startup welcome box) - banner_border: "#CD7F32" # Panel border - banner_title: "#FFD700" # Panel title text - banner_accent: "#FFBF00" # Section headers (Available Tools, Skills, etc.) - banner_dim: "#B8860B" # Dim/muted text (separators, model info) - banner_text: "#FFF8DC" # Body text (tool names, skill names) - - # UI elements - ui_accent: "#FFBF00" # General accent color - ui_label: "#4dd0e1" # Labels - ui_ok: "#4caf50" # Success indicators - ui_error: "#ef5350" # Error indicators - ui_warn: "#ffa726" # Warning indicators - - # Input area - prompt: "#FFF8DC" # Prompt text color - input_rule: "#CD7F32" # Horizontal rule around input - - # Response box - response_border: "#FFD700" # Response box border (ANSI color) - - # Session display - session_label: "#DAA520" # Session label - session_border: "#8B8682" # Session ID dim color - -# ── Spinner ───────────────────────────────────────────────────────────────── -# Customize the animated spinner shown during API calls and tool execution. -spinner: - # Faces shown while waiting for the API response - waiting_faces: - - "(｡◕‿◕｡)" - - "(◕‿◕✿)" - - "٩(◕‿◕｡)۶" - - # Faces shown during extended thinking/reasoning - thinking_faces: - - "(｡•́︿•̀｡)" - - "(◔_◔)" - - "(¬‿¬)" - - # Verbs used in spinner messages (e.g., "pondering your request...") - thinking_verbs: - - "pondering" - - "contemplating" - - "musing" - - "ruminating" - - # Optional: left/right decorations around the spinner - # Each entry is a [left, right] pair. Omit entirely for no wings. - # wings: - # - ["⟪⚔", "⚔⟫"] - # - ["⟪▲", "▲⟫"] - -# ── Branding ──────────────────────────────────────────────────────────────── -# Text strings used throughout the CLI interface. -branding: - agent_name: "Hermes Agent" # Banner title, about display - welcome: "Welcome! Type your message or /help for commands." - goodbye: "Goodbye! ⚕" # Exit message - response_label: " ⚕ Hermes " # Response box header label - prompt_symbol: "❯ " # Input prompt symbol - help_header: "(^_^)? Available Commands" # /help header text - -# ── Tool Output ───────────────────────────────────────────────────────────── -# Character used as the prefix for tool output lines. -# Default is "┊" (thin dotted vertical line). Some alternatives: -# "╎" (light triple dash vertical) -# "▏" (left one-eighth block) -# "│" (box drawing light vertical) -# "┃" (box drawing heavy vertical) -tool_prefix: "┊" diff --git a/docs/specs/container-cli-review-fixes.md b/docs/specs/container-cli-review-fixes.md deleted file mode 100644 index 0eb9070dbf52..000000000000 --- a/docs/specs/container-cli-review-fixes.md +++ /dev/null @@ -1,329 +0,0 @@ -# Container-Aware CLI Review Fixes Spec - -**PR:** NousResearch/hermes-agent#7543 -**Review:** cursor[bot] bugbot review (4094049442) + two prior rounds -**Date:** 2026-04-12 -**Branch:** `feat/container-aware-cli-clean` - -## Review Issues Summary - -Six issues were raised across three bugbot review rounds. Three were fixed in intermediate commits (38277a6a, 726cf90f). This spec addresses remaining design concerns surfaced by those reviews and simplifies the implementation based on interview decisions. - -| # | Issue | Severity | Status | -|---|-------|----------|--------| -| 1 | `os.execvp` retry loop unreachable | Medium | Fixed in 79e8cd12 (switched to subprocess.run) | -| 2 | Redundant `shutil.which("sudo")` | Medium | Fixed in 38277a6a (reuses `sudo` var) | -| 3 | Missing `chown -h` on symlink update | Low | Fixed in 38277a6a | -| 4 | Container routing after `parse_args()` | High | Fixed in 726cf90f | -| 5 | Hardcoded `/home/${user}` | Medium | Fixed in 726cf90f | -| 6 | Group membership not gated on `container.enable` | Low | Fixed in 726cf90f | - -The mechanical fixes are in place but the overall design needs revision. The retry loop, error swallowing, and process model have deeper issues than what the bugbot flagged. - ---- - -## Spec: Revised `_exec_in_container` - -### Design Principles - -1. **Let it crash.** No silent fallbacks. If `.container-mode` exists but something goes wrong, the error propagates naturally (Python traceback). The only case where container routing is skipped is when `.container-mode` doesn't exist or `HERMES_DEV=1`. -2. **No retries.** Probe once for sudo, exec once. If it fails, docker/podman's stderr reaches the user verbatim. -3. **Completely transparent.** No error wrapping, no prefixes, no spinners. Docker's output goes straight through. -4. **`os.execvp` on the happy path.** Replace the Python process entirely so there's no idle parent during interactive sessions. Note: `execvp` never returns on success (process is replaced) and raises `OSError` on failure (it does not return a value). The container process's exit code becomes the process exit code by definition — no explicit propagation needed. -5. **One human-readable exception to "let it crash".** `subprocess.TimeoutExpired` from the sudo probe gets a specific catch with a readable message, since a raw traceback for "your Docker daemon is slow" is confusing. All other exceptions propagate naturally. - -### Execution Flow - -``` -1. get_container_exec_info() - - HERMES_DEV=1 → return None (skip routing) - - Inside container → return None (skip routing) - - .container-mode doesn't exist → return None (skip routing) - - .container-mode exists → parse and return dict - - .container-mode exists but malformed/unreadable → LET IT CRASH (no try/except) - -2. _exec_in_container(container_info, sys.argv[1:]) - a. shutil.which(backend) → if None, print "{backend} not found on PATH" and sys.exit(1) - b. Sudo probe: subprocess.run([runtime, "inspect", "--format", "ok", container_name], timeout=15) - - If succeeds → needs_sudo = False - - If fails → try subprocess.run([sudo, "-n", runtime, "inspect", ...], timeout=15) - - If succeeds → needs_sudo = True - - If fails → print error with sudoers hint (including why -n is required) and sys.exit(1) - - If TimeoutExpired → catch specifically, print human-readable message about slow daemon - c. Build exec_cmd: [sudo? + runtime, "exec", tty_flags, "-u", exec_user, env_flags, container, hermes_bin, *cli_args] - d. os.execvp(exec_cmd[0], exec_cmd) - - On success: process is replaced — Python is gone, container exit code IS the process exit code - - On OSError: let it crash (natural traceback) -``` - -### Changes to `hermes_cli/main.py` - -#### `_exec_in_container` — rewrite - -Remove: -- The entire retry loop (`max_retries`, `for attempt in range(...)`) -- Spinner logic (`"Waiting for container..."`, dots) -- Exit code classification (125/126/127 handling) -- `subprocess.run` for the exec call (keep it only for the sudo probe) -- Special TTY vs non-TTY retry counts -- The `time` import (no longer needed) - -Change: -- Use `os.execvp(exec_cmd[0], exec_cmd)` as the final call -- Keep the `subprocess` import only for the sudo probe -- Keep TTY detection for the `-it` vs `-i` flag -- Keep env var forwarding (TERM, COLORTERM, LANG, LC_ALL) -- Keep the sudo probe as-is (it's the one "smart" part) -- Bump probe `timeout` from 5s to 15s — cold podman on a loaded machine needs headroom -- Catch `subprocess.TimeoutExpired` specifically on both probe calls — print a readable message about the daemon being unresponsive instead of a raw traceback -- Expand the sudoers hint error message to explain *why* `-n` (non-interactive) is required: a password prompt would hang the CLI or break piped commands - -The function becomes roughly: - -```python -def _exec_in_container(container_info: dict, cli_args: list): - """Replace the current process with a command inside the managed container. - - Probes whether sudo is needed (rootful containers), then os.execvp - into the container. If exec fails, the OS error propagates naturally. - """ - import shutil - import subprocess - - backend = container_info["backend"] - container_name = container_info["container_name"] - exec_user = container_info["exec_user"] - hermes_bin = container_info["hermes_bin"] - - runtime = shutil.which(backend) - if not runtime: - print(f"Error: {backend} not found on PATH. Cannot route to container.", - file=sys.stderr) - sys.exit(1) - - # Probe whether we need sudo to see the rootful container. - # Timeout is 15s — cold podman on a loaded machine can take a while. - # TimeoutExpired is caught specifically for a human-readable message; - # all other exceptions propagate naturally. - needs_sudo = False - sudo = None - try: - probe = subprocess.run( - [runtime, "inspect", "--format", "ok", container_name], - capture_output=True, text=True, timeout=15, - ) - except subprocess.TimeoutExpired: - print( - f"Error: timed out waiting for {backend} to respond.\n" - f"The {backend} daemon may be unresponsive or starting up.", - file=sys.stderr, - ) - sys.exit(1) - - if probe.returncode != 0: - sudo = shutil.which("sudo") - if sudo: - try: - probe2 = subprocess.run( - [sudo, "-n", runtime, "inspect", "--format", "ok", container_name], - capture_output=True, text=True, timeout=15, - ) - except subprocess.TimeoutExpired: - print( - f"Error: timed out waiting for sudo {backend} to respond.", - file=sys.stderr, - ) - sys.exit(1) - - if probe2.returncode == 0: - needs_sudo = True - else: - print( - f"Error: container '{container_name}' not found via {backend}.\n" - f"\n" - f"The NixOS service runs the container as root. Your user cannot\n" - f"see it because {backend} uses per-user namespaces.\n" - f"\n" - f"Fix: grant passwordless sudo for {backend}. The -n (non-interactive)\n" - f"flag is required because the CLI calls sudo non-interactively —\n" - f"a password prompt would hang or break piped commands:\n" - f"\n" - f' security.sudo.extraRules = [{{\n' - f' users = [ "{os.getenv("USER", "your-user")}" ];\n' - f' commands = [{{ command = "{runtime}"; options = [ "NOPASSWD" ]; }}];\n' - f' }}];\n' - f"\n" - f"Or run: sudo hermes {' '.join(cli_args)}", - file=sys.stderr, - ) - sys.exit(1) - else: - print( - f"Error: container '{container_name}' not found via {backend}.\n" - f"The container may be running under root. Try: sudo hermes {' '.join(cli_args)}", - file=sys.stderr, - ) - sys.exit(1) - - is_tty = sys.stdin.isatty() - tty_flags = ["-it"] if is_tty else ["-i"] - - env_flags = [] - for var in ("TERM", "COLORTERM", "LANG", "LC_ALL"): - val = os.environ.get(var) - if val: - env_flags.extend(["-e", f"{var}={val}"]) - - cmd_prefix = [sudo, "-n", runtime] if needs_sudo else [runtime] - exec_cmd = ( - cmd_prefix + ["exec"] - + tty_flags - + ["-u", exec_user] - + env_flags - + [container_name, hermes_bin] - + cli_args - ) - - # execvp replaces this process entirely — it never returns on success. - # On failure it raises OSError, which propagates naturally. - os.execvp(exec_cmd[0], exec_cmd) -``` - -#### Container routing call site in `main()` — remove try/except - -Current: -```python -try: - from hermes_cli.config import get_container_exec_info - container_info = get_container_exec_info() - if container_info: - _exec_in_container(container_info, sys.argv[1:]) - sys.exit(1) # exec failed if we reach here -except SystemExit: - raise -except Exception: - pass # Container routing unavailable, proceed locally -``` - -Revised: -```python -from hermes_cli.config import get_container_exec_info -container_info = get_container_exec_info() -if container_info: - _exec_in_container(container_info, sys.argv[1:]) - # Unreachable: os.execvp never returns on success (process is replaced) - # and raises OSError on failure (which propagates as a traceback). - # This line exists only as a defensive assertion. - sys.exit(1) -``` - -No try/except. If `.container-mode` doesn't exist, `get_container_exec_info()` returns `None` and we skip routing. If it exists but is broken, the exception propagates with a natural traceback. - -Note: `sys.exit(1)` after `_exec_in_container` is dead code in all paths — `os.execvp` either replaces the process or raises. It's kept as a belt-and-suspenders assertion with a comment marking it unreachable, not as actual error handling. - -### Changes to `hermes_cli/config.py` - -#### `get_container_exec_info` — remove inner try/except - -Current code catches `(OSError, IOError)` and returns `None`. This silently hides permission errors, corrupt files, etc. - -Change: Remove the try/except around file reading. Keep the early returns for `HERMES_DEV=1` and `_is_inside_container()`. The `FileNotFoundError` from `open()` when `.container-mode` doesn't exist should still return `None` (this is the "container mode not enabled" case). All other exceptions propagate. - -```python -def get_container_exec_info() -> Optional[dict]: - if os.environ.get("HERMES_DEV") == "1": - return None - if _is_inside_container(): - return None - - container_mode_file = get_hermes_home() / ".container-mode" - - try: - with open(container_mode_file, "r") as f: - # ... parse key=value lines ... - except FileNotFoundError: - return None - # All other exceptions (PermissionError, malformed data, etc.) propagate - - return { ... } -``` - ---- - -## Spec: NixOS Module Changes - -### Symlink creation — simplify to two branches - -Current: 4 branches (symlink exists, directory exists, other file, doesn't exist). - -Revised: 2 branches. - -```bash -if [ -d "${symlinkPath}" ] && [ ! -L "${symlinkPath}" ]; then - # Real directory — back it up, then create symlink - _backup="${symlinkPath}.bak.$(date +%s)" - echo "hermes-agent: backing up existing ${symlinkPath} to $_backup" - mv "${symlinkPath}" "$_backup" -fi -# For everything else (symlink, doesn't exist, etc.) — just force-create -ln -sfn "${target}" "${symlinkPath}" -chown -h ${user}:${cfg.group} "${symlinkPath}" -``` - -`ln -sfn` handles: existing symlink (replaces), doesn't exist (creates), and after the `mv` above (creates). The only case that needs special handling is a real directory, because `ln -sfn` cannot atomically replace a directory. - -Note: there is a theoretical race between the `[ -d ... ]` check and the `mv` (something could create/remove the directory in between). In practice this is a NixOS activation script running as root during `nixos-rebuild switch` — no other process should be touching `~/.hermes` at that moment. Not worth adding locking for. - -### Sudoers — document, don't auto-configure - -Do NOT add `security.sudo.extraRules` to the module. Document the sudoers requirement in the module's description/comments and in the error message the CLI prints when sudo probe fails. - -### Group membership gating — keep as-is - -The fix in 726cf90f (`cfg.container.enable && cfg.container.hostUsers != []`) is correct. Leftover group membership when container mode is disabled is harmless. No cleanup needed. - ---- - -## Spec: Test Rewrite - -The existing test file (`tests/hermes_cli/test_container_aware_cli.py`) has 16 tests. With the simplified exec model, several are obsolete. - -### Tests to keep (update as needed) - -- `test_is_inside_container_dockerenv` — unchanged -- `test_is_inside_container_containerenv` — unchanged -- `test_is_inside_container_cgroup_docker` — unchanged -- `test_is_inside_container_false_on_host` — unchanged -- `test_get_container_exec_info_returns_metadata` — unchanged -- `test_get_container_exec_info_none_inside_container` — unchanged -- `test_get_container_exec_info_none_without_file` — unchanged -- `test_get_container_exec_info_skipped_when_hermes_dev` — unchanged -- `test_get_container_exec_info_not_skipped_when_hermes_dev_zero` — unchanged -- `test_get_container_exec_info_defaults` — unchanged -- `test_get_container_exec_info_docker_backend` — unchanged - -### Tests to add - -- `test_get_container_exec_info_crashes_on_permission_error` — verify that `PermissionError` propagates (no silent `None` return) -- `test_exec_in_container_calls_execvp` — verify `os.execvp` is called with correct args (runtime, tty flags, user, env, container, binary, cli args) -- `test_exec_in_container_sudo_probe_sets_prefix` — verify that when first probe fails and sudo probe succeeds, `os.execvp` is called with `sudo -n` prefix -- `test_exec_in_container_no_runtime_hard_fails` — keep existing, verify `sys.exit(1)` when `shutil.which` returns None -- `test_exec_in_container_non_tty_uses_i_only` — update to check `os.execvp` args instead of `subprocess.run` args -- `test_exec_in_container_probe_timeout_prints_message` — verify that `subprocess.TimeoutExpired` from the probe produces a human-readable error and `sys.exit(1)`, not a raw traceback -- `test_exec_in_container_container_not_running_no_sudo` — verify the path where runtime exists (`shutil.which` returns a path) but probe returns non-zero and no sudo is available. Should print the "container may be running under root" error. This is distinct from `no_runtime_hard_fails` which covers `shutil.which` returning None. - -### Tests to delete - -- `test_exec_in_container_tty_retries_on_container_failure` — retry loop removed -- `test_exec_in_container_non_tty_retries_silently_exits_126` — retry loop removed -- `test_exec_in_container_propagates_hermes_exit_code` — no subprocess.run to check exit codes; execvp replaces the process. Note: exit code propagation still works correctly — when `os.execvp` succeeds, the container's process *becomes* this process, so its exit code is the process exit code by OS semantics. No application code needed, no test needed. A comment in the function docstring documents this intent for future readers. - ---- - -## Out of Scope - -- Auto-configuring sudoers rules in the NixOS module -- Any changes to `get_container_exec_info` parsing logic beyond the try/except narrowing -- Changes to `.container-mode` file format -- Changes to the `HERMES_DEV=1` bypass -- Changes to container detection logic (`_is_inside_container`) diff --git a/environments/tool_context.py b/environments/tool_context.py index 10f537d72432..550c5e851c1d 100644 --- a/environments/tool_context.py +++ b/environments/tool_context.py @@ -53,7 +53,6 @@ def _run_tool_in_thread(tool_name: str, arguments: Dict[str, Any], task_id: str) try: loop = asyncio.get_running_loop() # We're in an async context -- need to run in thread - import concurrent.futures with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: future = pool.submit( handle_function_call, tool_name, arguments, task_id diff --git a/flake.lock b/flake.lock index 78ceba92d7e7..305b79526e03 100644 --- a/flake.lock +++ b/flake.lock @@ -36,6 +36,26 @@ "type": "github" } }, + "npm-lockfile-fix": { + "inputs": { + "nixpkgs": [ + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1775903712, + "narHash": "sha256-2GV79U6iVH4gKAPWYrxUReB0S41ty/Y3dBLquU8AlaA=", + "owner": "jeslie0", + "repo": "npm-lockfile-fix", + "rev": "c6093acb0c0548e0f9b8b3d82918823721930fe8", + "type": "github" + }, + "original": { + "owner": "jeslie0", + "repo": "npm-lockfile-fix", + "type": "github" + } + }, "pyproject-build-systems": { "inputs": { "nixpkgs": [ @@ -124,6 +144,7 @@ "inputs": { "flake-parts": "flake-parts", "nixpkgs": "nixpkgs", + "npm-lockfile-fix": "npm-lockfile-fix", "pyproject-build-systems": "pyproject-build-systems", "pyproject-nix": "pyproject-nix_2", "uv2nix": "uv2nix_2" diff --git a/flake.nix b/flake.nix index 919fa434dc63..fcb5eaa61992 100644 --- a/flake.nix +++ b/flake.nix @@ -19,11 +19,20 @@ url = "github:pyproject-nix/build-system-pkgs"; inputs.nixpkgs.follows = "nixpkgs"; }; + npm-lockfile-fix = { + url = "github:jeslie0/npm-lockfile-fix"; + inputs.nixpkgs.follows = "nixpkgs"; + }; }; - outputs = inputs: + outputs = + inputs: inputs.flake-parts.lib.mkFlake { inherit inputs; } { - systems = [ "x86_64-linux" "aarch64-linux" "aarch64-darwin" ]; + systems = [ + "x86_64-linux" + "aarch64-linux" + "aarch64-darwin" + ]; imports = [ ./nix/packages.nix diff --git a/gateway/builtin_hooks/boot_md.py b/gateway/builtin_hooks/boot_md.py index c4b6c2d46ac5..c2868a1e6360 100644 --- a/gateway/builtin_hooks/boot_md.py +++ b/gateway/builtin_hooks/boot_md.py @@ -18,9 +18,7 @@ """ import logging -import os import threading -from pathlib import Path logger = logging.getLogger("hooks.boot-md") diff --git a/gateway/channel_directory.py b/gateway/channel_directory.py index ae2beda9eff5..2489b718f830 100644 --- a/gateway/channel_directory.py +++ b/gateway/channel_directory.py @@ -100,7 +100,7 @@ def build_channel_directory(adapters: Dict[Any, Any]) -> Dict[str, Any]: def _build_discord(adapter) -> List[Dict[str, str]]: - """Enumerate all text channels the Discord bot can see.""" + """Enumerate all text channels and forum channels the Discord bot can see.""" channels = [] client = getattr(adapter, "_client", None) if not client: @@ -119,6 +119,15 @@ def _build_discord(adapter) -> List[Dict[str, str]]: "guild": guild.name, "type": "channel", }) + # Forum channels (type 15) — creating a message auto-spawns a thread post. + forums = getattr(guild, "forum_channels", None) or [] + for ch in forums: + channels.append({ + "id": str(ch.id), + "name": ch.name, + "guild": guild.name, + "type": "forum", + }) # Also include DM-capable users we've interacted with is not # feasible via guild enumeration; those come from sessions. @@ -191,6 +200,15 @@ def load_directory() -> Dict[str, Any]: return {"updated_at": None, "platforms": {}} +def lookup_channel_type(platform_name: str, chat_id: str) -> Optional[str]: + """Return the channel ``type`` string (e.g. ``"channel"``, ``"forum"``) for *chat_id*, or *None* if unknown.""" + directory = load_directory() + for ch in directory.get("platforms", {}).get(platform_name, []): + if ch.get("id") == chat_id: + return ch.get("type") + return None + + def resolve_channel_name(platform_name: str, name: str) -> Optional[str]: """ Resolve a human-friendly channel name to a numeric ID. diff --git a/gateway/config.py b/gateway/config.py index 7d6165927968..67ebf7346189 100644 --- a/gateway/config.py +++ b/gateway/config.py @@ -66,6 +66,7 @@ class Platform(Enum): WECOM_CALLBACK = "wecom_callback" WEIXIN = "weixin" BLUEBUBBLES = "bluebubbles" + QQBOT = "qqbot" @dataclass @@ -257,6 +258,13 @@ class GatewayConfig: # Streaming configuration streaming: StreamingConfig = field(default_factory=StreamingConfig) + # Session store pruning: drop SessionEntry records older than this many + # days from the in-memory dict and sessions.json. Keeps the store from + # growing unbounded in gateways serving many chats/threads/users over + # months. Pruning is invisible to users — if they resume, they get a + # fresh session exactly as if the reset policy had fired. 0 = disabled. + session_store_max_age_days: int = 90 + def get_connected_platforms(self) -> List[Platform]: """Return list of platforms that are enabled and configured.""" connected = [] @@ -303,6 +311,17 @@ def get_connected_platforms(self) -> List[Platform]: # BlueBubbles uses extra dict for local server config elif platform == Platform.BLUEBUBBLES and config.extra.get("server_url") and config.extra.get("password"): connected.append(platform) + # QQBot uses extra dict for app credentials + elif platform == Platform.QQBOT and config.extra.get("app_id") and config.extra.get("client_secret"): + connected.append(platform) + # DingTalk uses client_id/client_secret from config.extra or env vars + elif platform == Platform.DINGTALK and ( + config.extra.get("client_id") or os.getenv("DINGTALK_CLIENT_ID") + ) and ( + config.extra.get("client_secret") or os.getenv("DINGTALK_CLIENT_SECRET") + ): + connected.append(platform) + return connected def get_home_channel(self, platform: Platform) -> Optional[HomeChannel]: @@ -353,6 +372,7 @@ def to_dict(self) -> Dict[str, Any]: "thread_sessions_per_user": self.thread_sessions_per_user, "unauthorized_dm_behavior": self.unauthorized_dm_behavior, "streaming": self.streaming.to_dict(), + "session_store_max_age_days": self.session_store_max_age_days, } @classmethod @@ -400,6 +420,13 @@ def from_dict(cls, data: Dict[str, Any]) -> "GatewayConfig": "pair", ) + try: + session_store_max_age_days = int(data.get("session_store_max_age_days", 90)) + if session_store_max_age_days < 0: + session_store_max_age_days = 0 + except (TypeError, ValueError): + session_store_max_age_days = 90 + return cls( platforms=platforms, default_reset_policy=default_policy, @@ -414,6 +441,7 @@ def from_dict(cls, data: Dict[str, Any]) -> "GatewayConfig": thread_sessions_per_user=_coerce_bool(thread_sessions_per_user, False), unauthorized_dm_behavior=unauthorized_dm_behavior, streaming=StreamingConfig.from_dict(data.get("streaming", {})), + session_store_max_age_days=session_store_max_age_days, ) def get_unauthorized_dm_behavior(self, platform: Optional[Platform] = None) -> str: @@ -548,8 +576,22 @@ def load_gateway_config() -> GatewayConfig: bridged["free_response_channels"] = platform_cfg["free_response_channels"] if "mention_patterns" in platform_cfg: bridged["mention_patterns"] = platform_cfg["mention_patterns"] + if "dm_policy" in platform_cfg: + bridged["dm_policy"] = platform_cfg["dm_policy"] + if "allow_from" in platform_cfg: + bridged["allow_from"] = platform_cfg["allow_from"] + if "group_policy" in platform_cfg: + bridged["group_policy"] = platform_cfg["group_policy"] + if "group_allow_from" in platform_cfg: + bridged["group_allow_from"] = platform_cfg["group_allow_from"] if plat == Platform.DISCORD and "channel_skill_bindings" in platform_cfg: bridged["channel_skill_bindings"] = platform_cfg["channel_skill_bindings"] + if "channel_prompts" in platform_cfg: + channel_prompts = platform_cfg["channel_prompts"] + if isinstance(channel_prompts, dict): + bridged["channel_prompts"] = {str(k): v for k, v in channel_prompts.items()} + else: + bridged["channel_prompts"] = channel_prompts if not bridged: continue plat_data = platforms_data.setdefault(plat.value, {}) @@ -574,6 +616,8 @@ def load_gateway_config() -> GatewayConfig: if isinstance(frc, list): frc = ",".join(str(v) for v in frc) os.environ["SLACK_FREE_RESPONSE_CHANNELS"] = str(frc) + if "reactions" in slack_cfg and not os.getenv("SLACK_REACTIONS"): + os.environ["SLACK_REACTIONS"] = str(slack_cfg["reactions"]).lower() # Discord settings → env vars (env vars take precedence) discord_cfg = yaml_cfg.get("discord", {}) @@ -607,6 +651,20 @@ def load_gateway_config() -> GatewayConfig: if isinstance(ntc, list): ntc = ",".join(str(v) for v in ntc) os.environ["DISCORD_NO_THREAD_CHANNELS"] = str(ntc) + # allow_mentions: granular control over what the bot can ping. + # Safe defaults (no @everyone/roles) are applied in the adapter; + # these YAML keys only override when set and let users opt back + # into unsafe modes (e.g. roles=true) if they actually want it. + allow_mentions_cfg = discord_cfg.get("allow_mentions") + if isinstance(allow_mentions_cfg, dict): + for yaml_key, env_key in ( + ("everyone", "DISCORD_ALLOW_MENTION_EVERYONE"), + ("roles", "DISCORD_ALLOW_MENTION_ROLES"), + ("users", "DISCORD_ALLOW_MENTION_USERS"), + ("replied_user", "DISCORD_ALLOW_MENTION_REPLIED_USER"), + ): + if yaml_key in allow_mentions_cfg and not os.getenv(env_key): + os.environ[env_key] = str(allow_mentions_cfg[yaml_key]).lower() # Telegram settings → env vars (env vars take precedence) telegram_cfg = yaml_cfg.get("telegram", {}) @@ -614,15 +672,31 @@ def load_gateway_config() -> GatewayConfig: if "require_mention" in telegram_cfg and not os.getenv("TELEGRAM_REQUIRE_MENTION"): os.environ["TELEGRAM_REQUIRE_MENTION"] = str(telegram_cfg["require_mention"]).lower() if "mention_patterns" in telegram_cfg and not os.getenv("TELEGRAM_MENTION_PATTERNS"): - import json as _json - os.environ["TELEGRAM_MENTION_PATTERNS"] = _json.dumps(telegram_cfg["mention_patterns"]) + os.environ["TELEGRAM_MENTION_PATTERNS"] = json.dumps(telegram_cfg["mention_patterns"]) frc = telegram_cfg.get("free_response_chats") if frc is not None and not os.getenv("TELEGRAM_FREE_RESPONSE_CHATS"): if isinstance(frc, list): frc = ",".join(str(v) for v in frc) os.environ["TELEGRAM_FREE_RESPONSE_CHATS"] = str(frc) + ignored_threads = telegram_cfg.get("ignored_threads") + if ignored_threads is not None and not os.getenv("TELEGRAM_IGNORED_THREADS"): + if isinstance(ignored_threads, list): + ignored_threads = ",".join(str(v) for v in ignored_threads) + os.environ["TELEGRAM_IGNORED_THREADS"] = str(ignored_threads) if "reactions" in telegram_cfg and not os.getenv("TELEGRAM_REACTIONS"): os.environ["TELEGRAM_REACTIONS"] = str(telegram_cfg["reactions"]).lower() + if "proxy_url" in telegram_cfg and not os.getenv("TELEGRAM_PROXY"): + os.environ["TELEGRAM_PROXY"] = str(telegram_cfg["proxy_url"]).strip() + if "disable_link_previews" in telegram_cfg: + plat_data = platforms_data.setdefault(Platform.TELEGRAM.value, {}) + if not isinstance(plat_data, dict): + plat_data = {} + platforms_data[Platform.TELEGRAM.value] = plat_data + extra = plat_data.setdefault("extra", {}) + if not isinstance(extra, dict): + extra = {} + plat_data["extra"] = extra + extra["disable_link_previews"] = telegram_cfg["disable_link_previews"] whatsapp_cfg = yaml_cfg.get("whatsapp", {}) if isinstance(whatsapp_cfg, dict): @@ -635,6 +709,38 @@ def load_gateway_config() -> GatewayConfig: if isinstance(frc, list): frc = ",".join(str(v) for v in frc) os.environ["WHATSAPP_FREE_RESPONSE_CHATS"] = str(frc) + if "dm_policy" in whatsapp_cfg and not os.getenv("WHATSAPP_DM_POLICY"): + os.environ["WHATSAPP_DM_POLICY"] = str(whatsapp_cfg["dm_policy"]).lower() + af = whatsapp_cfg.get("allow_from") + if af is not None and not os.getenv("WHATSAPP_ALLOWED_USERS"): + if isinstance(af, list): + af = ",".join(str(v) for v in af) + os.environ["WHATSAPP_ALLOWED_USERS"] = str(af) + if "group_policy" in whatsapp_cfg and not os.getenv("WHATSAPP_GROUP_POLICY"): + os.environ["WHATSAPP_GROUP_POLICY"] = str(whatsapp_cfg["group_policy"]).lower() + gaf = whatsapp_cfg.get("group_allow_from") + if gaf is not None and not os.getenv("WHATSAPP_GROUP_ALLOWED_USERS"): + if isinstance(gaf, list): + gaf = ",".join(str(v) for v in gaf) + os.environ["WHATSAPP_GROUP_ALLOWED_USERS"] = str(gaf) + + # DingTalk settings → env vars (env vars take precedence) + dingtalk_cfg = yaml_cfg.get("dingtalk", {}) + if isinstance(dingtalk_cfg, dict): + if "require_mention" in dingtalk_cfg and not os.getenv("DINGTALK_REQUIRE_MENTION"): + os.environ["DINGTALK_REQUIRE_MENTION"] = str(dingtalk_cfg["require_mention"]).lower() + if "mention_patterns" in dingtalk_cfg and not os.getenv("DINGTALK_MENTION_PATTERNS"): + os.environ["DINGTALK_MENTION_PATTERNS"] = json.dumps(dingtalk_cfg["mention_patterns"]) + frc = dingtalk_cfg.get("free_response_chats") + if frc is not None and not os.getenv("DINGTALK_FREE_RESPONSE_CHATS"): + if isinstance(frc, list): + frc = ",".join(str(v) for v in frc) + os.environ["DINGTALK_FREE_RESPONSE_CHATS"] = str(frc) + allowed = dingtalk_cfg.get("allowed_users") + if allowed is not None and not os.getenv("DINGTALK_ALLOWED_USERS"): + if isinstance(allowed, list): + allowed = ",".join(str(v) for v in allowed) + os.environ["DINGTALK_ALLOWED_USERS"] = str(allowed) # Matrix settings → env vars (env vars take precedence) matrix_cfg = yaml_cfg.get("matrix", {}) @@ -979,6 +1085,25 @@ def _apply_env_overrides(config: GatewayConfig) -> None: if webhook_secret: config.platforms[Platform.WEBHOOK].extra["secret"] = webhook_secret + # DingTalk + dingtalk_client_id = os.getenv("DINGTALK_CLIENT_ID") + dingtalk_client_secret = os.getenv("DINGTALK_CLIENT_SECRET") + if dingtalk_client_id and dingtalk_client_secret: + if Platform.DINGTALK not in config.platforms: + config.platforms[Platform.DINGTALK] = PlatformConfig() + config.platforms[Platform.DINGTALK].enabled = True + config.platforms[Platform.DINGTALK].extra.update({ + "client_id": dingtalk_client_id, + "client_secret": dingtalk_client_secret, + }) + dingtalk_home = os.getenv("DINGTALK_HOME_CHANNEL") + if dingtalk_home: + config.platforms[Platform.DINGTALK].home_channel = HomeChannel( + platform=Platform.DINGTALK, + chat_id=dingtalk_home, + name=os.getenv("DINGTALK_HOME_CHANNEL_NAME", "Home"), + ) + # Feishu / Lark feishu_app_id = os.getenv("FEISHU_APP_ID") feishu_app_secret = os.getenv("FEISHU_APP_SECRET") @@ -1109,6 +1234,43 @@ def _apply_env_overrides(config: GatewayConfig) -> None: name=os.getenv("BLUEBUBBLES_HOME_CHANNEL_NAME", "Home"), ) + # QQ (Official Bot API v2) + qq_app_id = os.getenv("QQ_APP_ID") + qq_client_secret = os.getenv("QQ_CLIENT_SECRET") + if qq_app_id or qq_client_secret: + if Platform.QQBOT not in config.platforms: + config.platforms[Platform.QQBOT] = PlatformConfig() + config.platforms[Platform.QQBOT].enabled = True + extra = config.platforms[Platform.QQBOT].extra + if qq_app_id: + extra["app_id"] = qq_app_id + if qq_client_secret: + extra["client_secret"] = qq_client_secret + qq_allowed_users = os.getenv("QQ_ALLOWED_USERS", "").strip() + if qq_allowed_users: + extra["allow_from"] = qq_allowed_users + qq_group_allowed = os.getenv("QQ_GROUP_ALLOWED_USERS", "").strip() + if qq_group_allowed: + extra["group_allow_from"] = qq_group_allowed + qq_home = os.getenv("QQBOT_HOME_CHANNEL", "").strip() + qq_home_name_env = "QQBOT_HOME_CHANNEL_NAME" + if not qq_home: + # Back-compat: accept the pre-rename name and log a one-time warning. + legacy_home = os.getenv("QQ_HOME_CHANNEL", "").strip() + if legacy_home: + qq_home = legacy_home + qq_home_name_env = "QQ_HOME_CHANNEL_NAME" + logging.getLogger(__name__).warning( + "QQ_HOME_CHANNEL is deprecated; rename to QQBOT_HOME_CHANNEL " + "in your .env for consistency with the platform key." + ) + if qq_home: + config.platforms[Platform.QQBOT].home_channel = HomeChannel( + platform=Platform.QQBOT, + chat_id=qq_home, + name=os.getenv("QQBOT_HOME_CHANNEL_NAME") or os.getenv(qq_home_name_env, "Home"), + ) + # Session settings idle_minutes = os.getenv("SESSION_IDLE_MINUTES") if idle_minutes: diff --git a/gateway/delivery.py b/gateway/delivery.py index d7fa6afdbf03..bc901c2adb37 100644 --- a/gateway/delivery.py +++ b/gateway/delivery.py @@ -12,7 +12,7 @@ from pathlib import Path from datetime import datetime from dataclasses import dataclass -from typing import Dict, List, Optional, Any, Union +from typing import Dict, List, Optional, Any from hermes_cli.config import get_hermes_home diff --git a/gateway/display_config.py b/gateway/display_config.py index 9375266ca681..78e8bc9afac0 100644 --- a/gateway/display_config.py +++ b/gateway/display_config.py @@ -9,6 +9,10 @@ 3. ``_PLATFORM_DEFAULTS[][]`` — built-in sensible default 4. ``_GLOBAL_DEFAULTS[]`` — built-in global default +Exception: ``display.streaming`` is CLI-only. Gateway streaming follows the +top-level ``streaming`` config unless ``display.platforms..streaming`` +sets an explicit per-platform override. + Backward compatibility: ``display.tool_progress_overrides`` is still read as a fallback for ``tool_progress`` when no ``display.platforms`` entry exists. A config migration (version bump) automatically moves the old format into the new @@ -143,10 +147,13 @@ def resolve_display_setting( if val is not None: return _normalise(setting, val) - # 2. Global user setting (display.) - val = display_cfg.get(setting) - if val is not None: - return _normalise(setting, val) + # 2. Global user setting (display.). Skip display.streaming because + # that key controls only CLI terminal streaming; gateway token streaming is + # governed by the top-level streaming config plus per-platform overrides. + if setting != "streaming": + val = display_cfg.get(setting) + if val is not None: + return _normalise(setting, val) # 3. Built-in platform default plat_defaults = _PLATFORM_DEFAULTS.get(platform_key) @@ -163,25 +170,6 @@ def resolve_display_setting( return fallback -def get_platform_defaults(platform_key: str) -> dict[str, Any]: - """Return the built-in default display settings for a platform. - - Falls back to ``_GLOBAL_DEFAULTS`` for unknown platforms. - """ - return dict(_PLATFORM_DEFAULTS.get(platform_key, _GLOBAL_DEFAULTS)) - - -def get_effective_display(user_config: dict, platform_key: str) -> dict[str, Any]: - """Return the fully-resolved display settings for a platform. - - Useful for status commands that want to show all effective settings. - """ - return { - key: resolve_display_setting(user_config, platform_key, key) - for key in OVERRIDEABLE_KEYS - } - - # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- diff --git a/gateway/hooks.py b/gateway/hooks.py index c50394b201f6..374e5b25fc8e 100644 --- a/gateway/hooks.py +++ b/gateway/hooks.py @@ -135,9 +135,22 @@ def discover_and_load(self) -> None: except Exception as e: print(f"[hooks] Error loading hook {hook_dir.name}: {e}", flush=True) + def _resolve_handlers(self, event_type: str) -> List[Callable]: + """Return all handlers that should fire for ``event_type``. + + Exact matches fire first, followed by wildcard matches (e.g. + ``command:*`` matches ``command:reset``). + """ + handlers = list(self._handlers.get(event_type, [])) + if ":" in event_type: + base = event_type.split(":")[0] + wildcard_key = f"{base}:*" + handlers.extend(self._handlers.get(wildcard_key, [])) + return handlers + async def emit(self, event_type: str, context: Optional[Dict[str, Any]] = None) -> None: """ - Fire all handlers registered for an event. + Fire all handlers registered for an event, discarding return values. Supports wildcard matching: handlers registered for "command:*" will fire for any "command:..." event. Handlers registered for a base type @@ -151,16 +164,7 @@ async def emit(self, event_type: str, context: Optional[Dict[str, Any]] = None) if context is None: context = {} - # Collect handlers: exact match + wildcard match - handlers = list(self._handlers.get(event_type, [])) - - # Check for wildcard patterns (e.g., "command:*" matches "command:reset") - if ":" in event_type: - base = event_type.split(":")[0] - wildcard_key = f"{base}:*" - handlers.extend(self._handlers.get(wildcard_key, [])) - - for fn in handlers: + for fn in self._resolve_handlers(event_type): try: result = fn(event_type, context) # Support both sync and async handlers @@ -168,3 +172,32 @@ async def emit(self, event_type: str, context: Optional[Dict[str, Any]] = None) await result except Exception as e: print(f"[hooks] Error in handler for '{event_type}': {e}", flush=True) + + async def emit_collect( + self, + event_type: str, + context: Optional[Dict[str, Any]] = None, + ) -> List[Any]: + """Fire handlers and return their non-None return values in order. + + Like :meth:`emit` but captures each handler's return value. Used for + decision-style hooks (e.g. ``command:`` policies that want to + allow/deny/rewrite the command before normal dispatch). + + Exceptions from individual handlers are logged but do not abort the + remaining handlers. + """ + if context is None: + context = {} + + results: List[Any] = [] + for fn in self._resolve_handlers(event_type): + try: + result = fn(event_type, context) + if asyncio.iscoroutine(result): + result = await result + if result is not None: + results.append(result) + except Exception as e: + print(f"[hooks] Error in handler for '{event_type}': {e}", flush=True) + return results diff --git a/gateway/platforms/__init__.py b/gateway/platforms/__init__.py index dae74568d024..4eb26edf0612 100644 --- a/gateway/platforms/__init__.py +++ b/gateway/platforms/__init__.py @@ -9,9 +9,11 @@ """ from .base import BasePlatformAdapter, MessageEvent, SendResult +from .qqbot import QQAdapter __all__ = [ "BasePlatformAdapter", "MessageEvent", "SendResult", + "QQAdapter", ] diff --git a/gateway/platforms/api_server.py b/gateway/platforms/api_server.py index 94d5f42a2d15..76c2eddfa5d5 100644 --- a/gateway/platforms/api_server.py +++ b/gateway/platforms/api_server.py @@ -10,6 +10,7 @@ - POST /v1/runs — start a run, returns run_id immediately (202) - GET /v1/runs/{run_id}/events — SSE stream of structured lifecycle events - GET /health — health check +- GET /health/detailed — rich status for cross-container dashboard probing - POST /warmup, POST /api/warmup — desktop local model warmup (Electron only) Any OpenAI-compatible frontend (Open WebUI, LobeChat, LibreChat, @@ -519,7 +520,7 @@ def _advertised_model_name(self, profile_id: Optional[str] = None) -> str: effective_profile = profile_id or self._host_profile_id if effective_profile and effective_profile not in ("default", "custom"): return effective_profile - return "hermes-agent" + return self._model_name or "hermes-agent" def _cors_headers_for_origin(self, origin: str) -> Optional[Dict[str, str]]: """Return CORS headers for an allowed browser origin.""" @@ -658,6 +659,27 @@ async def _handle_health(self, request: "web.Request") -> "web.Response": """GET /health — simple health check.""" return web.json_response({"status": "ok", "platform": "hermes-agent"}) + async def _handle_health_detailed(self, request: "web.Request") -> "web.Response": + """GET /health/detailed — rich status for cross-container dashboard probing. + + Returns gateway state, connected platforms, PID, and uptime so the + dashboard can display full status without needing a shared PID file or + /proc access. No authentication required. + """ + from gateway.status import read_runtime_status + + runtime = read_runtime_status() or {} + return web.json_response({ + "status": "ok", + "platform": "hermes-agent", + "gateway_state": runtime.get("gateway_state"), + "platforms": runtime.get("platforms", {}), + "active_agents": runtime.get("active_agents", 0), + "exit_reason": runtime.get("exit_reason"), + "updated_at": runtime.get("updated_at"), + "pid": os.getpid(), + }) + async def _handle_models(self, request: "web.Request") -> "web.Response": """GET /v1/models — return hermes-agent as an available model.""" auth_err = self._check_auth(request) @@ -2080,6 +2102,7 @@ async def connect(self) -> bool: self._app = web.Application(middlewares=mws, client_max_size=500 * 1024 * 1024) self._app["api_server_adapter"] = self self._app.router.add_get("/health", self._handle_health) + self._app.router.add_get("/health/detailed", self._handle_health_detailed) self._app.router.add_get("/v1/health", self._handle_health) self._app.router.add_get("/v1/models", self._handle_models) self._app.router.add_post("/v1/chat/completions", self._handle_chat_completions) diff --git a/gateway/platforms/base.py b/gateway/platforms/base.py index f7943da4739d..db76034989c5 100644 --- a/gateway/platforms/base.py +++ b/gateway/platforms/base.py @@ -6,6 +6,7 @@ """ import asyncio +import inspect import ipaddress import logging import os @@ -18,6 +19,8 @@ from abc import ABC, abstractmethod from urllib.parse import urlsplit +from utils import normalize_proxy_url + logger = logging.getLogger(__name__) @@ -158,13 +161,13 @@ def resolve_proxy_url(platform_env_var: str | None = None) -> str | None: if platform_env_var: value = (os.environ.get(platform_env_var) or "").strip() if value: - return value + return normalize_proxy_url(value) for key in ("HTTPS_PROXY", "HTTP_PROXY", "ALL_PROXY", "https_proxy", "http_proxy", "all_proxy"): value = (os.environ.get(key) or "").strip() if value: - return value - return _detect_macos_system_proxy() + return normalize_proxy_url(value) + return normalize_proxy_url(_detect_macos_system_proxy()) def proxy_kwargs_for_bot(proxy_url: str | None) -> dict: @@ -390,12 +393,9 @@ async def cache_image_from_url(url: str, ext: str = ".jpg", retries: int = 2) -> if not is_safe_url(url): raise ValueError(f"Blocked unsafe URL (SSRF protection): {safe_url_for_log(url)}") - import asyncio import httpx - import logging as _logging - _log = _logging.getLogger(__name__) + _log = logging.getLogger(__name__) - last_exc = None async with httpx.AsyncClient( timeout=30.0, follow_redirects=True, @@ -413,7 +413,6 @@ async def cache_image_from_url(url: str, ext: str = ".jpg", retries: int = 2) -> response.raise_for_status() return cache_image_from_bytes(response.content, ext) except (httpx.TimeoutException, httpx.HTTPStatusError) as exc: - last_exc = exc if isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code < 429: raise if attempt < retries: @@ -429,7 +428,6 @@ async def cache_image_from_url(url: str, ext: str = ".jpg", retries: int = 2) -> await asyncio.sleep(wait) continue raise - raise last_exc def cleanup_image_cache(max_age_hours: int = 24) -> int: @@ -509,12 +507,9 @@ async def cache_audio_from_url(url: str, ext: str = ".ogg", retries: int = 2) -> if not is_safe_url(url): raise ValueError(f"Blocked unsafe URL (SSRF protection): {safe_url_for_log(url)}") - import asyncio import httpx - import logging as _logging - _log = _logging.getLogger(__name__) + _log = logging.getLogger(__name__) - last_exc = None async with httpx.AsyncClient( timeout=30.0, follow_redirects=True, @@ -532,7 +527,6 @@ async def cache_audio_from_url(url: str, ext: str = ".ogg", retries: int = 2) -> response.raise_for_status() return cache_audio_from_bytes(response.content, ext) except (httpx.TimeoutException, httpx.HTTPStatusError) as exc: - last_exc = exc if isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code < 429: raise if attempt < retries: @@ -548,7 +542,39 @@ async def cache_audio_from_url(url: str, ext: str = ".ogg", retries: int = 2) -> await asyncio.sleep(wait) continue raise - raise last_exc + + +# --------------------------------------------------------------------------- +# Video cache utilities +# +# Same pattern as image/audio cache -- videos from platforms are downloaded +# here so the agent can reference them by local file path. +# --------------------------------------------------------------------------- + +VIDEO_CACHE_DIR = get_hermes_dir("cache/videos", "video_cache") + +SUPPORTED_VIDEO_TYPES = { + ".mp4": "video/mp4", + ".mov": "video/quicktime", + ".webm": "video/webm", + ".mkv": "video/x-matroska", + ".avi": "video/x-msvideo", +} + + +def get_video_cache_dir() -> Path: + """Return the video cache directory, creating it if it doesn't exist.""" + VIDEO_CACHE_DIR.mkdir(parents=True, exist_ok=True) + return VIDEO_CACHE_DIR + + +def cache_video_from_bytes(data: bytes, ext: str = ".mp4") -> str: + """Save raw video bytes to the cache and return the absolute file path.""" + cache_dir = get_video_cache_dir() + filename = f"video_{uuid.uuid4().hex[:12]}{ext}" + filepath = cache_dir / filename + filepath.write_bytes(data) + return str(filepath) # --------------------------------------------------------------------------- @@ -669,6 +695,15 @@ class MessageEvent: # Original platform data raw_message: Any = None message_id: Optional[str] = None + + # Platform-specific update identifier. For Telegram this is the + # ``update_id`` from the PTB Update wrapper; other platforms currently + # ignore it. Used by ``/restart`` to record the triggering update so the + # new gateway can advance the Telegram offset past it and avoid processing + # the same ``/restart`` twice if PTB's graceful-shutdown ACK times out + # ("Error while calling `get_updates` one more time to mark all fetched + # updates" in gateway.log). + platform_update_id: Optional[int] = None # Media attachments # media_urls: local file paths (for vision tool access) @@ -682,6 +717,10 @@ class MessageEvent: # Auto-loaded skill(s) for topic/channel bindings (e.g., Telegram DM Topics, # Discord channel_skill_bindings). A single name or ordered list. auto_skill: Optional[str | list[str]] = None + + # Per-channel ephemeral system prompt (e.g. Discord channel_prompts). + # Applied at API call time and never persisted to transcript history. + channel_prompt: Optional[str] = None # Internal flag — set for synthetic events (e.g. background process # completion notifications) that must bypass user authorization checks. @@ -713,7 +752,10 @@ def get_command_args(self) -> str: if not self.is_command(): return self.text parts = self.text.split(maxsplit=1) - return parts[1] if len(parts) > 1 else "" + args = parts[1] if len(parts) > 1 else "" + # iOS auto-corrects -- to — (em dash) and - to – (en dash) + args = args.replace("\u2014\u2014", "--").replace("\u2014", "--").replace("\u2013", "-") + return args @dataclass @@ -730,25 +772,56 @@ def merge_pending_message_event( pending_messages: Dict[str, MessageEvent], session_key: str, event: MessageEvent, + *, + merge_text: bool = False, ) -> None: """Store or merge a pending event for a session. Photo bursts/albums often arrive as multiple near-simultaneous PHOTO events. Merge those into the existing queued event so the next turn sees - the whole burst, while non-photo follow-ups still replace the pending - event normally. + the whole burst. + + When ``merge_text`` is enabled, rapid follow-up TEXT events are appended + instead of replacing the pending turn. This is used for Telegram bursty + follow-ups so a multi-part user thought is not silently truncated to only + the last queued fragment. """ existing = pending_messages.get(session_key) - if ( - existing - and getattr(existing, "message_type", None) == MessageType.PHOTO - and event.message_type == MessageType.PHOTO - ): - existing.media_urls.extend(event.media_urls) - existing.media_types.extend(event.media_types) - if event.text: - existing.text = BasePlatformAdapter._merge_caption(existing.text, event.text) - return + if existing: + existing_is_photo = getattr(existing, "message_type", None) == MessageType.PHOTO + incoming_is_photo = event.message_type == MessageType.PHOTO + existing_has_media = bool(existing.media_urls) + incoming_has_media = bool(event.media_urls) + + if existing_is_photo and incoming_is_photo: + existing.media_urls.extend(event.media_urls) + existing.media_types.extend(event.media_types) + if event.text: + existing.text = BasePlatformAdapter._merge_caption(existing.text, event.text) + return + + if existing_has_media or incoming_has_media: + if incoming_has_media: + existing.media_urls.extend(event.media_urls) + existing.media_types.extend(event.media_types) + if event.text: + if existing.text: + existing.text = BasePlatformAdapter._merge_caption(existing.text, event.text) + else: + existing.text = event.text + if existing_is_photo or incoming_is_photo: + existing.message_type = MessageType.PHOTO + return + + if ( + merge_text + and getattr(existing, "message_type", None) == MessageType.TEXT + and event.message_type == MessageType.TEXT + ): + if event.text: + existing.text = f"{existing.text}\n{event.text}" if existing.text else event.text + return + pending_messages[session_key] = event @@ -776,6 +849,36 @@ def merge_pending_message_event( MessageHandler = Callable[[MessageEvent], Awaitable[Optional[str]]] +def resolve_channel_prompt( + config_extra: dict, + channel_id: str, + parent_id: str | None = None, +) -> str | None: + """Resolve a per-channel ephemeral prompt from platform config. + + Looks up ``channel_prompts`` in the adapter's ``config.extra`` dict. + Prefers an exact match on *channel_id*; falls back to *parent_id* + (useful for forum threads / child channels inheriting a parent prompt). + + Returns the prompt string, or None if no match is found. Blank/whitespace- + only prompts are treated as absent. + """ + prompts = config_extra.get("channel_prompts") or {} + if not isinstance(prompts, dict): + return None + + for key in (channel_id, parent_id): + if not key: + continue + prompt = prompts.get(key) + if prompt is None: + continue + prompt = str(prompt).strip() + if prompt: + return prompt + return None + + class BasePlatformAdapter(ABC): """ Base class for platform adapters. @@ -797,14 +900,26 @@ def __init__(self, config: PlatformConfig, platform: Platform): self._fatal_error_retryable = True self._fatal_error_handler: Optional[Callable[["BasePlatformAdapter"], Awaitable[None] | None]] = None - # Track active message handlers per session for interrupt support - # Key: session_key (e.g., chat_id), Value: (event, asyncio.Event for interrupt) + # Track active message handlers per session for interrupt support. + # _active_sessions stores the per-session interrupt Event; _session_tasks + # maps session → the specific Task currently processing it so that + # session-terminating commands (/stop, /new, /reset) can cancel the + # right task and release the adapter-level guard deterministically. + # Without the owner-task map, an old task's finally block could delete + # a newer task's guard, leaving stale busy state. self._active_sessions: Dict[str, asyncio.Event] = {} self._pending_messages: Dict[str, MessageEvent] = {} + self._session_tasks: Dict[str, asyncio.Task] = {} # Background message-processing tasks spawned by handle_message(). # Gateway shutdown cancels these so an old gateway instance doesn't keep # working on a task after --replace or manual restarts. self._background_tasks: set[asyncio.Task] = set() + # One-shot callbacks to fire after the main response is delivered. + # Keyed by session_key. Values are either a bare callback (legacy) or + # a ``(generation, callback)`` tuple so GatewayRunner can make deferred + # deliveries generation-aware and avoid stale runs clearing callbacks + # registered by a fresher run for the same session. + self._post_delivery_callbacks: Dict[str, Any] = {} self._expected_cancelled_tasks: set[asyncio.Task] = set() self._busy_session_handler: Optional[Callable[[MessageEvent, str], Awaitable[bool]]] = None # Chats where auto-TTS on voice input is disabled (set by /voice off) @@ -975,16 +1090,40 @@ async def send( """ pass + # Default: the adapter treats ``finalize=True`` on edit_message as a + # no-op and is happy to have the stream consumer skip redundant final + # edits. Subclasses that *require* an explicit finalize call to close + # out the message lifecycle (e.g. rich card / AI assistant surfaces + # such as DingTalk AI Cards) override this to True (class attribute or + # property) so the stream consumer knows not to short-circuit. + REQUIRES_EDIT_FINALIZE: bool = False + async def edit_message( self, chat_id: str, message_id: str, content: str, + *, + finalize: bool = False, ) -> SendResult: """ Edit a previously sent message. Optional — platforms that don't support editing return success=False and callers fall back to sending a new message. + + ``finalize`` signals that this is the last edit in a streaming + sequence. Most platforms (Telegram, Slack, Discord, Matrix, + etc.) treat it as a no-op because their edit APIs have no notion + of message lifecycle state — an edit is an edit. Platforms that + render streaming updates with a distinct "in progress" state and + require explicit closure (e.g. rich card / AI assistant surfaces + such as DingTalk AI Cards) use it to finalize the message and + transition the UI out of the streaming indicator — those should + also set ``REQUIRES_EDIT_FINALIZE = True`` so callers route a + final edit through even when content is unchanged. Callers + should set ``finalize=True`` on the final edit of a streamed + response (typically when ``got_done`` fires in the stream + consumer) and leave it ``False`` on intermediate edits. """ return SendResult(success=False, error="Not supported") @@ -1213,7 +1352,7 @@ def extract_media(content: str) -> Tuple[List[Tuple[str, bool]], str]: # Extract MEDIA: tags, allowing optional whitespace after the colon # and quoted/backticked paths for LLM-formatted outputs. media_pattern = re.compile( - r'''[`"']?MEDIA:\s*(?P`[^`\n]+`|"[^"\n]+"|'[^'\n]+'|(?:~/|/)\S+(?:[^\S\n]+\S+)*?\.(?:png|jpe?g|gif|webp|mp4|mov|avi|mkv|webm|ogg|opus|mp3|wav|m4a)(?=[\s`"',;:)\]}]|$)|\S+)[`"']?''' + r'''[`"']?MEDIA:\s*(?P`[^`\n]+`|"[^"\n]+"|'[^'\n]+'|(?:~/|/)\S+(?:[^\S\n]+\S+)*?\.(?:png|jpe?g|gif|webp|mp4|mov|avi|mkv|webm|ogg|opus|mp3|wav|m4a|epub|pdf|zip|rar|7z|docx?|xlsx?|pptx?|txt|csv|apk|ipa)(?=[\s`"',;:)\]}]|$)|\S+)[`"']?''' ) for match in media_pattern.finditer(content): path = match.group("path").strip() @@ -1221,7 +1360,7 @@ def extract_media(content: str) -> Tuple[List[Tuple[str, bool]], str]: path = path[1:-1].strip() path = path.lstrip("`\"'").rstrip("`\"',.;:)}]") if path: - media.append((path, has_voice_tag)) + media.append((os.path.expanduser(path), has_voice_tag)) # Remove MEDIA tags from content (including surrounding quote/backtick wrappers) if media: @@ -1298,7 +1437,13 @@ def _in_code(pos: int) -> bool: return paths, cleaned - async def _keep_typing(self, chat_id: str, interval: float = 2.0, metadata=None) -> None: + async def _keep_typing( + self, + chat_id: str, + interval: float = 2.0, + metadata=None, + stop_event: asyncio.Event | None = None, + ) -> None: """ Continuously send typing indicator until cancelled. @@ -1312,9 +1457,18 @@ async def _keep_typing(self, chat_id: str, interval: float = 2.0, metadata=None) """ try: while True: + if stop_event is not None and stop_event.is_set(): + return if chat_id not in self._typing_paused: await self.send_typing(chat_id, metadata=metadata) - await asyncio.sleep(interval) + if stop_event is None: + await asyncio.sleep(interval) + continue + try: + await asyncio.wait_for(stop_event.wait(), timeout=interval) + except asyncio.TimeoutError: + continue + return except asyncio.CancelledError: pass # Normal cancellation when handler completes finally: @@ -1341,6 +1495,59 @@ def resume_typing_for_chat(self, chat_id: str) -> None: """Resume typing indicator for a chat after approval resolves.""" self._typing_paused.discard(chat_id) + async def interrupt_session_activity(self, session_key: str, chat_id: str) -> None: + """Signal the active session loop to stop and clear typing immediately.""" + if session_key: + interrupt_event = self._active_sessions.get(session_key) + if interrupt_event is not None: + interrupt_event.set() + try: + await self.stop_typing(chat_id) + except Exception: + pass + + def register_post_delivery_callback( + self, + session_key: str, + callback: Callable, + *, + generation: int | None = None, + ) -> None: + """Register a deferred callback to fire after the main response. + + ``generation`` lets callers tie the callback to a specific gateway run + generation so stale runs cannot clear callbacks owned by a fresher run. + """ + if not session_key or not callable(callback): + return + if generation is None: + self._post_delivery_callbacks[session_key] = callback + else: + self._post_delivery_callbacks[session_key] = (int(generation), callback) + + def pop_post_delivery_callback( + self, + session_key: str, + *, + generation: int | None = None, + ) -> Callable | None: + """Pop a deferred callback, optionally requiring generation ownership.""" + if not session_key: + return None + entry = self._post_delivery_callbacks.get(session_key) + if entry is None: + return None + if isinstance(entry, tuple) and len(entry) == 2: + entry_generation, callback = entry + if generation is not None and int(entry_generation) != int(generation): + return None + self._post_delivery_callbacks.pop(session_key, None) + return callback if callable(callback) else None + if generation is not None: + return None + self._post_delivery_callbacks.pop(session_key, None) + return entry if callable(entry) else None + # ── Processing lifecycle hooks ────────────────────────────────────────── # Subclasses override these to react to message processing events # (e.g. Discord adds 👀/✅/❌ reactions). @@ -1479,6 +1686,222 @@ def _merge_caption(existing_text: Optional[str], new_text: str) -> str: return f"{existing_text}\n\n{new_text}".strip() return existing_text + # ------------------------------------------------------------------ + # Session task + guard ownership helpers + # ------------------------------------------------------------------ + # These were introduced together with the _session_tasks owner map to + # make session lifecycle reconciliation deterministic across (a) the + # normal completion path, (b) /stop/ /new/ /reset bypass commands, + # and (c) stale-lock self-heal on the next inbound message. + + def _release_session_guard( + self, + session_key: str, + *, + guard: Optional[asyncio.Event] = None, + ) -> None: + """Release the adapter-level guard for a session. + + When ``guard`` is provided, only release the entry if it still points + at that exact Event. This lets reset-like commands swap in a temporary + guard while the old processing task unwinds, without having the old + task's cleanup accidentally clear the replacement guard. + """ + current_guard = self._active_sessions.get(session_key) + if current_guard is None: + return + if guard is not None and current_guard is not guard: + return + del self._active_sessions[session_key] + + def _session_task_is_stale(self, session_key: str) -> bool: + """Return True if the owner task for ``session_key`` is done/cancelled. + + A lock is "stale" when the adapter still has ``_active_sessions[key]`` + AND a known owner task in ``_session_tasks`` that has already exited. + When there is no owner task at all, that usually means the guard was + installed by some path other than handle_message() (tests sometimes + install guards directly) — don't treat that as stale. The on-entry + self-heal only needs to handle the production split-brain case where + an owner task was recorded, then exited without clearing its guard. + """ + task = self._session_tasks.get(session_key) + if task is None: + return False + done = getattr(task, "done", None) + return bool(done and done()) + + def _heal_stale_session_lock(self, session_key: str) -> bool: + """Clear a stale session lock if the owner task is already gone. + + Returns True if a stale lock was healed. Returns False if there is + no lock, or the owner task is still alive (the normal busy case). + + This is the on-entry safety net sidbin's issue #11016 analysis calls + for: without it, a split-brain — adapter still thinks the session is + active, but nothing is actually processing — traps the chat in + infinite "Interrupting current task..." until the gateway is + restarted. + """ + if session_key not in self._active_sessions: + return False + if not self._session_task_is_stale(session_key): + return False + logger.warning( + "[%s] Healing stale session lock for %s (owner task is done/absent)", + self.name, + session_key, + ) + self._active_sessions.pop(session_key, None) + self._pending_messages.pop(session_key, None) + self._session_tasks.pop(session_key, None) + return True + + def _start_session_processing( + self, + event: MessageEvent, + session_key: str, + *, + interrupt_event: Optional[asyncio.Event] = None, + ) -> bool: + """Spawn a background processing task under the given session guard. + + Returns True on success. If the runtime stubs ``create_task`` with a + non-Task sentinel (some tests do this), the guard is rolled back and + False is returned so the caller isn't left holding a half-installed + session lock. + """ + guard = interrupt_event or asyncio.Event() + self._active_sessions[session_key] = guard + + task = asyncio.create_task(self._process_message_background(event, session_key)) + self._session_tasks[session_key] = task + try: + self._background_tasks.add(task) + except TypeError: + # Tests stub create_task() with lightweight sentinels that are not + # hashable and do not support lifecycle callbacks. + self._session_tasks.pop(session_key, None) + self._release_session_guard(session_key, guard=guard) + return False + if hasattr(task, "add_done_callback"): + task.add_done_callback(self._background_tasks.discard) + task.add_done_callback(self._expected_cancelled_tasks.discard) + return True + + async def cancel_session_processing( + self, + session_key: str, + *, + release_guard: bool = True, + discard_pending: bool = True, + ) -> None: + """Cancel in-flight processing for a single session. + + ``release_guard=False`` keeps the adapter-level session guard in place + so reset-like commands can finish atomically before follow-up messages + are allowed to start a fresh background task. + """ + task = self._session_tasks.pop(session_key, None) + if task is not None and not task.done(): + logger.debug( + "[%s] Cancelling active processing for session %s", + self.name, + session_key, + ) + self._expected_cancelled_tasks.add(task) + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + except Exception: + logger.debug( + "[%s] Session cancellation raised while unwinding %s", + self.name, + session_key, + exc_info=True, + ) + if discard_pending: + self._pending_messages.pop(session_key, None) + if release_guard: + self._release_session_guard(session_key) + + async def _drain_pending_after_session_command( + self, + session_key: str, + command_guard: asyncio.Event, + ) -> None: + """Resume the latest queued follow-up once a session command completes. + + Called at the tail of /stop, /new, and /reset dispatch. Releases the + command-scoped guard, then — if a follow-up message landed while the + command was running — spawns a fresh processing task for it. + """ + pending_event = self._pending_messages.pop(session_key, None) + self._release_session_guard(session_key, guard=command_guard) + if pending_event is None: + return + self._start_session_processing(pending_event, session_key) + + async def _dispatch_active_session_command( + self, + event: MessageEvent, + session_key: str, + cmd: str, + ) -> None: + """Dispatch a reset-like bypass command while preserving guard ordering. + + /stop, /new, and /reset must: + 1. Keep the session guard installed while the runner processes the + command (so a racing follow-up message stays queued, not + dispatched as a second parallel run). + 2. Cancel the old in-flight adapter task only AFTER the runner has + finished handling the command (so the runner sees consistent + state and its response is sent in order). + 3. Release the command-scoped guard and drain the latest queued + follow-up exactly once, after 1 and 2 complete. + """ + logger.debug( + "[%s] Command '/%s' bypassing active-session guard for %s", + self.name, + cmd, + session_key, + ) + + current_guard = self._active_sessions.get(session_key) + command_guard = asyncio.Event() + self._active_sessions[session_key] = command_guard + thread_meta = {"thread_id": event.source.thread_id} if event.source.thread_id else None + + try: + response = await self._message_handler(event) + # Old adapter task (if any) is cancelled AFTER the runner has + # fully handled the command — keeps ordering deterministic. + await self.cancel_session_processing( + session_key, + release_guard=False, + discard_pending=False, + ) + if response: + await self._send_with_retry( + chat_id=event.source.chat_id, + content=response, + reply_to=event.message_id, + metadata=thread_meta, + ) + except Exception: + # On failure, restore the original guard if one still exists so + # we don't leave the session in a half-reset state. + if self._active_sessions.get(session_key) is command_guard: + if session_key in self._session_tasks and current_guard is not None: + self._active_sessions[session_key] = current_guard + else: + self._release_session_guard(session_key, guard=command_guard) + raise + + await self._drain_pending_after_session_command(session_key, command_guard) + async def handle_message(self, event: MessageEvent) -> None: """ Process an incoming message. @@ -1495,7 +1918,15 @@ async def handle_message(self, event: MessageEvent) -> None: group_sessions_per_user=self.config.extra.get("group_sessions_per_user", True), thread_sessions_per_user=self.config.extra.get("thread_sessions_per_user", False), ) - + + # On-entry self-heal: if the adapter still has an _active_sessions + # entry for this key but the owner task has already exited (done or + # cancelled), the lock is stale. Clear it and fall through to + # normal dispatch so the user isn't trapped behind a dead guard — + # this is the split-brain tail described in issue #11016. + if session_key in self._active_sessions: + self._heal_stale_session_lock(session_key) + # Check if there's already an active handler for this session if session_key in self._active_sessions: # Certain commands must bypass the active-session guard and be @@ -1509,7 +1940,26 @@ async def handle_message(self, event: MessageEvent) -> None: # session lifecycle and its cleanup races with the running task # (see PR #4926). cmd = event.get_command() - if cmd in ("approve", "deny", "status", "stop", "new", "reset", "background", "restart"): + from hermes_cli.commands import should_bypass_active_session + + if should_bypass_active_session(cmd): + # /stop, /new, /reset must cancel the in-flight adapter task + # and preserve ordering of queued follow-ups. Route those + # through the dedicated handoff path that serializes + # cancellation + runner response + pending drain. + if cmd in ("stop", "new", "reset"): + try: + await self._dispatch_active_session_command(event, session_key, cmd) + except Exception as e: + logger.error( + "[%s] Command '/%s' dispatch failed: %s", + self.name, cmd, e, exc_info=True, + ) + return + + # Other bypass commands (/approve, /deny, /status, + # /background, /restart) just need direct dispatch — they + # don't cancel the running task. logger.debug( "[%s] Command '/%s' bypassing active-session guard for %s", self.name, cmd, session_key, @@ -1555,19 +2005,9 @@ async def handle_message(self, event: MessageEvent) -> None: # starts would also pass the _active_sessions check and spawn a # duplicate task. (grammY sequentialize / aiogram EventIsolation # pattern — set the guard synchronously, not inside the task.) - self._active_sessions[session_key] = asyncio.Event() - - # Spawn background task to process this message - task = asyncio.create_task(self._process_message_background(event, session_key)) - try: - self._background_tasks.add(task) - except TypeError: - # Some tests stub create_task() with lightweight sentinels that are not - # hashable and do not support lifecycle callbacks. - return - if hasattr(task, "add_done_callback"): - task.add_done_callback(self._background_tasks.discard) - task.add_done_callback(self._expected_cancelled_tasks.discard) + # _start_session_processing installs the guard AND the owner-task + # mapping atomically so stale-lock detection works. + self._start_session_processing(event, session_key) @staticmethod def _get_human_delay() -> float: @@ -1579,8 +2019,6 @@ def _get_human_delay() -> float: HERMES_HUMAN_DELAY_MIN_MS: minimum delay in ms (default 800, custom mode) HERMES_HUMAN_DELAY_MAX_MS: maximum delay in ms (default 2500, custom mode) """ - import random - mode = os.getenv("HERMES_HUMAN_DELAY_MODE", "off").lower() if mode == "off": return 0.0 @@ -1609,10 +2047,23 @@ def _record_delivery(result): # Fall back to a new Event only if the entry was removed externally. interrupt_event = self._active_sessions.get(session_key) or asyncio.Event() self._active_sessions[session_key] = interrupt_event + callback_generation = getattr(interrupt_event, "_hermes_run_generation", None) # Start continuous typing indicator (refreshes every 2 seconds) _thread_metadata = {"thread_id": event.source.thread_id} if event.source.thread_id else None - typing_task = asyncio.create_task(self._keep_typing(event.source.chat_id, metadata=_thread_metadata)) + _keep_typing_kwargs = {"metadata": _thread_metadata} + try: + _keep_typing_sig = inspect.signature(self._keep_typing) + except (TypeError, ValueError): + _keep_typing_sig = None + if _keep_typing_sig is None or "stop_event" in _keep_typing_sig.parameters: + _keep_typing_kwargs["stop_event"] = interrupt_event + typing_task = asyncio.create_task( + self._keep_typing( + event.source.chat_id, + **_keep_typing_kwargs, + ) + ) try: await self._run_processing_hook("on_processing_start", event) @@ -1624,6 +2075,21 @@ def _record_delivery(result): # streaming already delivered the text (already_sent=True) or # when the message was queued behind an active agent. Log at # DEBUG to avoid noisy warnings for expected behavior. + # + # Suppress stale response when the session was interrupted by a + # new message that hasn't been consumed yet. The pending message + # is processed by the pending-message handler below (#8221/#2483). + if ( + response + and interrupt_event.is_set() + and session_key in self._pending_messages + ): + logger.info( + "[%s] Suppressing stale response for interrupted session %s", + self.name, + session_key, + ) + response = None if not response: logger.debug("[%s] Handler returned empty/None response for %s", self.name, event.source.chat_id) if response: @@ -1806,9 +2272,18 @@ def _record_delivery(result): if session_key in self._pending_messages: pending_event = self._pending_messages.pop(session_key) logger.debug("[%s] Processing queued message from interrupt", self.name) - # Clean up current session before processing pending - if session_key in self._active_sessions: - del self._active_sessions[session_key] + # Keep the _active_sessions entry live across the turn chain + # and only CLEAR the interrupt Event — do NOT delete the entry. + # If we deleted here, a concurrent inbound message arriving + # during the awaits below would pass the Level-1 guard, spawn + # its own _process_message_background, and run simultaneously + # with the recursive drain below. Two agents on one + # session_key = duplicate responses, duplicate tool calls. + # Clearing the Event keeps the guard live so follow-ups take + # the busy-handler path (queue + interrupt) as intended. + _active = self._active_sessions.get(session_key) + if _active is not None: + _active.clear() typing_task.cancel() try: await typing_task @@ -1845,6 +2320,21 @@ def _record_delivery(result): except Exception: pass # Last resort — don't let error reporting crash the handler finally: + # Fire any one-shot post-delivery callback registered for this + # session (e.g. deferred background-review notifications). + _callback_generation = callback_generation + if hasattr(self, "pop_post_delivery_callback"): + _post_cb = self.pop_post_delivery_callback( + session_key, + generation=_callback_generation, + ) + else: + _post_cb = getattr(self, "_post_delivery_callbacks", {}).pop(session_key, None) + if callable(_post_cb): + try: + _post_cb() + except Exception: + pass # Stop typing indicator typing_task.cancel() try: @@ -1858,9 +2348,45 @@ def _record_delivery(result): await self.stop_typing(event.source.chat_id) except Exception: pass - # Clean up session tracking - if session_key in self._active_sessions: - del self._active_sessions[session_key] + # Late-arrival drain: a message may have arrived during the + # cleanup awaits above (typing_task cancel, stop_typing). Such + # messages passed the Level-1 guard (entry still live, Event + # possibly set) and landed in _pending_messages via the + # busy-handler path. Without this block, we would delete the + # active-session entry and the queued message would be silently + # dropped (user never gets a reply). + late_pending = self._pending_messages.pop(session_key, None) + if late_pending is not None: + logger.debug( + "[%s] Late-arrival pending message during cleanup — spawning drain task", + self.name, + ) + _active = self._active_sessions.get(session_key) + if _active is not None: + _active.clear() + drain_task = asyncio.create_task( + self._process_message_background(late_pending, session_key) + ) + # Hand ownership of the session to the drain task so stale-lock + # detection keeps working while it runs. + self._session_tasks[session_key] = drain_task + try: + self._background_tasks.add(drain_task) + drain_task.add_done_callback(self._background_tasks.discard) + except TypeError: + # Tests stub create_task() with non-hashable sentinels; tolerate. + pass + # Leave _active_sessions[session_key] populated — the drain + # task's own lifecycle will clean it up. + else: + # Clean up session tracking. Guard-match both deletes so a + # reset-like command that already swapped in its own + # command_guard (and cancelled us) can't be accidentally + # cleared by our unwind. The command owns the session now. + current_task = asyncio.current_task() + if current_task is not None and self._session_tasks.get(session_key) is current_task: + del self._session_tasks[session_key] + self._release_session_guard(session_key, guard=interrupt_event) async def cancel_background_tasks(self) -> None: """Cancel any in-flight background message-processing tasks. @@ -1868,14 +2394,29 @@ async def cancel_background_tasks(self) -> None: Used during gateway shutdown/replacement so active sessions from the old process do not keep running after adapters are being torn down. """ - tasks = [task for task in self._background_tasks if not task.done()] - for task in tasks: - self._expected_cancelled_tasks.add(task) - task.cancel() - if tasks: + # Loop until no new tasks appear. Without this, a message + # arriving during the `await asyncio.gather` below would spawn + # a fresh _process_message_background task (added to + # self._background_tasks at line ~1668 via handle_message), + # and the _background_tasks.clear() at the end of this method + # would drop the reference — the task runs untracked against a + # disconnecting adapter, logs send-failures, and may linger + # until it completes on its own. Retrying the drain until the + # task set stabilizes closes the window. + MAX_DRAIN_ROUNDS = 5 + for _ in range(MAX_DRAIN_ROUNDS): + tasks = [task for task in self._background_tasks if not task.done()] + if not tasks: + break + for task in tasks: + self._expected_cancelled_tasks.add(task) + task.cancel() await asyncio.gather(*tasks, return_exceptions=True) + # Loop: late-arrival tasks spawned during the gather above + # will be in self._background_tasks now. Re-check. self._background_tasks.clear() self._expected_cancelled_tasks.clear() + self._session_tasks.clear() self._pending_messages.clear() self._active_sessions.clear() @@ -1898,6 +2439,7 @@ def build_source( chat_topic: Optional[str] = None, user_id_alt: Optional[str] = None, chat_id_alt: Optional[str] = None, + is_bot: bool = False, ) -> SessionSource: """Helper to build a SessionSource for this platform.""" # Normalize empty topic to None @@ -1914,6 +2456,7 @@ def build_source( chat_topic=chat_topic.strip() if chat_topic else None, user_id_alt=user_id_alt, chat_id_alt=chat_id_alt, + is_bot=is_bot, ) @abstractmethod diff --git a/gateway/platforms/bluebubbles.py b/gateway/platforms/bluebubbles.py index 1150009965f7..39d4e537ebd4 100644 --- a/gateway/platforms/bluebubbles.py +++ b/gateway/platforms/bluebubbles.py @@ -75,7 +75,7 @@ def _redact(text: str) -> str: def check_bluebubbles_requirements() -> bool: try: import aiohttp # noqa: F401 - import httpx as _httpx # noqa: F401 + import httpx # noqa: F401 except ImportError: return False return True @@ -224,6 +224,21 @@ def _webhook_url(self) -> str: host = "localhost" return f"http://{host}:{self.webhook_port}{self.webhook_path}" + @property + def _webhook_register_url(self) -> str: + """Webhook URL registered with BlueBubbles, including the password as + a query param so inbound webhook POSTs carry credentials. + + BlueBubbles posts events to the exact URL registered via + ``/api/v1/webhook``. Its webhook registration API does not support + custom headers, so embedding the password in the URL is the only + way to authenticate inbound webhooks without disabling auth. + """ + base = self._webhook_url + if self.password: + return f"{base}?password={quote(self.password, safe='')}" + return base + async def _find_registered_webhooks(self, url: str) -> list: """Return list of BB webhook entries matching *url*.""" try: @@ -245,7 +260,7 @@ async def _register_webhook(self) -> bool: if not self.client: return False - webhook_url = self._webhook_url + webhook_url = self._webhook_register_url # Crash resilience — reuse an existing registration if present existing = await self._find_registered_webhooks(webhook_url) @@ -257,7 +272,7 @@ async def _register_webhook(self) -> bool: payload = { "url": webhook_url, - "events": ["new-message", "updated-message", "message"], + "events": ["new-message", "updated-message"], } try: @@ -292,7 +307,7 @@ async def _unregister_webhook(self) -> bool: if not self.client: return False - webhook_url = self._webhook_url + webhook_url = self._webhook_register_url removed = False try: @@ -604,35 +619,6 @@ async def mark_read(self, chat_id: str) -> bool: # Tapback reactions # ------------------------------------------------------------------ - async def send_reaction( - self, - chat_id: str, - message_guid: str, - reaction: str, - part_index: int = 0, - ) -> SendResult: - """Send a tapback reaction (requires Private API helper).""" - if not self._private_api_enabled or not self._helper_connected: - return SendResult( - success=False, error="Private API helper not connected" - ) - guid = await self._resolve_chat_guid(chat_id) - if not guid: - return SendResult(success=False, error=f"Chat not found: {chat_id}") - try: - res = await self._api_post( - "/api/v1/message/react", - { - "chatGuid": guid, - "selectedMessageGuid": message_guid, - "reaction": reaction, - "partIndex": part_index, - }, - ) - return SendResult(success=True, raw_response=res) - except Exception as exc: - return SendResult(success=False, error=str(exc)) - # ------------------------------------------------------------------ # Chat info # ------------------------------------------------------------------ @@ -864,6 +850,12 @@ async def _handle_webhook(self, request): payload.get("chat_guid"), payload.get("guid"), ) + # Fallback: BlueBubbles v1.9+ webhook payloads omit top-level chatGuid; + # the chat GUID is nested under data.chats[0].guid instead. + if not chat_guid: + _chats = record.get("chats") or [] + if _chats and isinstance(_chats[0], dict): + chat_guid = _chats[0].get("guid") or _chats[0].get("chatGuid") chat_identifier = self._value( record.get("chatIdentifier"), record.get("identifier"), diff --git a/gateway/platforms/dingtalk.py b/gateway/platforms/dingtalk.py index 5d50deca58c5..3037e402b2cd 100644 --- a/gateway/platforms/dingtalk.py +++ b/gateway/platforms/dingtalk.py @@ -1,46 +1,92 @@ """ DingTalk platform adapter using Stream Mode. -Uses dingtalk-stream SDK for real-time message reception without webhooks. +Uses dingtalk-stream SDK (>=0.20) for real-time message reception without webhooks. Responses are sent via DingTalk's session webhook (markdown format). +Supports: text, images, audio, video, rich text, files, and group @mentions. Requires: - pip install dingtalk-stream httpx + pip install "dingtalk-stream>=0.20" httpx DINGTALK_CLIENT_ID and DINGTALK_CLIENT_SECRET env vars Configuration in config.yaml: platforms: dingtalk: enabled: true + # Optional group-chat gating (mirrors Slack/Telegram/Discord): + require_mention: true # or DINGTALK_REQUIRE_MENTION env var + # free_response_chats: # conversations that skip require_mention + # - cidABC== + # mention_patterns: # regex wake-words (e.g. Chinese bot names) + # - "^小马" + # allowed_users: # staff_id or sender_id list; "*" = any + # - "manager1234" extra: client_id: "your-app-key" # or DINGTALK_CLIENT_ID env var client_secret: "your-secret" # or DINGTALK_CLIENT_SECRET env var """ import asyncio +import json import logging import os import re -import time +import traceback import uuid from datetime import datetime, timezone -from typing import Any, Dict, Optional +from typing import Any, Dict, List, Optional, Set try: import dingtalk_stream - from dingtalk_stream import ChatbotHandler, ChatbotMessage + from dingtalk_stream import ChatbotMessage + from dingtalk_stream.frames import CallbackMessage, AckMessage + DINGTALK_STREAM_AVAILABLE = True except ImportError: DINGTALK_STREAM_AVAILABLE = False dingtalk_stream = None # type: ignore[assignment] + ChatbotMessage = None # type: ignore[assignment] + CallbackMessage = None # type: ignore[assignment] + AckMessage = type( + "AckMessage", + (), + { + "STATUS_OK": 200, + "STATUS_SYSTEM_EXCEPTION": 500, + }, + ) # type: ignore[assignment] try: import httpx + HTTPX_AVAILABLE = True except ImportError: HTTPX_AVAILABLE = False httpx = None # type: ignore[assignment] +# Card SDK for AI Cards (following QwenPaw pattern) +try: + from alibabacloud_dingtalk.card_1_0 import ( + client as dingtalk_card_client, + models as dingtalk_card_models, + ) + from alibabacloud_dingtalk.robot_1_0 import ( + client as dingtalk_robot_client, + models as dingtalk_robot_models, + ) + from alibabacloud_tea_openapi import models as open_api_models + from alibabacloud_tea_util import models as tea_util_models + + CARD_SDK_AVAILABLE = True +except ImportError: + CARD_SDK_AVAILABLE = False + dingtalk_card_client = None + dingtalk_card_models = None + dingtalk_robot_client = None + dingtalk_robot_models = None + open_api_models = None + tea_util_models = None + from gateway.config import Platform, PlatformConfig from gateway.platforms.helpers import MessageDeduplicator from gateway.platforms.base import ( @@ -55,7 +101,13 @@ MAX_MESSAGE_LENGTH = 20000 RECONNECT_BACKOFF = [2, 5, 10, 30, 60] _SESSION_WEBHOOKS_MAX = 500 -_DINGTALK_WEBHOOK_RE = re.compile(r'^https://api\.dingtalk\.com/') +_DINGTALK_WEBHOOK_RE = re.compile(r'^https://(?:api|oapi)\.dingtalk\.com/') + +# DingTalk message type → runtime content type +DINGTALK_TYPE_MAPPING = { + "picture": "image", + "voice": "audio", +} def check_dingtalk_requirements() -> bool: @@ -73,46 +125,136 @@ class DingTalkAdapter(BasePlatformAdapter): The dingtalk-stream SDK maintains a long-lived WebSocket connection. Incoming messages arrive via a ChatbotHandler callback. Replies are sent via the incoming message's session_webhook URL using httpx. + + Features: + - Text messages (plain + rich text) + - Images, audio, video, files (via download codes) + - Group chat @mention detection + - Session webhook caching with expiry tracking + - Markdown formatted replies """ MAX_MESSAGE_LENGTH = MAX_MESSAGE_LENGTH + @property + def SUPPORTS_MESSAGE_EDITING(self) -> bool: # noqa: N802 + """Edits only meaningful when AI Cards are configured. + + The gateway gates streaming cursor + edit behaviour on this flag, + so we must reflect the actual adapter capability at runtime. + """ + return bool(self._card_template_id and self._card_sdk) + + @property + def REQUIRES_EDIT_FINALIZE(self) -> bool: # noqa: N802 + """AI Card lifecycle requires an explicit ``finalize=True`` edit + to close the streaming indicator, even when the final content is + identical to the last streamed update. Enabled only when cards + are configured — webhook-only DingTalk doesn't need it. + """ + return bool(self._card_template_id and self._card_sdk) + def __init__(self, config: PlatformConfig): super().__init__(config, Platform.DINGTALK) extra = config.extra or {} - self._client_id: str = extra.get("client_id") or os.getenv("DINGTALK_CLIENT_ID", "") - self._client_secret: str = extra.get("client_secret") or os.getenv("DINGTALK_CLIENT_SECRET", "") + self._client_id: str = extra.get("client_id") or os.getenv( + "DINGTALK_CLIENT_ID", "" + ) + self._client_secret: str = extra.get("client_secret") or os.getenv( + "DINGTALK_CLIENT_SECRET", "" + ) + + # Group-chat gating (mirrors Slack/Telegram/Discord/WhatsApp conventions). + # Mention state is the structured ``is_in_at_list`` attribute from the + # dingtalk-stream SDK (set from the callback's ``isInAtList`` flag), + # not text parsing. + self._mention_patterns: List[re.Pattern] = self._compile_mention_patterns() + self._allowed_users: Set[str] = self._load_allowed_users() self._stream_client: Any = None self._stream_task: Optional[asyncio.Task] = None self._http_client: Optional["httpx.AsyncClient"] = None + self._card_sdk: Optional[Any] = None + self._robot_sdk: Optional[Any] = None + self._robot_code: str = extra.get("robot_code") or self._client_id # Message deduplication self._dedup = MessageDeduplicator(max_size=1000) - # Map chat_id -> session_webhook for reply routing - self._session_webhooks: Dict[str, str] = {} + # Map chat_id -> (session_webhook, expired_time_ms) for reply routing + self._session_webhooks: Dict[str, tuple[str, int]] = {} + # Map chat_id -> last inbound ChatbotMessage. Keyed by chat_id instead + # of a single class attribute to avoid cross-message clobbering when + # multiple conversations run concurrently. + self._message_contexts: Dict[str, Any] = {} + self._card_template_id: Optional[str] = extra.get("card_template_id") + + # Chats for which we've already fired the Done reaction — prevents + # double-firing across segment boundaries or parallel flows + # (tool-progress + stream-consumer both finalizing their cards). + # Reset each inbound message. + self._done_emoji_fired: Set[str] = set() + # Cards in streaming state per chat: chat_id -> { out_track_id -> last_content }. + # Every `send()` creates+finalizes a card (closed state). A subsequent + # `edit_message(finalize=False)` re-opens the card (DingTalk's API + # allows streaming_update on a finalized card — it flips back to + # streaming). We track those reopened cards so the next `send()` can + # auto-close them as siblings — otherwise tool-progress cards get + # stuck in streaming state forever. + self._streaming_cards: Dict[str, Dict[str, str]] = {} + # Track fire-and-forget emoji/reaction coroutines so Python's GC + # doesn't drop them mid-flight, and we can cancel them on disconnect. + self._bg_tasks: Set[asyncio.Task] = set() # -- Connection lifecycle ----------------------------------------------- async def connect(self) -> bool: """Connect to DingTalk via Stream Mode.""" if not DINGTALK_STREAM_AVAILABLE: - logger.warning("[%s] dingtalk-stream not installed. Run: pip install dingtalk-stream", self.name) + logger.warning( + "[%s] dingtalk-stream not installed. Run: pip install 'dingtalk-stream>=0.20'", + self.name, + ) return False if not HTTPX_AVAILABLE: - logger.warning("[%s] httpx not installed. Run: pip install httpx", self.name) + logger.warning( + "[%s] httpx not installed. Run: pip install httpx", self.name + ) return False if not self._client_id or not self._client_secret: - logger.warning("[%s] DINGTALK_CLIENT_ID and DINGTALK_CLIENT_SECRET required", self.name) + logger.warning( + "[%s] DINGTALK_CLIENT_ID and DINGTALK_CLIENT_SECRET required", self.name + ) return False try: self._http_client = httpx.AsyncClient(timeout=30.0) - credential = dingtalk_stream.Credential(self._client_id, self._client_secret) + credential = dingtalk_stream.Credential( + self._client_id, self._client_secret + ) self._stream_client = dingtalk_stream.DingTalkStreamClient(credential) + # Initialize card SDK if available and configured + if CARD_SDK_AVAILABLE and self._card_template_id: + sdk_config = open_api_models.Config() + sdk_config.protocol = "https" + sdk_config.region_id = "central" + self._card_sdk = dingtalk_card_client.Client(sdk_config) + self._robot_sdk = dingtalk_robot_client.Client(sdk_config) + logger.info( + "[%s] Card SDK initialized with template: %s", + self.name, + self._card_template_id, + ) + elif CARD_SDK_AVAILABLE: + # Initialize robot SDK even without card template (for media download) + sdk_config = open_api_models.Config() + sdk_config.protocol = "https" + sdk_config.region_id = "central" + self._robot_sdk = dingtalk_robot_client.Client(sdk_config) + logger.info("[%s] Robot SDK initialized (media download)", self.name) + # Capture the current event loop for cross-thread dispatch loop = asyncio.get_running_loop() handler = _IncomingHandler(self, loop) @@ -129,12 +271,12 @@ async def connect(self) -> bool: return False async def _run_stream(self) -> None: - """Run the blocking stream client with auto-reconnection.""" + """Run the async stream client with auto-reconnection.""" backoff_idx = 0 while self._running: try: logger.debug("[%s] Starting stream client...", self.name) - await asyncio.to_thread(self._stream_client.start) + await self._stream_client.start() except asyncio.CancelledError: return except Exception as e: @@ -155,37 +297,240 @@ async def disconnect(self) -> None: self._running = False self._mark_disconnected() + # Close the active websocket first so the stream task sees the + # disconnection and exits cleanly, rather than getting stuck + # awaiting frames that will never arrive. + websocket = getattr(self._stream_client, "websocket", None) if self._stream_client else None + if websocket is not None: + try: + await websocket.close() + except Exception as e: + logger.debug("[%s] websocket close during disconnect failed: %s", self.name, e) + if self._stream_task: + # Try graceful close first if SDK supports it. The SDK's close() + # is sync and may block on network I/O, so offload to a thread. + if hasattr(self._stream_client, "close"): + try: + await asyncio.to_thread(self._stream_client.close) + except Exception: + pass + self._stream_task.cancel() try: - await self._stream_task - except asyncio.CancelledError: - pass + await asyncio.wait_for(self._stream_task, timeout=5.0) + except (asyncio.CancelledError, asyncio.TimeoutError): + logger.debug("[%s] stream task did not exit cleanly during disconnect", self.name) self._stream_task = None + # Cancel any in-flight background tasks (emoji reactions, etc.) + if self._bg_tasks: + for task in list(self._bg_tasks): + task.cancel() + await asyncio.gather(*self._bg_tasks, return_exceptions=True) + self._bg_tasks.clear() + if self._http_client: await self._http_client.aclose() self._http_client = None self._stream_client = None self._session_webhooks.clear() + self._message_contexts.clear() + self._streaming_cards.clear() + self._done_emoji_fired.clear() self._dedup.clear() logger.info("[%s] Disconnected", self.name) + # -- Group gating -------------------------------------------------------- + + def _dingtalk_require_mention(self) -> bool: + """Return whether group chats should require an explicit bot trigger.""" + configured = self.config.extra.get("require_mention") + if configured is not None: + if isinstance(configured, str): + return configured.lower() in ("true", "1", "yes", "on") + return bool(configured) + return os.getenv("DINGTALK_REQUIRE_MENTION", "false").lower() in ("true", "1", "yes", "on") + + def _dingtalk_free_response_chats(self) -> Set[str]: + raw = self.config.extra.get("free_response_chats") + if raw is None: + raw = os.getenv("DINGTALK_FREE_RESPONSE_CHATS", "") + if isinstance(raw, list): + return {str(part).strip() for part in raw if str(part).strip()} + return {part.strip() for part in str(raw).split(",") if part.strip()} + + def _compile_mention_patterns(self) -> List[re.Pattern]: + """Compile optional regex wake-word patterns for group triggers.""" + patterns = self.config.extra.get("mention_patterns") if self.config.extra else None + if patterns is None: + raw = os.getenv("DINGTALK_MENTION_PATTERNS", "").strip() + if raw: + try: + loaded = json.loads(raw) + except Exception: + loaded = [part.strip() for part in raw.splitlines() if part.strip()] + if not loaded: + loaded = [part.strip() for part in raw.split(",") if part.strip()] + patterns = loaded + + if patterns is None: + return [] + if isinstance(patterns, str): + patterns = [patterns] + if not isinstance(patterns, list): + logger.warning( + "[%s] dingtalk mention_patterns must be a list or string; got %s", + self.name, + type(patterns).__name__, + ) + return [] + + compiled: List[re.Pattern] = [] + for pattern in patterns: + if not isinstance(pattern, str) or not pattern.strip(): + continue + try: + compiled.append(re.compile(pattern, re.IGNORECASE)) + except re.error as exc: + logger.warning("[%s] Invalid DingTalk mention pattern %r: %s", self.name, pattern, exc) + if compiled: + logger.info("[%s] Loaded %d DingTalk mention pattern(s)", self.name, len(compiled)) + return compiled + + def _load_allowed_users(self) -> Set[str]: + """Load allowed-users list from config.extra or env var. + + IDs are matched case-insensitively against the sender's ``staff_id`` and + ``sender_id``. A wildcard ``*`` disables the check. + """ + raw = self.config.extra.get("allowed_users") if self.config.extra else None + if raw is None: + raw = os.getenv("DINGTALK_ALLOWED_USERS", "") + if isinstance(raw, list): + items = [str(part).strip() for part in raw if str(part).strip()] + else: + items = [part.strip() for part in str(raw).split(",") if part.strip()] + return {item.lower() for item in items} + + def _is_user_allowed(self, sender_id: str, sender_staff_id: str) -> bool: + if not self._allowed_users or "*" in self._allowed_users: + return True + candidates = {(sender_id or "").lower(), (sender_staff_id or "").lower()} + candidates.discard("") + return bool(candidates & self._allowed_users) + + def _message_mentions_bot(self, message: "ChatbotMessage") -> bool: + """True if the bot was @-mentioned in a group message. + + dingtalk-stream sets ``is_in_at_list`` on the incoming ChatbotMessage + when the bot is addressed via @-mention. + """ + return bool(getattr(message, "is_in_at_list", False)) + + def _message_matches_mention_patterns(self, text: str) -> bool: + if not text or not self._mention_patterns: + return False + return any(pattern.search(text) for pattern in self._mention_patterns) + + def _should_process_message(self, message: "ChatbotMessage", text: str, is_group: bool, chat_id: str) -> bool: + """Apply DingTalk group trigger rules. + + DMs remain unrestricted (subject to ``allowed_users`` which is enforced + earlier). Group messages are accepted when: + - the chat is explicitly allowlisted in ``free_response_chats`` + - ``require_mention`` is disabled + - the bot is @mentioned (``is_in_at_list``) + - the text matches a configured regex wake-word pattern + """ + if not is_group: + return True + if chat_id and chat_id in self._dingtalk_free_response_chats(): + return True + if not self._dingtalk_require_mention(): + return True + if self._message_mentions_bot(message): + return True + return self._message_matches_mention_patterns(text) + + def _spawn_bg(self, coro) -> None: + """Start a fire-and-forget coroutine and track it for cleanup.""" + task = asyncio.create_task(coro) + self._bg_tasks.add(task) + task.add_done_callback(self._bg_tasks.discard) + + # -- AI Card lifecycle helpers ------------------------------------------ + + async def _close_streaming_siblings(self, chat_id: str) -> None: + """Finalize any previously-open streaming cards for this chat. + + Called at the start of every ``send()`` so lingering tool-progress + cards that were reopened by ``edit_message(finalize=False)`` get + cleanly closed before the next card is created. Without this, + tool-progress cards stay stuck in streaming state after the agent + moves on (there is no explicit "turn end" signal from the gateway). + """ + cards = self._streaming_cards.pop(chat_id, None) + if not cards: + return + token = await self._get_access_token() + if not token: + return + for out_track_id, last_content in list(cards.items()): + try: + await self._stream_card_content( + out_track_id, token, last_content, finalize=True, + ) + logger.debug( + "[%s] AI Card sibling closed: %s", + self.name, out_track_id, + ) + except Exception as e: + logger.debug( + "[%s] Sibling close failed for %s: %s", + self.name, out_track_id, e, + ) + + def _fire_done_reaction(self, chat_id: str) -> None: + """Swap 🤔Thinking → 🥳Done on the original user message. + + Idempotent per chat_id — safe to call from segment-break flushes + and final-done flushes without double-firing. + """ + if chat_id in self._done_emoji_fired: + return + self._done_emoji_fired.add(chat_id) + msg = self._message_contexts.get(chat_id) + if not msg: + return + msg_id = getattr(msg, "message_id", "") or "" + conversation_id = getattr(msg, "conversation_id", "") or "" + if not (msg_id and conversation_id): + return + + async def _swap() -> None: + await self._send_emotion( + msg_id, conversation_id, "🤔Thinking", recall=True, + ) + await self._send_emotion( + msg_id, conversation_id, "🥳Done", recall=False, + ) + + self._spawn_bg(_swap()) + # -- Inbound message processing ----------------------------------------- - async def _on_message(self, message: "ChatbotMessage") -> None: + async def _on_message( + self, + message: "ChatbotMessage", + ) -> None: """Process an incoming DingTalk chatbot message.""" msg_id = getattr(message, "message_id", None) or uuid.uuid4().hex if self._dedup.is_duplicate(msg_id): logger.debug("[%s] Duplicate message %s, skipping", self.name, msg_id) return - text = self._extract_text(message) - if not text: - logger.debug("[%s] Empty message, skipping", self.name) - return - # Chat context conversation_id = getattr(message, "conversation_id", "") or "" conversation_type = getattr(message, "conversation_type", "1") @@ -197,16 +542,62 @@ async def _on_message(self, message: "ChatbotMessage") -> None: chat_id = conversation_id or sender_id chat_type = "group" if is_group else "dm" - # Store session webhook for reply routing (validate origin to prevent SSRF) + # Allowed-users gate (applies to both DM and group) + if not self._is_user_allowed(sender_id, sender_staff_id): + logger.debug( + "[%s] Dropping message from non-allowlisted user staff_id=%s sender_id=%s", + self.name, sender_staff_id, sender_id, + ) + return + + # Group mention/pattern gate. DMs pass through unconditionally. + # We need the message text for regex wake-word matching; extract it + # early but don't consume the rest of the pipeline until after the + # gate decides whether to process. + _early_text = self._extract_text(message) or "" + if not self._should_process_message(message, _early_text, is_group, chat_id): + logger.debug( + "[%s] Dropping group message that failed mention gate message_id=%s chat_id=%s", + self.name, msg_id, chat_id, + ) + return + + # Stash the incoming message keyed by chat_id so concurrent + # conversations don't clobber each other's context. Also reset + # the per-chat "Done emoji fired" marker so a new inbound message + # gets its own Thinking→Done cycle. + if chat_id: + self._message_contexts[chat_id] = message + self._done_emoji_fired.discard(chat_id) + + # Store session webhook session_webhook = getattr(message, "session_webhook", None) or "" + session_webhook_expired_time = ( + getattr(message, "session_webhook_expired_time", 0) or 0 + ) if session_webhook and chat_id and _DINGTALK_WEBHOOK_RE.match(session_webhook): if len(self._session_webhooks) >= _SESSION_WEBHOOKS_MAX: - # Evict oldest entry to cap memory growth try: self._session_webhooks.pop(next(iter(self._session_webhooks))) except StopIteration: pass - self._session_webhooks[chat_id] = session_webhook + self._session_webhooks[chat_id] = ( + session_webhook, + session_webhook_expired_time, + ) + + # Resolve media download codes to URLs so vision tools can use them + await self._resolve_media_codes(message) + + # Extract text content + text = self._extract_text(message) + + # Determine message type and build media list + msg_type, media_urls, media_types = self._extract_media(message) + + if not text and not media_urls: + logger.debug("[%s] Empty message, skipping", self.name) + return source = self.build_source( chat_id=chat_id, @@ -220,41 +611,141 @@ async def _on_message(self, message: "ChatbotMessage") -> None: # Parse timestamp create_at = getattr(message, "create_at", None) try: - timestamp = datetime.fromtimestamp(int(create_at) / 1000, tz=timezone.utc) if create_at else datetime.now(tz=timezone.utc) + timestamp = ( + datetime.fromtimestamp(int(create_at) / 1000, tz=timezone.utc) + if create_at + else datetime.now(tz=timezone.utc) + ) except (ValueError, OSError, TypeError): timestamp = datetime.now(tz=timezone.utc) event = MessageEvent( text=text, - message_type=MessageType.TEXT, + message_type=msg_type, source=source, message_id=msg_id, raw_message=message, + media_urls=media_urls, + media_types=media_types, timestamp=timestamp, ) - logger.debug("[%s] Message from %s in %s: %s", - self.name, sender_nick, chat_id[:20] if chat_id else "?", text[:50]) + logger.debug( + "[%s] Message from %s in %s: %s", + self.name, + sender_nick, + chat_id[:20] if chat_id else "?", + text[:80] if text else "(media)", + ) await self.handle_message(event) @staticmethod def _extract_text(message: "ChatbotMessage") -> str: - """Extract plain text from a DingTalk chatbot message.""" + """Extract plain text from a DingTalk chatbot message. + + Handles both legacy and current dingtalk-stream SDK payload shapes: + * legacy: ``message.text`` was a dict ``{"content": "..."}`` + * >= 0.20: ``message.text`` is a ``TextContent`` dataclass whose + ``__str__`` returns ``"TextContent(content=...)"`` — never fall + back to ``str(text)`` without extracting ``.content`` first. + * rich text moved from ``message.rich_text`` (list) to + ``message.rich_text_content.rich_text_list`` (list of dicts). + """ text = getattr(message, "text", None) or "" - if isinstance(text, dict): + + # Handle TextContent object (SDK style) + if hasattr(text, "content"): + content = (text.content or "").strip() + elif isinstance(text, dict): content = text.get("content", "").strip() else: content = str(text).strip() - # Fall back to rich text if present if not content: - rich_text = getattr(message, "rich_text", None) - if rich_text and isinstance(rich_text, list): - parts = [item["text"] for item in rich_text - if isinstance(item, dict) and item.get("text")] - content = " ".join(parts).strip() + rich_text = getattr(message, "rich_text_content", None) or getattr( + message, "rich_text", None + ) + if rich_text: + rich_list = getattr(rich_text, "rich_text_list", None) or rich_text + if isinstance(rich_list, list): + parts = [] + for item in rich_list: + if isinstance(item, dict): + t = item.get("text") or item.get("content") or "" + if t: + parts.append(t) + elif hasattr(item, "text") and item.text: + parts.append(item.text) + content = " ".join(parts).strip() + + # Do NOT strip "@bot" from the text. The mention is a routing + # signal (delivered structurally via callback `isInAtList`), and + # regex-stripping @handles would collateral-damage e-mails + # (alice@example.com), SSH URLs (git@github.com), and literal + # references the user wrote ("what does @openai think"). Let the + # LLM see the raw text — it handles "@bot hello" cleanly. return content + def _extract_media(self, message: "ChatbotMessage"): + """Extract media info from message. Returns (MessageType, [urls], [mime_types]).""" + msg_type = MessageType.TEXT + media_urls = [] + media_types = [] + + # Check for image/picture + image_content = getattr(message, "image_content", None) + if image_content: + download_code = getattr(image_content, "download_code", None) + if download_code: + media_urls.append(download_code) + media_types.append("image") + msg_type = MessageType.PHOTO + + # Check for rich text with mixed content + rich_text = getattr(message, "rich_text_content", None) or getattr( + message, "rich_text", None + ) + if rich_text: + rich_list = getattr(rich_text, "rich_text_list", None) or rich_text + if isinstance(rich_list, list): + for item in rich_list: + if isinstance(item, dict): + dl_code = ( + item.get("downloadCode") or item.get("download_code") or "" + ) + item_type = item.get("type", "") + if dl_code: + mapped = DINGTALK_TYPE_MAPPING.get(item_type, "file") + media_urls.append(dl_code) + if mapped == "image": + media_types.append("image") + if msg_type == MessageType.TEXT: + msg_type = MessageType.PHOTO + elif mapped == "audio": + media_types.append("audio") + if msg_type == MessageType.TEXT: + msg_type = MessageType.AUDIO + elif mapped == "video": + media_types.append("video") + if msg_type == MessageType.TEXT: + msg_type = MessageType.VIDEO + else: + media_types.append("application/octet-stream") + if msg_type == MessageType.TEXT: + msg_type = MessageType.DOCUMENT + + msg_type_str = getattr(message, "message_type", "") or "" + if msg_type_str == "picture" and not media_urls: + msg_type = MessageType.PHOTO + elif msg_type_str == "richText": + msg_type = ( + MessageType.PHOTO + if any("image" in t for t in media_types) + else MessageType.TEXT + ) + + return msg_type, media_urls, media_types + # -- Outbound messaging ------------------------------------------------- async def send( @@ -266,29 +757,101 @@ async def send( ) -> SendResult: """Send a markdown reply via DingTalk session webhook.""" metadata = metadata or {} + logger.debug( + "[%s] send() chat_id=%s card_enabled=%s", + self.name, + chat_id, + bool(self._card_template_id and self._card_sdk), + ) - session_webhook = metadata.get("session_webhook") or self._session_webhooks.get(chat_id) + # Check metadata first (for direct webhook sends) + session_webhook = metadata.get("session_webhook") if not session_webhook: - return SendResult(success=False, - error="No session_webhook available. Reply must follow an incoming message.") + webhook_info = self._get_valid_webhook(chat_id) + if not webhook_info: + logger.warning( + "[%s] No valid session_webhook for chat_id=%s", + self.name, chat_id, + ) + return SendResult( + success=False, + error="No valid session_webhook available. Reply must follow an incoming message.", + ) + session_webhook, _ = webhook_info if not self._http_client: return SendResult(success=False, error="HTTP client not initialized") + # Look up the inbound message for this chat (for AI Card routing) + current_message = self._message_contexts.get(chat_id) + + # ``reply_to`` is the signal that this send is the FINAL response + # to an inbound user message — only `base.py:_send_with_retry` sets + # it. Tool-progress, commentary, and stream-consumer first-sends + # all leave it None. We use it for two orthogonal decisions: + # 1. finalize on create? Yes if final reply, No if intermediate + # (intermediate cards stay in streaming state so edit_message + # updates don't flicker closed→streaming→closed repeatedly). + # 2. fire Done reaction? Only when this is the final reply. + is_final_reply = reply_to is not None + + # Try AI Card first (using alibabacloud_dingtalk.card_1_0 SDK). + if self._card_template_id and current_message and self._card_sdk: + # Close any previously-open streaming cards for this chat + # before creating a new one (handles tool-progress → final- + # response handoff; also cleans up lingering commentary cards). + await self._close_streaming_siblings(chat_id) + + result = await self._create_and_stream_card( + chat_id, current_message, content, + finalize=is_final_reply, + ) + if result and result.success: + if is_final_reply: + # Final reply: card closed, swap Thinking → Done. + self._fire_done_reaction(chat_id) + else: + # Intermediate (tool progress / commentary / streaming + # first chunk): keep the card open and track it so the + # next send() auto-closes it as a sibling, or + # edit_message(finalize=True) closes it explicitly. + self._streaming_cards.setdefault(chat_id, {})[ + result.message_id + ] = content + return result + + logger.warning("[%s] AI Card send failed, falling back to webhook", self.name) + + logger.debug("[%s] Sending via webhook", self.name) + # Normalize markdown for DingTalk + normalized = self._normalize_markdown(content[: self.MAX_MESSAGE_LENGTH]) + payload = { "msgtype": "markdown", - "markdown": {"title": "Hermes", "text": content[:self.MAX_MESSAGE_LENGTH]}, + "markdown": {"title": "Hermes", "text": normalized}, } try: - resp = await self._http_client.post(session_webhook, json=payload, timeout=15.0) + resp = await self._http_client.post( + session_webhook, json=payload, timeout=15.0 + ) if resp.status_code < 300: + # Webhook path: fire Done only for final replies, same as + # the card path. + if is_final_reply: + self._fire_done_reaction(chat_id) return SendResult(success=True, message_id=uuid.uuid4().hex[:12]) body = resp.text - logger.warning("[%s] Send failed HTTP %d: %s", self.name, resp.status_code, body[:200]) - return SendResult(success=False, error=f"HTTP {resp.status_code}: {body[:200]}") + logger.warning( + "[%s] Send failed HTTP %d: %s", self.name, resp.status_code, body[:200] + ) + return SendResult( + success=False, error=f"HTTP {resp.status_code}: {body[:200]}" + ) except httpx.TimeoutException: - return SendResult(success=False, error="Timeout sending message to DingTalk") + return SendResult( + success=False, error="Timeout sending message to DingTalk" + ) except Exception as e: logger.error("[%s] Send error: %s", self.name, e) return SendResult(success=False, error=str(e)) @@ -299,36 +862,501 @@ async def send_typing(self, chat_id: str, metadata=None) -> None: async def get_chat_info(self, chat_id: str) -> Dict[str, Any]: """Return basic info about a DingTalk conversation.""" - return {"name": chat_id, "type": "group" if "group" in chat_id.lower() else "dm"} + return { + "name": chat_id, + "type": "group" if "group" in chat_id.lower() else "dm", + } + + def _get_valid_webhook(self, chat_id: str) -> Optional[tuple[str, int]]: + """Get a valid (non-expired) session webhook for the given chat_id.""" + info = self._session_webhooks.get(chat_id) + if not info: + return None + webhook, expired_time_ms = info + # Check expiry with 5-minute safety margin + if expired_time_ms and expired_time_ms > 0: + now_ms = int(datetime.now(tz=timezone.utc).timestamp() * 1000) + safety_margin_ms = 5 * 60 * 1000 + if now_ms + safety_margin_ms >= expired_time_ms: + # Expired, remove from cache + self._session_webhooks.pop(chat_id, None) + return None + return info + + async def _create_and_stream_card( + self, + chat_id: str, + message: Any, + content: str, + *, + finalize: bool = True, + ) -> Optional[SendResult]: + """Create an AI Card, deliver it to the conversation, and stream initial content. + + Always called with ``finalize=True`` from ``send()`` (closed state). + If the caller later issues ``edit_message(finalize=False)``, the + DingTalk streaming_update API reopens the card into streaming + state, and we track that in ``_streaming_cards`` for sibling + cleanup on the next send. + """ + try: + token = await self._get_access_token() + if not token: + return None + + out_track_id = f"hermes_{uuid.uuid4().hex[:12]}" + + conversation_id = getattr(message, "conversation_id", "") or "" + conversation_type = getattr(message, "conversation_type", "1") + is_group = str(conversation_type) == "2" + sender_staff_id = getattr(message, "sender_staff_id", "") or "" + + runtime = tea_util_models.RuntimeOptions() + + # Step 1: Create card with STREAM callback type + create_request = dingtalk_card_models.CreateCardRequest( + card_template_id=self._card_template_id, + out_track_id=out_track_id, + card_data=dingtalk_card_models.CreateCardRequestCardData( + card_param_map={"content": ""}, + ), + callback_type="STREAM", + im_group_open_space_model=( + dingtalk_card_models.CreateCardRequestImGroupOpenSpaceModel( + support_forward=True, + ) + ), + im_robot_open_space_model=( + dingtalk_card_models.CreateCardRequestImRobotOpenSpaceModel( + support_forward=True, + ) + ), + ) + + create_headers = dingtalk_card_models.CreateCardHeaders( + x_acs_dingtalk_access_token=token, + ) + + await self._card_sdk.create_card_with_options_async( + create_request, create_headers, runtime + ) + + # Step 2: Deliver card to the conversation + if is_group: + open_space_id = f"dtv1.card//IM_GROUP.{conversation_id}" + deliver_request = dingtalk_card_models.DeliverCardRequest( + out_track_id=out_track_id, + user_id_type=1, + open_space_id=open_space_id, + im_group_open_deliver_model=( + dingtalk_card_models.DeliverCardRequestImGroupOpenDeliverModel( + robot_code=self._robot_code, + ) + ), + ) + else: + if not sender_staff_id: + logger.warning( + "[%s] AI Card skipped: missing sender_staff_id for DM", + self.name, + ) + return None + open_space_id = f"dtv1.card//IM_ROBOT.{sender_staff_id}" + deliver_request = dingtalk_card_models.DeliverCardRequest( + out_track_id=out_track_id, + user_id_type=1, + open_space_id=open_space_id, + im_robot_open_deliver_model=( + dingtalk_card_models.DeliverCardRequestImRobotOpenDeliverModel( + space_type="IM_ROBOT", + ) + ), + ) + + deliver_headers = dingtalk_card_models.DeliverCardHeaders( + x_acs_dingtalk_access_token=token, + ) + + await self._card_sdk.deliver_card_with_options_async( + deliver_request, deliver_headers, runtime + ) + + # Step 3: Stream initial content. finalize=True closes the + # card immediately (one-shot); finalize=False keeps it open + # for streaming edit_message updates by out_track_id. + await self._stream_card_content( + out_track_id, token, content, finalize=finalize, + ) + + logger.info( + "[%s] AI Card %s: %s", + self.name, + "created+finalized" if finalize else "created (streaming)", + out_track_id, + ) + return SendResult(success=True, message_id=out_track_id) + + except Exception as e: + logger.warning( + "[%s] AI Card create failed: %s\n%s", + self.name, e, traceback.format_exc(), + ) + return None + + async def edit_message( + self, + chat_id: str, + message_id: str, + content: str, + *, + finalize: bool = False, + ) -> SendResult: + """Edit an AI Card by streaming updated content. + + ``message_id`` is the out_track_id returned by the initial ``send()`` + call that created this card. Callers (stream_consumer, tool + progress) track their own ids independently so two parallel flows + on the same chat_id don't interfere. + """ + if not message_id: + return SendResult(success=False, error="message_id required") + token = await self._get_access_token() + if not token: + return SendResult(success=False, error="No access token") + + try: + await self._stream_card_content( + message_id, token, content, finalize=finalize, + ) + if finalize: + # Remove from streaming-cards tracking and fire Done. This + # is the canonical "response ended" signal from stream + # consumer's final edit. + self._streaming_cards.get(chat_id, {}).pop(message_id, None) + if not self._streaming_cards.get(chat_id): + self._streaming_cards.pop(chat_id, None) + logger.debug( + "[%s] AI Card finalized (edit): %s", + self.name, message_id, + ) + self._fire_done_reaction(chat_id) + else: + # Non-final edit reopens the card into streaming state — + # track it so the next send() can auto-close it as a + # sibling. + self._streaming_cards.setdefault(chat_id, {})[message_id] = content + return SendResult(success=True, message_id=message_id) + except Exception as e: + logger.warning("[%s] Card edit failed: %s", self.name, e) + return SendResult(success=False, error=str(e)) + + async def _stream_card_content( + self, + out_track_id: str, + token: str, + content: str, + finalize: bool = False, + ) -> None: + """Stream content to an existing AI Card.""" + stream_request = dingtalk_card_models.StreamingUpdateRequest( + out_track_id=out_track_id, + guid=str(uuid.uuid4()), + key="content", + content=content[: self.MAX_MESSAGE_LENGTH], + is_full=True, + is_finalize=finalize, + is_error=False, + ) + + stream_headers = dingtalk_card_models.StreamingUpdateHeaders( + x_acs_dingtalk_access_token=token, + ) + + runtime = tea_util_models.RuntimeOptions() + await self._card_sdk.streaming_update_with_options_async( + stream_request, stream_headers, runtime + ) + + async def _get_access_token(self) -> Optional[str]: + """Get access token using SDK's cached token.""" + if not self._stream_client: + return None + try: + # SDK's get_access_token is sync and uses requests + token = await asyncio.to_thread(self._stream_client.get_access_token) + return token + except Exception as e: + logger.error("[%s] Failed to get access token: %s", self.name, e) + return None + + async def _send_emotion( + self, + open_msg_id: str, + open_conversation_id: str, + emoji_name: str, + *, + recall: bool = False, + ) -> None: + """Add or recall an emoji reaction on a message.""" + if not self._robot_sdk or not open_msg_id or not open_conversation_id: + return + action = "recall" if recall else "reply" + try: + token = await self._get_access_token() + if not token: + return + + emotion_kwargs = { + "robot_code": self._robot_code, + "open_msg_id": open_msg_id, + "open_conversation_id": open_conversation_id, + "emotion_type": 2, + "emotion_name": emoji_name, + } + runtime = tea_util_models.RuntimeOptions() + + if recall: + emotion_kwargs["text_emotion"] = ( + dingtalk_robot_models.RobotRecallEmotionRequestTextEmotion( + emotion_id="2659900", + emotion_name=emoji_name, + text=emoji_name, + background_id="im_bg_1", + ) + ) + request = dingtalk_robot_models.RobotRecallEmotionRequest( + **emotion_kwargs, + ) + sdk_headers = dingtalk_robot_models.RobotRecallEmotionHeaders( + x_acs_dingtalk_access_token=token, + ) + await self._robot_sdk.robot_recall_emotion_with_options_async( + request, sdk_headers, runtime + ) + else: + emotion_kwargs["text_emotion"] = ( + dingtalk_robot_models.RobotReplyEmotionRequestTextEmotion( + emotion_id="2659900", + emotion_name=emoji_name, + text=emoji_name, + background_id="im_bg_1", + ) + ) + request = dingtalk_robot_models.RobotReplyEmotionRequest( + **emotion_kwargs, + ) + sdk_headers = dingtalk_robot_models.RobotReplyEmotionHeaders( + x_acs_dingtalk_access_token=token, + ) + await self._robot_sdk.robot_reply_emotion_with_options_async( + request, sdk_headers, runtime + ) + logger.info( + "[%s] _send_emotion: %s %s on msg=%s", + self.name, action, emoji_name, open_msg_id[:24], + ) + except Exception: + logger.debug( + "[%s] _send_emotion %s failed", self.name, action, exc_info=True + ) + + async def _resolve_media_codes(self, message: "ChatbotMessage") -> None: + """Resolve download codes in message to actual URLs.""" + token = await self._get_access_token() + if not token: + return + + robot_code = getattr(message, "robot_code", None) or self._client_id + codes_to_resolve = [] + + # Collect codes and references to update + # 1. Single image content + img_content = getattr(message, "image_content", None) + if img_content and getattr(img_content, "download_code", None): + codes_to_resolve.append((img_content, "download_code")) + + # 2. Rich text list + rich_text = getattr(message, "rich_text_content", None) + if rich_text: + rich_list = getattr(rich_text, "rich_text_list", []) or [] + for item in rich_list: + if isinstance(item, dict): + for key in ("downloadCode", "pictureDownloadCode", "download_code"): + if item.get(key): + codes_to_resolve.append((item, key)) + + if not codes_to_resolve: + return + + # Resolve all codes in parallel + tasks = [] + for obj, key in codes_to_resolve: + code = getattr(obj, key, None) if hasattr(obj, key) else obj.get(key) + if code: + tasks.append( + self._fetch_download_url(code, robot_code, token, obj, key) + ) + + await asyncio.gather(*tasks, return_exceptions=True) + + async def _fetch_download_url( + self, code: str, robot_code: str, token: str, obj, key: str + ) -> None: + """Fetch download URL for a single code using the robot SDK.""" + if not self._robot_sdk: + logger.warning( + "[%s] Robot SDK not initialized, cannot resolve media code", + self.name, + ) + return + try: + request = dingtalk_robot_models.RobotMessageFileDownloadRequest( + download_code=code, + robot_code=robot_code, + ) + headers = dingtalk_robot_models.RobotMessageFileDownloadHeaders( + x_acs_dingtalk_access_token=token, + ) + runtime = tea_util_models.RuntimeOptions() + response = await self._robot_sdk.robot_message_file_download_with_options_async( + request, headers, runtime + ) + body = response.body if response else None + if body: + url = getattr(body, "download_url", None) + if url: + if hasattr(obj, key): + setattr(obj, key, url) + elif isinstance(obj, dict): + obj[key] = url + else: + logger.warning( + "[%s] Failed to download media: empty response for code %s", + self.name, + code, + ) + except Exception as e: + logger.error("[%s] Error resolving media code %s: %s", self.name, code, e) + + @staticmethod + def _normalize_markdown(text: str) -> str: + """Normalize markdown for DingTalk's parser. + + DingTalk's markdown renderer has quirks: + - Numbered lists need blank line before them + - Indented code blocks may render incorrectly + """ + lines = text.split("\n") + out = [] + for i, line in enumerate(lines): + # Ensure blank line before numbered list items + is_numbered = re.match(r"^\d+\.\s", line.strip()) + if is_numbered and i > 0: + prev = lines[i - 1] + if prev.strip() and not re.match(r"^\d+\.\s", prev.strip()): + out.append("") + # Dedent fenced code blocks + if line.strip().startswith("```") and line != line.lstrip(): + indent = len(line) - len(line.lstrip()) + line = line[indent:] + out.append(line) + return "\n".join(out) # --------------------------------------------------------------------------- # Internal stream handler # --------------------------------------------------------------------------- -class _IncomingHandler(ChatbotHandler if DINGTALK_STREAM_AVAILABLE else object): - """dingtalk-stream ChatbotHandler that forwards messages to the adapter.""" - def __init__(self, adapter: DingTalkAdapter, loop: asyncio.AbstractEventLoop): +class _IncomingHandler( + dingtalk_stream.ChatbotHandler if DINGTALK_STREAM_AVAILABLE else object +): + """dingtalk-stream ChatbotHandler that forwards messages to the adapter. + + SDK >= 0.20 changed process() from sync to async, and the message + parameter from ChatbotMessage to CallbackMessage. We parse the + CallbackMessage.data dict into a ChatbotMessage before forwarding. + """ + + def __init__(self, adapter: DingTalkAdapter, loop: Optional[asyncio.AbstractEventLoop] = None): if DINGTALK_STREAM_AVAILABLE: super().__init__() self._adapter = adapter self._loop = loop - def process(self, message: "ChatbotMessage"): - """Called by dingtalk-stream in its thread when a message arrives. + async def process(self, message: "CallbackMessage"): + """Called by dingtalk-stream (>=0.20) when a message arrives. - Schedules the async handler on the main event loop. - """ - loop = self._loop - if loop is None or loop.is_closed(): - logger.error("[DingTalk] Event loop unavailable, cannot dispatch message") - return dingtalk_stream.AckMessage.STATUS_OK, "OK" + dingtalk-stream >= 0.24 passes a CallbackMessage whose ``.data`` contains + the chatbot payload. Convert it to ChatbotMessage via + ``ChatbotMessage.from_dict()``. - future = asyncio.run_coroutine_threadsafe(self._adapter._on_message(message), loop) + Message processing is dispatched as a background task so that this + method returns the ACK immediately — blocking here would prevent the + SDK from sending heartbeats, eventually causing a disconnect. + """ try: - future.result(timeout=60) + # CallbackMessage.data is a dict containing the raw DingTalk payload + data = message.data + if isinstance(data, str): + data = json.loads(data) + + # Parse dict into ChatbotMessage using SDK's from_dict + chatbot_msg = ChatbotMessage.from_dict(data) + + # Ensure session_webhook is populated even if the SDK's + # from_dict() did not map it (field name mismatch across + # SDK versions). + if not getattr(chatbot_msg, "session_webhook", None): + webhook = ( + data.get("sessionWebhook") + or data.get("session_webhook") + or "" + ) if isinstance(data, dict) else "" + if webhook: + chatbot_msg.session_webhook = webhook + + # Ensure is_in_at_list is populated from the structured callback + # flag even if from_dict() did not map it. DingTalk sends + # ``isInAtList`` in the raw payload; the adapter's mention check + # reads the ChatbotMessage attribute ``is_in_at_list``. + if not getattr(chatbot_msg, "is_in_at_list", False): + raw_flag = ( + data.get("isInAtList") if isinstance(data, dict) else False + ) + if raw_flag: + chatbot_msg.is_in_at_list = True + + msg_id = getattr(chatbot_msg, "message_id", None) or "" + conversation_id = getattr(chatbot_msg, "conversation_id", None) or "" + + # Thinking reaction — fire-and-forget, tracked + if msg_id and conversation_id: + self._adapter._spawn_bg( + self._adapter._send_emotion( + msg_id, conversation_id, "🤔Thinking", recall=False, + ) + ) + + # Fire-and-forget: return ACK immediately, process in background. + # Blocking here would prevent the SDK from sending heartbeats, + # eventually causing a disconnect. _on_message is wrapped so + # exceptions inside the task surface in logs instead of + # disappearing into the event loop. + asyncio.create_task(self._safe_on_message(chatbot_msg)) except Exception: - logger.exception("[DingTalk] Error processing incoming message") + logger.exception( + "[%s] Error preparing incoming message", self._adapter.name + ) + return AckMessage.STATUS_SYSTEM_EXCEPTION, "error" + + return AckMessage.STATUS_OK, "OK" - return dingtalk_stream.AckMessage.STATUS_OK, "OK" + async def _safe_on_message(self, chatbot_msg: "ChatbotMessage") -> None: + """Wrapper that catches exceptions from _on_message.""" + try: + await self._adapter._on_message(chatbot_msg) + except Exception: + logger.exception( + "[%s] Error processing incoming message", self._adapter.name + ) diff --git a/gateway/platforms/discord.py b/gateway/platforms/discord.py index f92cdf8db0f8..f741d45b5f35 100644 --- a/gateway/platforms/discord.py +++ b/gateway/platforms/discord.py @@ -10,7 +10,6 @@ """ import asyncio -import json import logging import os import struct @@ -19,12 +18,12 @@ import threading import time from collections import defaultdict -from pathlib import Path from typing import Callable, Dict, Optional, Any logger = logging.getLogger(__name__) VALID_THREAD_AUTO_ARCHIVE_MINUTES = {60, 1440, 4320, 10080} +_DISCORD_COMMAND_SYNC_POLICIES = {"safe", "bulk", "off"} try: import discord @@ -53,7 +52,9 @@ ProcessingOutcome, SendResult, cache_image_from_url, + cache_image_from_bytes, cache_audio_from_url, + cache_audio_from_bytes, cache_document_from_bytes, SUPPORTED_DOCUMENT_TYPES, ) @@ -82,6 +83,41 @@ def check_discord_requirements() -> bool: return DISCORD_AVAILABLE +def _build_allowed_mentions(): + """Build Discord ``AllowedMentions`` with safe defaults, overridable via env. + + Discord bots default to parsing ``@everyone``, ``@here``, role pings, and + user pings when ``allowed_mentions`` is unset on the client — any LLM + output or echoed user content that contains ``@everyone`` would therefore + ping the whole server. We explicitly deny ``@everyone`` and role pings + by default and keep user / replied-user pings enabled so normal + conversation still works. + + Override via environment variables (or ``discord.allow_mentions.*`` in + config.yaml): + + DISCORD_ALLOW_MENTION_EVERYONE default false — @everyone + @here + DISCORD_ALLOW_MENTION_ROLES default false — @role pings + DISCORD_ALLOW_MENTION_USERS default true — @user pings + DISCORD_ALLOW_MENTION_REPLIED_USER default true — reply-ping author + """ + if not DISCORD_AVAILABLE: + return None + + def _b(name: str, default: bool) -> bool: + raw = os.getenv(name, "").strip().lower() + if not raw: + return default + return raw in ("true", "1", "yes", "on") + + return discord.AllowedMentions( + everyone=_b("DISCORD_ALLOW_MENTION_EVERYONE", False), + roles=_b("DISCORD_ALLOW_MENTION_ROLES", False), + users=_b("DISCORD_ALLOW_MENTION_USERS", True), + replied_user=_b("DISCORD_ALLOW_MENTION_REPLIED_USER", True), + ) + + class VoiceReceiver: """Captures and decodes voice audio from a Discord voice channel. @@ -237,6 +273,7 @@ def _on_packet(self, data: bytes): # Calculate dynamic RTP header size (RFC 9335 / rtpsize mode) cc = first_byte & 0x0F # CSRC count has_extension = bool(first_byte & 0x10) # extension bit + has_padding = bool(first_byte & 0x20) # padding bit (RFC 3550 §5.1) header_size = 12 + (4 * cc) + (4 if has_extension else 0) if len(data) < header_size + 4: # need at least header + nonce @@ -280,6 +317,31 @@ def _on_packet(self, data: bytes): if ext_data_len and len(decrypted) > ext_data_len: decrypted = decrypted[ext_data_len:] + # --- Strip RTP padding (RFC 3550 §5.1) --- + # When the P bit is set, the last payload byte holds the count of + # trailing padding bytes (including itself) that must be removed + # before further processing. Skipping this passes padding-contaminated + # bytes into DAVE/Opus and corrupts inbound audio. + if has_padding: + if not decrypted: + if self._packet_debug_count <= 10: + logger.warning( + "RTP padding bit set but no payload (ssrc=%d)", ssrc, + ) + return + pad_len = decrypted[-1] + if pad_len == 0 or pad_len > len(decrypted): + if self._packet_debug_count <= 10: + logger.warning( + "Invalid RTP padding length %d for payload size %d (ssrc=%d)", + pad_len, len(decrypted), ssrc, + ) + return + decrypted = decrypted[:-pad_len] + if not decrypted: + # Padding consumed entire payload — nothing to decode + return + # --- DAVE E2EE decrypt --- if self._dave_session: with self._lock: @@ -434,8 +496,10 @@ def __init__(self, config: PlatformConfig): self._client: Optional[commands.Bot] = None self._ready_event = asyncio.Event() self._allowed_user_ids: set = set() # For button approval authorization + self._allowed_role_ids: set = set() # For DISCORD_ALLOWED_ROLES filtering # Voice channel state (per-guild) self._voice_clients: Dict[int, Any] = {} # guild_id -> VoiceClient + self._voice_locks: Dict[int, asyncio.Lock] = {} # guild_id -> serialize join/leave # Text batching: merge rapid successive messages (Telegram-style) self._text_batch_delay_seconds = float(os.getenv("HERMES_DISCORD_TEXT_BATCH_DELAY_SECONDS", "0.6")) self._text_batch_split_delay_seconds = float(os.getenv("HERMES_DISCORD_TEXT_BATCH_SPLIT_DELAY_SECONDS", "2.0")) @@ -464,6 +528,7 @@ def __init__(self, config: PlatformConfig): # Reply threading mode: "off" (no replies), "first" (reply on first # chunk only, default), "all" (reply-reference on every chunk). self._reply_to_mode: str = getattr(config, 'reply_to_mode', 'first') or 'first' + self._slash_commands: bool = self.config.extra.get("slash_commands", True) async def connect(self) -> bool: """Connect to Discord and start receiving events.""" @@ -478,7 +543,6 @@ async def connect(self) -> bool: # ctypes.util.find_library fails on macOS with Homebrew-installed libs, # so fall back to known Homebrew paths if needed. if not opus_path: - import sys _homebrew_paths = ( "/opt/homebrew/lib/libopus.dylib", # Apple Silicon "/usr/local/lib/libopus.dylib", # Intel Mac @@ -512,6 +576,15 @@ async def connect(self) -> bool: if uid.strip() } + # Parse DISCORD_ALLOWED_ROLES — comma-separated role IDs. + # Users with ANY of these roles can interact with the bot. + roles_env = os.getenv("DISCORD_ALLOWED_ROLES", "") + if roles_env: + self._allowed_role_ids = { + int(rid.strip()) for rid in roles_env.split(",") + if rid.strip().isdigit() + } + # Set up intents. # Message Content is required for normal text replies. # Server Members is only needed when the allowlist contains usernames @@ -523,7 +596,10 @@ async def connect(self) -> bool: intents.message_content = True intents.dm_messages = True intents.guild_messages = True - intents.members = any(not entry.isdigit() for entry in self._allowed_user_ids) + intents.members = ( + any(not entry.isdigit() for entry in self._allowed_user_ids) + or bool(self._allowed_role_ids) # Need members intent for role lookup + ) intents.voice_states = True # Resolve proxy (DISCORD_PROXY > generic env vars > macOS system proxy) @@ -532,10 +608,15 @@ async def connect(self) -> bool: if proxy_url: logger.info("[%s] Using proxy for Discord: %s", self.name, proxy_url) - # Create bot — proxy= for HTTP, connector= for SOCKS + # Create bot — proxy= for HTTP, connector= for SOCKS. + # allowed_mentions is set with safe defaults (no @everyone/roles) + # so LLM output or echoed user content can't ping the whole + # server; override per DISCORD_ALLOW_MENTION_* env vars or the + # discord.allow_mentions.* block in config.yaml. self._client = commands.Bot( command_prefix="!", # Not really used, we handle raw messages intents=intents, + allowed_mentions=_build_allowed_mentions(), **proxy_kwargs_for_bot(proxy_url), ) adapter_self = self # capture for closure @@ -557,6 +638,15 @@ async def on_ready(): @self._client.event async def on_message(message: DiscordMessage): + # Block until _resolve_allowed_usernames has swapped + # any raw usernames in DISCORD_ALLOWED_USERS for numeric + # IDs (otherwise on_message's author.id lookup can miss). + if not adapter_self._ready_event.is_set(): + try: + await asyncio.wait_for(adapter_self._ready_event.wait(), timeout=30.0) + except asyncio.TimeoutError: + pass + # Dedup: Discord RESUME replays events after reconnects (#4777) if adapter_self._dedup.is_duplicate(str(message.id)): return @@ -570,14 +660,13 @@ async def on_message(message: DiscordMessage): if message.type not in (discord.MessageType.default, discord.MessageType.reply): return - # Check if the message author is in the allowed user list - if not self._is_allowed_user(str(message.author.id)): - return - # Bot message filtering (DISCORD_ALLOW_BOTS): # "none" — ignore all other bots (default) # "mentions" — accept bot messages only when they @mention us # "all" — accept all bot messages + # Must run BEFORE the user allowlist check so that bots + # permitted by DISCORD_ALLOW_BOTS are not rejected for + # not being in DISCORD_ALLOWED_USERS (fixes #4466). if getattr(message.author, "bot", False): allow_bots = os.getenv("DISCORD_ALLOW_BOTS", "none").lower().strip() if allow_bots == "none": @@ -585,7 +674,12 @@ async def on_message(message: DiscordMessage): elif allow_bots == "mentions": if not self._client.user or self._client.user not in message.mentions: return - # "all" falls through to handle_message + # "all" falls through; bot is permitted — skip the + # human-user allowlist below (bots aren't in it). + else: + # Non-bot: enforce the configured user/role allowlists. + if not self._is_allowed_user(str(message.author.id), message.author): + return # Multi-agent filtering: if the message mentions specific bots # but NOT this bot, the sender is talking to another agent — @@ -652,7 +746,8 @@ async def on_voice_state_update(member, before, after): ) # Register slash commands - self._register_slash_commands() + if self._slash_commands: + self._register_slash_commands() # Start the bot in background self._bot_task = asyncio.create_task(self._client.start(self.config.token)) @@ -708,8 +803,27 @@ async def _run_post_connect_initialization(self) -> None: if not self._client: return try: - synced = await asyncio.wait_for(self._client.tree.sync(), timeout=30) - logger.info("[%s] Synced %d slash command(s)", self.name, len(synced)) + sync_policy = self._get_discord_command_sync_policy() + if sync_policy == "off": + logger.info("[%s] Skipping Discord slash command sync (policy=off)", self.name) + return + + if sync_policy == "bulk": + synced = await asyncio.wait_for(self._client.tree.sync(), timeout=30) + logger.info("[%s] Synced %d slash command(s) via bulk tree sync", self.name, len(synced)) + return + + summary = await asyncio.wait_for(self._safe_sync_slash_commands(), timeout=30) + logger.info( + "[%s] Safely reconciled %d slash command(s): unchanged=%d updated=%d recreated=%d created=%d deleted=%d", + self.name, + summary["total"], + summary["unchanged"], + summary["updated"], + summary["recreated"], + summary["created"], + summary["deleted"], + ) except asyncio.TimeoutError: logger.warning("[%s] Slash command sync timed out after 30s", self.name) except asyncio.CancelledError: @@ -717,6 +831,183 @@ async def _run_post_connect_initialization(self) -> None: except Exception as e: # pragma: no cover - defensive logging logger.warning("[%s] Slash command sync failed: %s", self.name, e, exc_info=True) + def _get_discord_command_sync_policy(self) -> str: + raw = str(os.getenv("DISCORD_COMMAND_SYNC_POLICY", "safe") or "").strip().lower() + if raw in _DISCORD_COMMAND_SYNC_POLICIES: + return raw + if raw: + logger.warning( + "[%s] Invalid DISCORD_COMMAND_SYNC_POLICY=%r; falling back to 'safe'", + self.name, + raw, + ) + return "safe" + + def _canonicalize_app_command_payload(self, payload: Dict[str, Any]) -> Dict[str, Any]: + """Reduce command payloads to the semantic fields Hermes manages.""" + contexts = payload.get("contexts") + integration_types = payload.get("integration_types") + return { + "type": int(payload.get("type", 1) or 1), + "name": str(payload.get("name", "") or ""), + "description": str(payload.get("description", "") or ""), + "default_member_permissions": self._normalize_permissions( + payload.get("default_member_permissions") + ), + "dm_permission": bool(payload.get("dm_permission", True)), + "nsfw": bool(payload.get("nsfw", False)), + "contexts": sorted(int(c) for c in contexts) if contexts else None, + "integration_types": ( + sorted(int(i) for i in integration_types) if integration_types else None + ), + "options": [ + self._canonicalize_app_command_option(item) + for item in payload.get("options", []) or [] + if isinstance(item, dict) + ], + } + + @staticmethod + def _normalize_permissions(value: Any) -> Optional[str]: + """Discord emits default_member_permissions as str server-side but discord.py + sets it as int locally. Normalize to str-or-None so the comparison is stable.""" + if value is None: + return None + return str(value) + + def _existing_command_to_payload(self, command: Any) -> Dict[str, Any]: + """Build a canonical-ready dict from an AppCommand. + + discord.py's AppCommand.to_dict() does NOT include nsfw, + dm_permission, or default_member_permissions (they live only on the + attributes). Pull them from the attributes so the canonicalizer sees + the real server-side values instead of defaults — otherwise any + command using non-default permissions would diff on every startup. + """ + payload = dict(command.to_dict()) + nsfw = getattr(command, "nsfw", None) + if nsfw is not None: + payload["nsfw"] = bool(nsfw) + guild_only = getattr(command, "guild_only", None) + if guild_only is not None: + payload["dm_permission"] = not bool(guild_only) + default_permissions = getattr(command, "default_member_permissions", None) + if default_permissions is not None: + payload["default_member_permissions"] = getattr( + default_permissions, "value", default_permissions + ) + return payload + + def _canonicalize_app_command_option(self, payload: Dict[str, Any]) -> Dict[str, Any]: + return { + "type": int(payload.get("type", 0) or 0), + "name": str(payload.get("name", "") or ""), + "description": str(payload.get("description", "") or ""), + "required": bool(payload.get("required", False)), + "autocomplete": bool(payload.get("autocomplete", False)), + "choices": [ + { + "name": str(choice.get("name", "") or ""), + "value": choice.get("value"), + } + for choice in payload.get("choices", []) or [] + if isinstance(choice, dict) + ], + "channel_types": list(payload.get("channel_types", []) or []), + "min_value": payload.get("min_value"), + "max_value": payload.get("max_value"), + "min_length": payload.get("min_length"), + "max_length": payload.get("max_length"), + "options": [ + self._canonicalize_app_command_option(item) + for item in payload.get("options", []) or [] + if isinstance(item, dict) + ], + } + + def _patchable_app_command_payload(self, payload: Dict[str, Any]) -> Dict[str, Any]: + """Fields supported by discord.py's edit_global_command route.""" + canonical = self._canonicalize_app_command_payload(payload) + return { + "name": canonical["name"], + "description": canonical["description"], + "options": canonical["options"], + } + + async def _safe_sync_slash_commands(self) -> Dict[str, int]: + """Diff existing global commands and only mutate the commands that changed.""" + if not self._client: + return { + "total": 0, + "unchanged": 0, + "updated": 0, + "recreated": 0, + "created": 0, + "deleted": 0, + } + + tree = self._client.tree + app_id = getattr(self._client, "application_id", None) or getattr(getattr(self._client, "user", None), "id", None) + if not app_id: + raise RuntimeError("Discord application ID is unavailable for slash command sync") + + desired_payloads = [command.to_dict(tree) for command in tree.get_commands()] + desired_by_key = { + (int(payload.get("type", 1) or 1), str(payload.get("name", "") or "").lower()): payload + for payload in desired_payloads + } + existing_commands = await tree.fetch_commands() + existing_by_key = { + ( + int(getattr(getattr(command, "type", None), "value", getattr(command, "type", 1)) or 1), + str(command.name or "").lower(), + ): command + for command in existing_commands + } + + unchanged = 0 + updated = 0 + recreated = 0 + created = 0 + deleted = 0 + http = self._client.http + + for key, desired in desired_by_key.items(): + current = existing_by_key.pop(key, None) + if current is None: + await http.upsert_global_command(app_id, desired) + created += 1 + continue + + current_existing_payload = self._existing_command_to_payload(current) + current_payload = self._canonicalize_app_command_payload(current_existing_payload) + desired_payload = self._canonicalize_app_command_payload(desired) + if current_payload == desired_payload: + unchanged += 1 + continue + + if self._patchable_app_command_payload(current_existing_payload) == self._patchable_app_command_payload(desired): + await http.delete_global_command(app_id, current.id) + await http.upsert_global_command(app_id, desired) + recreated += 1 + continue + + await http.edit_global_command(app_id, current.id, desired) + updated += 1 + + for current in existing_by_key.values(): + await http.delete_global_command(app_id, current.id) + deleted += 1 + + return { + "total": len(desired_payloads), + "unchanged": unchanged, + "updated": updated, + "recreated": recreated, + "created": created, + "deleted": deleted, + } + async def _add_reaction(self, message: Any, emoji: str) -> bool: """Add an emoji reaction to a Discord message.""" if not message or not hasattr(message, "add_reaction"): @@ -774,6 +1065,9 @@ async def send( When metadata contains a thread_id, the message is sent to that thread instead of the parent channel identified by chat_id. + + Forum channels (type 15) reject direct messages — a thread post is + created automatically. """ if not self._client: return SendResult(success=False, error="Not connected") @@ -799,6 +1093,10 @@ async def send( if not channel: return SendResult(success=False, error=f"Channel {chat_id} not found") + # Forum channels reject channel.send() — create a thread post instead. + if self._is_forum_parent(channel): + return await self._send_to_forum(channel, content) + # Format and split message if needed formatted = self.format_message(content) chunks = self.truncate_message(formatted, self.MAX_MESSAGE_LENGTH) @@ -809,7 +1107,10 @@ async def send( if reply_to and self._reply_to_mode != "off": try: ref_msg = await channel.fetch_message(int(reply_to)) - reference = ref_msg + if hasattr(ref_msg, "to_reference"): + reference = ref_msg.to_reference(fail_if_not_exists=False) + else: + reference = ref_msg except Exception as e: logger.debug("Could not fetch reply-to message: %s", e) @@ -827,14 +1128,20 @@ async def send( err_text = str(e) if ( chunk_reference is not None - and "error code: 50035" in err_text - and "Cannot reply to a system message" in err_text + and ( + ( + "error code: 50035" in err_text + and "Cannot reply to a system message" in err_text + ) + or "error code: 10008" in err_text + ) ): logger.warning( - "[%s] Reply target %s is a Discord system message; retrying send without reply reference", + "[%s] Reply target %s rejected the reply reference; retrying send without reply reference", self.name, reply_to, ) + reference = None msg = await channel.send( content=chunk, reference=None, @@ -853,11 +1160,127 @@ async def send( logger.error("[%s] Failed to send Discord message: %s", self.name, e, exc_info=True) return SendResult(success=False, error=str(e)) + async def _send_to_forum(self, forum_channel: Any, content: str) -> SendResult: + """Create a thread post in a forum channel with the message as starter content. + + Forum channels (type 15) don't support direct messages. Instead we + POST to /channels/{forum_id}/threads with a thread name derived from + the first line of the message. Any follow-up chunk failures are + reported in ``raw_response['warnings']`` so the caller can surface + partial-send issues. + """ + from tools.send_message_tool import _derive_forum_thread_name + + formatted = self.format_message(content) + chunks = self.truncate_message(formatted, self.MAX_MESSAGE_LENGTH) + + thread_name = _derive_forum_thread_name(content) + + starter_content = chunks[0] if chunks else thread_name + + try: + thread = await forum_channel.create_thread( + name=thread_name, + content=starter_content, + ) + except Exception as e: + logger.error("[%s] Failed to create forum thread in %s: %s", self.name, forum_channel.id, e) + return SendResult(success=False, error=f"Forum thread creation failed: {e}") + + thread_channel = thread if hasattr(thread, "send") else getattr(thread, "thread", None) + thread_id = str(getattr(thread_channel, "id", getattr(thread, "id", ""))) + starter_msg = getattr(thread, "message", None) + message_id = str(getattr(starter_msg, "id", thread_id)) if starter_msg else thread_id + + # Send remaining chunks into the newly created thread. Track any + # per-chunk failures so the caller sees partial-send outcomes. + message_ids = [message_id] + warnings: list[str] = [] + for chunk in chunks[1:]: + try: + msg = await thread_channel.send(content=chunk) + message_ids.append(str(msg.id)) + except Exception as e: + warning = f"Failed to send follow-up chunk to forum thread {thread_id}: {e}" + logger.warning("[%s] %s", self.name, warning) + warnings.append(warning) + + raw_response: Dict[str, Any] = {"message_ids": message_ids, "thread_id": thread_id} + if warnings: + raw_response["warnings"] = warnings + + return SendResult( + success=True, + message_id=message_ids[0], + raw_response=raw_response, + ) + + async def _forum_post_file( + self, + forum_channel: Any, + *, + thread_name: Optional[str] = None, + content: str = "", + file: Any = None, + files: Optional[list] = None, + ) -> SendResult: + """Create a forum thread whose starter message carries file attachments. + + Used by the send_voice / send_image_file / send_document paths when + the target channel is a forum (type 15). ``create_thread`` on a + ForumChannel accepts the same file/files/content kwargs as + ``channel.send``, creating the thread and starter message atomically. + """ + from tools.send_message_tool import _derive_forum_thread_name + + if not thread_name: + # Prefer the text content, fall back to the first attached + # filename, fall back to the generic default. + hint = content or "" + if not hint.strip(): + if file is not None: + hint = getattr(file, "filename", "") or "" + elif files: + hint = getattr(files[0], "filename", "") or "" + thread_name = _derive_forum_thread_name(hint) if hint.strip() else "New Post" + + kwargs: Dict[str, Any] = {"name": thread_name} + if content: + kwargs["content"] = content + if file is not None: + kwargs["file"] = file + if files: + kwargs["files"] = files + + try: + thread = await forum_channel.create_thread(**kwargs) + except Exception as e: + logger.error( + "[%s] Failed to create forum thread with file in %s: %s", + self.name, + getattr(forum_channel, "id", "?"), + e, + ) + return SendResult(success=False, error=f"Forum thread creation failed: {e}") + + thread_channel = thread if hasattr(thread, "send") else getattr(thread, "thread", None) + thread_id = str(getattr(thread_channel, "id", getattr(thread, "id", ""))) + starter_msg = getattr(thread, "message", None) + message_id = str(getattr(starter_msg, "id", thread_id)) if starter_msg else thread_id + + return SendResult( + success=True, + message_id=message_id, + raw_response={"thread_id": thread_id}, + ) + async def edit_message( self, chat_id: str, message_id: str, content: str, + *, + finalize: bool = False, ) -> SendResult: """Edit a previously sent Discord message.""" if not self._client: @@ -883,7 +1306,11 @@ async def _send_file_attachment( caption: Optional[str] = None, file_name: Optional[str] = None, ) -> SendResult: - """Send a local file as a Discord attachment.""" + """Send a local file as a Discord attachment. + + Forum channels (type 15) get a new thread whose starter message + carries the file — they reject direct POST /messages. + """ if not self._client: return SendResult(success=False, error="Not connected") @@ -896,6 +1323,12 @@ async def _send_file_attachment( filename = file_name or os.path.basename(file_path) with open(file_path, "rb") as fh: file = discord.File(fh, filename=filename) + if self._is_forum_parent(channel): + return await self._forum_post_file( + channel, + content=(caption or "").strip(), + file=file, + ) msg = await channel.send(content=caption if caption else None, file=file) return SendResult(success=True, message_id=str(msg.id)) @@ -944,6 +1377,18 @@ async def send_voice( with open(audio_path, "rb") as f: file_data = f.read() + # Forum channels (type 15) reject direct POST /messages — the + # native voice flag path also targets /messages so it would fail + # too. Create a thread post with the audio as the starter + # attachment instead. + if self._is_forum_parent(channel): + forum_file = discord.File(io.BytesIO(file_data), filename=filename) + return await self._forum_post_file( + channel, + content=(caption or "").strip(), + file=forum_file, + ) + # Try sending as a native voice message via raw API (flags=8192). try: import base64 @@ -1002,51 +1447,53 @@ async def join_voice_channel(self, channel) -> bool: return False guild_id = channel.guild.id - # Already connected in this guild? - existing = self._voice_clients.get(guild_id) - if existing and existing.is_connected(): - if existing.channel.id == channel.id: + async with self._voice_locks.setdefault(guild_id, asyncio.Lock()): + # Already connected in this guild? + existing = self._voice_clients.get(guild_id) + if existing and existing.is_connected(): + if existing.channel.id == channel.id: + self._reset_voice_timeout(guild_id) + return True + await existing.move_to(channel) self._reset_voice_timeout(guild_id) return True - await existing.move_to(channel) - self._reset_voice_timeout(guild_id) - return True - vc = await channel.connect() - self._voice_clients[guild_id] = vc - self._reset_voice_timeout(guild_id) + vc = await channel.connect() + self._voice_clients[guild_id] = vc + self._reset_voice_timeout(guild_id) - # Start voice receiver (Phase 2: listen to users) - try: - receiver = VoiceReceiver(vc, allowed_user_ids=self._allowed_user_ids) - receiver.start() - self._voice_receivers[guild_id] = receiver - self._voice_listen_tasks[guild_id] = asyncio.ensure_future( - self._voice_listen_loop(guild_id) - ) - except Exception as e: - logger.warning("Voice receiver failed to start: %s", e) + # Start voice receiver (Phase 2: listen to users) + try: + receiver = VoiceReceiver(vc, allowed_user_ids=self._allowed_user_ids) + receiver.start() + self._voice_receivers[guild_id] = receiver + self._voice_listen_tasks[guild_id] = asyncio.ensure_future( + self._voice_listen_loop(guild_id) + ) + except Exception as e: + logger.warning("Voice receiver failed to start: %s", e) - return True + return True async def leave_voice_channel(self, guild_id: int) -> None: """Disconnect from the voice channel in a guild.""" - # Stop voice receiver first - receiver = self._voice_receivers.pop(guild_id, None) - if receiver: - receiver.stop() - listen_task = self._voice_listen_tasks.pop(guild_id, None) - if listen_task: - listen_task.cancel() - - vc = self._voice_clients.pop(guild_id, None) - if vc and vc.is_connected(): - await vc.disconnect() - task = self._voice_timeout_tasks.pop(guild_id, None) - if task: - task.cancel() - self._voice_text_channels.pop(guild_id, None) - self._voice_sources.pop(guild_id, None) + async with self._voice_locks.setdefault(guild_id, asyncio.Lock()): + # Stop voice receiver first + receiver = self._voice_receivers.pop(guild_id, None) + if receiver: + receiver.stop() + listen_task = self._voice_listen_tasks.pop(guild_id, None) + if listen_task: + listen_task.cancel() + + vc = self._voice_clients.pop(guild_id, None) + if vc and vc.is_connected(): + await vc.disconnect() + task = self._voice_timeout_tasks.pop(guild_id, None) + if task: + task.cancel() + self._voice_text_channels.pop(guild_id, None) + self._voice_sources.pop(guild_id, None) # Maximum seconds to wait for voice playback before giving up PLAYBACK_TIMEOUT = 120 @@ -1173,8 +1620,7 @@ def get_voice_channel_info(self, guild_id: int) -> Optional[Dict[str, Any]]: speaking_user_ids: set = set() receiver = self._voice_receivers.get(guild_id) if receiver: - import time as _time - now = _time.monotonic() + now = time.monotonic() with receiver._lock: for ssrc, last_t in receiver._last_packet_time.items(): # Consider "speaking" if audio received within last 2 seconds @@ -1286,11 +1732,48 @@ async def _process_voice_input(self, guild_id: int, user_id: int, pcm_data: byte except OSError: pass - def _is_allowed_user(self, user_id: str) -> bool: - """Check if user is in DISCORD_ALLOWED_USERS.""" - if not self._allowed_user_ids: + def _is_allowed_user(self, user_id: str, author=None) -> bool: + """Check if user is allowed via DISCORD_ALLOWED_USERS or DISCORD_ALLOWED_ROLES. + + Uses OR semantics: if the user matches EITHER allowlist, they're allowed. + If both allowlists are empty, everyone is allowed (backwards compatible). + When author is a Member, checks .roles directly; otherwise falls back + to scanning the bot's mutual guilds for a Member record. + """ + # ``getattr`` fallbacks here guard against test fixtures that build + # an adapter via ``object.__new__(DiscordAdapter)`` and skip __init__ + # (see AGENTS.md pitfall #17 — same pattern as gateway.run). + allowed_users = getattr(self, "_allowed_user_ids", set()) + allowed_roles = getattr(self, "_allowed_role_ids", set()) + has_users = bool(allowed_users) + has_roles = bool(allowed_roles) + if not has_users and not has_roles: + return True + # Check user ID allowlist + if has_users and user_id in allowed_users: return True - return user_id in self._allowed_user_ids + # Check role allowlist + if has_roles: + # Try direct role check from Member object + direct_roles = getattr(author, "roles", None) if author is not None else None + if direct_roles: + if any(getattr(r, "id", None) in allowed_roles for r in direct_roles): + return True + # Fallback: scan mutual guilds for member's roles + if self._client is not None: + try: + uid_int = int(user_id) + except (TypeError, ValueError): + uid_int = None + if uid_int is not None: + for guild in self._client.guilds: + m = guild.get_member(uid_int) + if m is None: + continue + m_roles = getattr(m, "roles", None) or [] + if any(getattr(r, "id", None) in allowed_roles for r in m_roles): + return True + return False async def send_image_file( self, @@ -1359,6 +1842,13 @@ async def send_image( import io file = discord.File(io.BytesIO(image_data), filename=f"image.{ext}") + if self._is_forum_parent(channel): + return await self._forum_post_file( + channel, + content=(caption or "").strip(), + file=file, + ) + msg = await channel.send( content=caption if caption else None, file=file, @@ -1381,6 +1871,75 @@ async def send_image( ) return await super().send_image(chat_id, image_url, caption, reply_to) + async def send_animation( + self, + chat_id: str, + animation_url: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + ) -> SendResult: + """Send an animated GIF natively as a Discord file attachment.""" + if not self._client: + return SendResult(success=False, error="Not connected") + + if not is_safe_url(animation_url): + logger.warning("[%s] Blocked unsafe animation URL during Discord send_animation", self.name) + return await super().send_animation(chat_id, animation_url, caption, reply_to, metadata=metadata) + + try: + import aiohttp + + channel = self._client.get_channel(int(chat_id)) + if not channel: + channel = await self._client.fetch_channel(int(chat_id)) + if not channel: + return SendResult(success=False, error=f"Channel {chat_id} not found") + + # Download the GIF and send as a Discord file attachment + # (Discord renders .gif attachments as auto-playing animations inline) + from gateway.platforms.base import resolve_proxy_url, proxy_kwargs_for_aiohttp + _proxy = resolve_proxy_url(platform_env_var="DISCORD_PROXY") + _sess_kw, _req_kw = proxy_kwargs_for_aiohttp(_proxy) + async with aiohttp.ClientSession(**_sess_kw) as session: + async with session.get(animation_url, timeout=aiohttp.ClientTimeout(total=30), **_req_kw) as resp: + if resp.status != 200: + raise Exception(f"Failed to download animation: HTTP {resp.status}") + + animation_data = await resp.read() + + import io + file = discord.File(io.BytesIO(animation_data), filename="animation.gif") + + if self._is_forum_parent(channel): + return await self._forum_post_file( + channel, + content=(caption or "").strip(), + file=file, + ) + + msg = await channel.send( + content=caption if caption else None, + file=file, + ) + return SendResult(success=True, message_id=str(msg.id)) + + except ImportError: + logger.warning( + "[%s] aiohttp not installed, falling back to URL. Run: pip install aiohttp", + self.name, + exc_info=True, + ) + return await super().send_animation(chat_id, animation_url, caption, reply_to, metadata=metadata) + except Exception as e: # pragma: no cover - defensive logging + logger.error( + "[%s] Failed to send animation attachment, falling back to URL: %s", + self.name, + e, + exc_info=True, + ) + return await super().send_animation(chat_id, animation_url, caption, reply_to, metadata=metadata) + async def send_video( self, chat_id: str, @@ -1585,6 +2144,24 @@ async def _run_simple_slash( the "thinking..." indicator is replaced with that text; otherwise it is deleted so the channel isn't cluttered. """ + # Log the invoker so ghost-command reports can be triaged. Discord + # native slash invocations are always user-initiated (no bot can fire + # them), but mobile autocomplete / keyboard shortcuts / other users + # in the same channel are easy to miss in post-mortems. + try: + _user = interaction.user + _chan_id = getattr(interaction.channel, "id", None) or getattr(interaction, "channel_id", None) + logger.info( + "[Discord] slash '%s' invoked by user=%s id=%s channel=%s guild=%s", + command_text, + getattr(_user, "name", "?"), + getattr(_user, "id", "?"), + _chan_id, + getattr(interaction, "guild_id", None), + ) + except Exception: + pass # logging must never block command dispatch + await interaction.response.defer(ephemeral=True) event = self._build_slash_event(interaction, command_text) await self.handle_message(event) @@ -1646,6 +2223,11 @@ async def slash_sethome(interaction: discord.Interaction): async def slash_stop(interaction: discord.Interaction): await self._run_simple_slash(interaction, "/stop", "Stop requested~") + @tree.command(name="steer", description="Inject a message after the next tool call (no interrupt)") + @discord.app_commands.describe(prompt="Text to inject into the agent's next tool result") + async def slash_steer(interaction: discord.Interaction, prompt: str): + await self._run_simple_slash(interaction, f"/steer {prompt}".strip()) + @tree.command(name="compress", description="Compress conversation context") async def slash_compress(interaction: discord.Interaction): await self._run_simple_slash(interaction, "/compress") @@ -1698,6 +2280,10 @@ async def slash_voice(interaction: discord.Interaction, mode: str = ""): async def slash_update(interaction: discord.Interaction): await self._run_simple_slash(interaction, "/update", "Update initiated~") + @tree.command(name="restart", description="Gracefully restart the Hermes gateway") + async def slash_restart(interaction: discord.Interaction): + await self._run_simple_slash(interaction, "/restart", "Restart requested~") + @tree.command(name="approve", description="Approve a pending dangerous command") @discord.app_commands.describe(scope="Optional: 'all', 'session', 'always', 'all session', 'all always'") async def slash_approve(interaction: discord.Interaction, scope: str = ""): @@ -1738,46 +2324,233 @@ async def slash_background(interaction: discord.Interaction, prompt: str): async def slash_btw(interaction: discord.Interaction, question: str): await self._run_simple_slash(interaction, f"/btw {question}") - # Register installed skills as native slash commands (parity with - # Telegram, which uses telegram_menu_commands() in commands.py). - # Discord allows up to 100 application commands globally. - _DISCORD_CMD_LIMIT = 100 + # ── Auto-register any gateway-available commands not yet on the tree ── + # This ensures new commands added to COMMAND_REGISTRY in + # hermes_cli/commands.py automatically appear as Discord slash + # commands without needing a manual entry here. + def _build_auto_slash_command(_name: str, _description: str, _args_hint: str = ""): + """Build a discord.app_commands.Command that proxies to _run_simple_slash.""" + discord_name = _name.lower()[:32] + desc = (_description or f"Run /{_name}")[:100] + has_args = bool(_args_hint) + + if has_args: + def _make_args_handler(__name: str, __hint: str): + @discord.app_commands.describe(args=f"Arguments: {__hint}"[:100]) + async def _handler(interaction: discord.Interaction, args: str = ""): + await self._run_simple_slash( + interaction, f"/{__name} {args}".strip() + ) + _handler.__name__ = f"auto_slash_{__name.replace('-', '_')}" + return _handler + + handler = _make_args_handler(_name, _args_hint) + else: + def _make_simple_handler(__name: str): + async def _handler(interaction: discord.Interaction): + await self._run_simple_slash(interaction, f"/{__name}") + _handler.__name__ = f"auto_slash_{__name.replace('-', '_')}" + return _handler + + handler = _make_simple_handler(_name) + + return discord.app_commands.Command( + name=discord_name, + description=desc, + callback=handler, + ) + + already_registered: set[str] = set() + try: + from hermes_cli.commands import COMMAND_REGISTRY, _is_gateway_available, _resolve_config_gates + + try: + already_registered = {cmd.name for cmd in tree.get_commands()} + except Exception: + pass + + config_overrides = _resolve_config_gates() + + for cmd_def in COMMAND_REGISTRY: + if not _is_gateway_available(cmd_def, config_overrides): + continue + # Discord command names: lowercase, hyphens OK, max 32 chars. + discord_name = cmd_def.name.lower()[:32] + if discord_name in already_registered: + continue + auto_cmd = _build_auto_slash_command( + cmd_def.name, + cmd_def.description, + cmd_def.args_hint, + ) + try: + tree.add_command(auto_cmd) + already_registered.add(discord_name) + except Exception: + # Silently skip commands that fail registration (e.g. + # name conflict with a subcommand group). + pass + + logger.debug( + "Discord auto-registered %d commands from COMMAND_REGISTRY", + len(already_registered), + ) + except Exception as e: + logger.warning("Discord auto-register from COMMAND_REGISTRY failed: %s", e) + + # ── Plugin-registered slash commands ── + # Plugins register via PluginContext.register_command(); we mirror + # those into Discord's native slash picker so users get the same + # autocomplete UX as for built-in commands. No per-platform plugin + # API needed — plugin commands are platform-agnostic. + try: + from hermes_cli.commands import _iter_plugin_command_entries + + for plugin_name, plugin_desc, plugin_args_hint in _iter_plugin_command_entries(): + discord_name = plugin_name.lower()[:32] + if discord_name in already_registered: + continue + auto_cmd = _build_auto_slash_command( + plugin_name, + plugin_desc, + plugin_args_hint, + ) + try: + tree.add_command(auto_cmd) + already_registered.add(discord_name) + except Exception: + # Silently skip commands that fail registration (e.g. + # name conflict with a subcommand group). + pass + except Exception as e: + logger.warning( + "Discord auto-register from plugin commands failed: %s", e + ) + + # Register skills under a single /skill command group with category + # subcommand groups. This uses 1 top-level slot instead of N, + # supporting up to 25 categories × 25 skills = 625 skills. + self._register_skill_group(tree) + + def _register_skill_group(self, tree) -> None: + """Register a single ``/skill`` command with autocomplete on the name. + + Discord enforces an ~8000-byte per-command payload limit. The older + nested layout (``/skill ``) registered one giant + command whose serialized payload grew linearly with the skill + catalog — with the default ~75 skills the payload was ~14 KB and + ``tree.sync()`` rejected the entire slash-command batch (issues + #11321, #10259, #11385, #10261, #10214). + + Autocomplete options are fetched dynamically by Discord when the + user types — they do NOT count against the per-command registration + budget. So we register ONE flat ``/skill`` command with + ``name: str`` (autocompleted) and ``args: str = ""``. This scales + to thousands of skills with no size math, no splitting, and no + hidden skills. The slash picker also becomes more discoverable — + Discord live-filters by the user's typed prefix against both the + skill name and its description. + """ try: - from hermes_cli.commands import discord_skill_commands + from hermes_cli.commands import discord_skill_commands_by_category - existing_names = {cmd.name for cmd in tree.get_commands()} - remaining_slots = max(0, _DISCORD_CMD_LIMIT - len(existing_names)) + existing_names = set() + try: + existing_names = {cmd.name for cmd in tree.get_commands()} + except Exception: + pass - skill_entries, skipped = discord_skill_commands( - max_slots=remaining_slots, + # Reuse the existing collector for consistent filtering + # (per-platform disabled, hub-excluded, name clamping), then + # flatten — the category grouping was only useful for the + # nested layout. + categories, uncategorized, hidden = discord_skill_commands_by_category( reserved_names=existing_names, ) + entries: list[tuple[str, str, str]] = list(uncategorized) + for cat_skills in categories.values(): + entries.extend(cat_skills) - for discord_name, description, cmd_key in skill_entries: - # Closure factory to capture cmd_key per iteration - def _make_skill_handler(_key: str): - async def _skill_slash(interaction: discord.Interaction, args: str = ""): - await self._run_simple_slash(interaction, f"{_key} {args}".strip()) - return _skill_slash + if not entries: + return - handler = _make_skill_handler(cmd_key) - handler.__name__ = f"skill_{discord_name.replace('-', '_')}" + # Stable alphabetical order so the autocomplete suggestion + # list is predictable across restarts. + entries.sort(key=lambda t: t[0]) + + # name -> (description, cmd_key) — used by both the autocomplete + # callback and the handler for O(1) dispatch. + skill_lookup: dict[str, tuple[str, str]] = { + n: (d, k) for n, d, k in entries + } - cmd = discord.app_commands.Command( - name=discord_name, - description=description, - callback=handler, + async def _autocomplete_name( + interaction: "discord.Interaction", current: str, + ) -> list: + """Filter skills by the user's typed prefix. + + Matches both the skill name and its description so + "/skill pdf" surfaces skills whose description mentions + PDFs even if the name doesn't. Discord caps this list at + 25 entries per query. + """ + q = (current or "").strip().lower() + choices: list = [] + for name, desc, _key in entries: + if not q or q in name.lower() or (desc and q in desc.lower()): + if desc: + label = f"{name} — {desc}" + else: + label = name + # Discord's Choice.name is capped at 100 chars. + if len(label) > 100: + label = label[:97] + "..." + choices.append( + discord.app_commands.Choice(name=label, value=name) + ) + if len(choices) >= 25: + break + return choices + + @discord.app_commands.describe( + name="Which skill to run", + args="Optional arguments for the skill", + ) + @discord.app_commands.autocomplete(name=_autocomplete_name) + async def _skill_handler( + interaction: "discord.Interaction", name: str, args: str = "", + ): + entry = skill_lookup.get(name) + if not entry: + await interaction.response.send_message( + f"Unknown skill: `{name}`. Start typing for " + f"autocomplete suggestions.", + ephemeral=True, + ) + return + _desc, cmd_key = entry + await self._run_simple_slash( + interaction, f"{cmd_key} {args}".strip() ) - discord.app_commands.describe(args="Optional arguments for the skill")(cmd) - tree.add_command(cmd) - if skipped: - logger.warning( - "[%s] Discord slash command limit reached (%d): %d skill(s) not registered", - self.name, _DISCORD_CMD_LIMIT, skipped, + cmd = discord.app_commands.Command( + name="skill", + description="Run a Hermes skill", + callback=_skill_handler, + ) + tree.add_command(cmd) + + logger.info( + "[%s] Registered /skill command with %d skill(s) via autocomplete", + self.name, len(entries), + ) + if hidden: + logger.info( + "[%s] %d skill(s) filtered out of /skill (name clamp / reserved)", + self.name, hidden, ) except Exception as exc: - logger.warning("[%s] Failed to register skill slash commands: %s", self.name, exc) + logger.warning("[%s] Failed to register /skill command: %s", self.name, exc) def _build_slash_event(self, interaction: discord.Interaction, text: str) -> MessageEvent: """Build a MessageEvent from a Discord slash command interaction.""" @@ -1814,11 +2587,14 @@ def _build_slash_event(self, interaction: discord.Interaction, text: str) -> Mes ) msg_type = MessageType.COMMAND if text.startswith("/") else MessageType.TEXT + channel_id = str(interaction.channel_id) + parent_id = str(getattr(getattr(interaction, "channel", None), "parent_id", "") or "") return MessageEvent( text=text, message_type=msg_type, source=source, raw_message=interaction, + channel_prompt=self._resolve_channel_prompt(channel_id, parent_id or None), ) # ------------------------------------------------------------------ @@ -1889,14 +2665,17 @@ async def _dispatch_thread_session( chat_topic=chat_topic, ) - _parent_id = str(getattr(getattr(interaction, "channel", None), "parent_id", "") or "") + _parent_channel = self._thread_parent_channel(getattr(interaction, "channel", None)) + _parent_id = str(getattr(_parent_channel, "id", "") or "") _skills = self._resolve_channel_skills(thread_id, _parent_id or None) + _channel_prompt = self._resolve_channel_prompt(thread_id, _parent_id or None) event = MessageEvent( text=text, message_type=MessageType.TEXT, source=source, raw_message=interaction, auto_skill=_skills, + channel_prompt=_channel_prompt, ) await self.handle_message(event) @@ -1925,6 +2704,31 @@ def _resolve_channel_skills(self, channel_id: str, parent_id: str | None = None) return list(dict.fromkeys(skills)) # dedup, preserve order return None + def _resolve_channel_prompt(self, channel_id: str, parent_id: str | None = None) -> str | None: + """Resolve a Discord per-channel prompt, preferring the exact channel over its parent.""" + from gateway.platforms.base import resolve_channel_prompt + return resolve_channel_prompt(self.config.extra, channel_id, parent_id) + + def _discord_require_mention(self) -> bool: + """Return whether Discord channel messages require a bot mention.""" + configured = self.config.extra.get("require_mention") + if configured is not None: + if isinstance(configured, str): + return configured.lower() not in ("false", "0", "no", "off") + return bool(configured) + return os.getenv("DISCORD_REQUIRE_MENTION", "true").lower() not in ("false", "0", "no", "off") + + def _discord_free_response_channels(self) -> set: + """Return Discord channel IDs where no bot mention is required.""" + raw = self.config.extra.get("free_response_channels") + if raw is None: + raw = os.getenv("DISCORD_FREE_RESPONSE_CHANNELS", "") + if isinstance(raw, list): + return {str(part).strip() for part in raw if str(part).strip()} + if isinstance(raw, str) and raw.strip(): + return {part.strip() for part in raw.split(",") if part.strip()} + return set() + def _thread_parent_channel(self, channel: Any) -> Any: """Return the parent text channel when invoked from a thread.""" return getattr(channel, "parent", None) or channel @@ -2027,8 +2831,15 @@ async def _auto_create_thread(self, message: 'DiscordMessage') -> Optional[Any]: Returns the created thread object, or ``None`` on failure. """ - # Build a short thread name from the message + # Build a short thread name from the message. Strip Discord mention + # syntax (users / roles / channels) so thread titles don't end up + # showing raw <@id>, <@&id>, or <#id> markers — the ID isn't + # meaningful to humans glancing at the thread list (#6336). content = (message.content or "").strip() + # <@123>, <@!123>, <@&123>, <#123> — collapse to empty; normalize spaces. + content = re.sub(r"<@[!&]?\d+>", "", content) + content = re.sub(r"<#\d+>", "", content) + content = re.sub(r"\s+", " ", content).strip() thread_name = content[:80] if content else "Hermes" if len(content) > 80: thread_name = thread_name[:77] + "..." @@ -2036,9 +2847,25 @@ async def _auto_create_thread(self, message: 'DiscordMessage') -> Optional[Any]: try: thread = await message.create_thread(name=thread_name, auto_archive_duration=1440) return thread - except Exception as e: - logger.warning("[%s] Auto-thread creation failed: %s", self.name, e) - return None + except Exception as direct_error: + display_name = getattr(getattr(message, "author", None), "display_name", None) or "unknown user" + reason = f"Auto-threaded from mention by {display_name}" + try: + seed_msg = await message.channel.send(f"\U0001f9f5 Thread created by Hermes: **{thread_name}**") + thread = await seed_msg.create_thread( + name=thread_name, + auto_archive_duration=1440, + reason=reason, + ) + return thread + except Exception as fallback_error: + logger.warning( + "[%s] Auto-thread creation failed. Direct error: %s. Fallback error: %s", + self.name, + direct_error, + fallback_error, + ) + return None async def send_exec_approval( self, chat_id: str, command: str, session_key: str, @@ -2225,6 +3052,124 @@ def _format_thread_chat_name(self, thread: Any) -> str: return f"{parent_name} / {thread_name}" return thread_name + # ------------------------------------------------------------------ + # Attachment download helpers + # + # Discord attachments (images / audio / documents) are fetched via the + # authenticated bot session whenever the Attachment object exposes + # ``read()``. That sidesteps two classes of bug that hit the older + # plain-HTTP path: + # + # 1. ``cdn.discordapp.com`` URLs increasingly require bot auth on + # download — unauthenticated httpx sees 403 Forbidden. + # (issue #8242) + # 2. Some user environments (VPNs, corporate DNS, tunnels) resolve + # ``cdn.discordapp.com`` to private-looking IPs that our + # ``is_safe_url`` guard classifies as SSRF risks. Routing the + # fetch through discord.py's own HTTP client handles DNS + # internally so our guard isn't consulted for the attachment + # path. (issue #6587) + # + # If ``att.read()`` is unavailable (unexpected object shape / test + # stub) or the bot session fetch fails, we fall back to the existing + # SSRF-gated URL downloaders. The fallback keeps defense-in-depth + # against any future Discord payload-schema drift that could slip a + # non-CDN URL into the ``att.url`` field. (issue #11345) + # ------------------------------------------------------------------ + + async def _read_attachment_bytes(self, att) -> Optional[bytes]: + """Read an attachment via discord.py's authenticated bot session. + + Returns the raw bytes on success, or ``None`` if ``att`` doesn't + expose a callable ``read()`` or the read itself fails. Callers + should treat ``None`` as a signal to fall back to the URL-based + downloaders. + """ + reader = getattr(att, "read", None) + if reader is None or not callable(reader): + return None + try: + return await reader() + except Exception as e: + logger.warning( + "[Discord] Authenticated attachment read failed for %s: %s", + getattr(att, "filename", None) or getattr(att, "url", ""), + e, + ) + return None + + async def _cache_discord_image(self, att, ext: str) -> str: + """Cache a Discord image attachment to local disk. + + Primary path: ``att.read()`` + ``cache_image_from_bytes`` + (authenticated, no SSRF gate). + + Fallback: ``cache_image_from_url`` (plain httpx, SSRF-gated). + """ + raw_bytes = await self._read_attachment_bytes(att) + if raw_bytes is not None: + try: + return cache_image_from_bytes(raw_bytes, ext=ext) + except Exception as e: + logger.debug( + "[Discord] cache_image_from_bytes rejected att.read() data; falling back to URL: %s", + e, + ) + return await cache_image_from_url(att.url, ext=ext) + + async def _cache_discord_audio(self, att, ext: str) -> str: + """Cache a Discord audio attachment to local disk. + + Primary path: ``att.read()`` + ``cache_audio_from_bytes`` + (authenticated, no SSRF gate). + + Fallback: ``cache_audio_from_url`` (plain httpx, SSRF-gated). + """ + raw_bytes = await self._read_attachment_bytes(att) + if raw_bytes is not None: + try: + return cache_audio_from_bytes(raw_bytes, ext=ext) + except Exception as e: + logger.debug( + "[Discord] cache_audio_from_bytes failed; falling back to URL: %s", + e, + ) + return await cache_audio_from_url(att.url, ext=ext) + + async def _cache_discord_document(self, att, ext: str) -> bytes: + """Download a Discord document attachment and return the raw bytes. + + Primary path: ``att.read()`` (authenticated, no SSRF gate). + + Fallback: SSRF-gated ``aiohttp`` download. This closes the gap + where the old document path made raw ``aiohttp.ClientSession`` + requests with no safety check (#11345). The caller is responsible + for passing the returned bytes to ``cache_document_from_bytes`` + (and, where applicable, for injecting text content). + """ + raw_bytes = await self._read_attachment_bytes(att) + if raw_bytes is not None: + return raw_bytes + + # Fallback: SSRF-gated URL download. + if not is_safe_url(att.url): + raise ValueError( + f"Blocked unsafe attachment URL (SSRF protection): {att.url}" + ) + import aiohttp + from gateway.platforms.base import resolve_proxy_url, proxy_kwargs_for_aiohttp + _proxy = resolve_proxy_url(platform_env_var="DISCORD_PROXY") + _sess_kw, _req_kw = proxy_kwargs_for_aiohttp(_proxy) + async with aiohttp.ClientSession(**_sess_kw) as session: + async with session.get( + att.url, + timeout=aiohttp.ClientTimeout(total=30), + **_req_kw, + ) as resp: + if resp.status != 200: + raise Exception(f"HTTP {resp.status}") + return await resp.read() + async def _handle_message(self, message: DiscordMessage) -> None: """Handle incoming Discord messages.""" # In server channels (not DMs), require the bot to be @mentioned @@ -2247,6 +3192,17 @@ async def _handle_message(self, message: DiscordMessage) -> None: parent_channel_id = self._get_parent_channel_id(message.channel) is_voice_linked_channel = False + + # Save mention-stripped text before auto-threading since create_thread() + # can clobber message.content, breaking /command detection in channels. + raw_content = message.content.strip() + normalized_content = raw_content + mention_prefix = False + if self._client.user and self._client.user in message.mentions: + mention_prefix = True + normalized_content = normalized_content.replace(f"<@{self._client.user.id}>", "").strip() + normalized_content = normalized_content.replace(f"<@!{self._client.user.id}>", "").strip() + message.content = normalized_content if not isinstance(message.channel, discord.DMChannel): channel_ids = {str(message.channel.id)} if parent_channel_id: @@ -2267,12 +3223,11 @@ async def _handle_message(self, message: DiscordMessage) -> None: logger.debug("[%s] Ignoring message in ignored channel: %s", self.name, channel_ids) return - free_channels_raw = os.getenv("DISCORD_FREE_RESPONSE_CHANNELS", "") - free_channels = {ch.strip() for ch in free_channels_raw.split(",") if ch.strip()} + free_channels = self._discord_free_response_channels() if parent_channel_id: channel_ids.add(parent_channel_id) - require_mention = os.getenv("DISCORD_REQUIRE_MENTION", "true").lower() not in ("false", "0", "no") + require_mention = self._discord_require_mention() # Voice-linked text channels act as free-response while voice is active. # Only the exact bound channel gets the exemption, not sibling threads. voice_linked_ids = {str(ch_id) for ch_id in self._voice_text_channels.values()} @@ -2285,13 +3240,8 @@ async def _handle_message(self, message: DiscordMessage) -> None: in_bot_thread = is_thread and thread_id in self._threads if require_mention and not is_free_channel and not in_bot_thread: - if self._client.user not in message.mentions: + if self._client.user not in message.mentions and not mention_prefix: return - - if self._client.user and self._client.user in message.mentions: - message.content = message.content.replace(f"<@{self._client.user.id}>", "").strip() - message.content = message.content.replace(f"<@!{self._client.user.id}>", "").strip() - # Auto-thread: when enabled, automatically create a thread for every # @mention in a text channel so each conversation is isolated (like Slack). # Messages already inside threads or DMs are unaffected. @@ -2300,9 +3250,10 @@ async def _handle_message(self, message: DiscordMessage) -> None: if not is_thread and not isinstance(message.channel, discord.DMChannel): no_thread_channels_raw = os.getenv("DISCORD_NO_THREAD_CHANNELS", "") no_thread_channels = {ch.strip() for ch in no_thread_channels_raw.split(",") if ch.strip()} - skip_thread = bool(channel_ids & no_thread_channels) + skip_thread = bool(channel_ids & no_thread_channels) or is_free_channel auto_thread = os.getenv("DISCORD_AUTO_THREAD", "true").lower() in ("true", "1", "yes") - if auto_thread and not skip_thread and not is_voice_linked_channel: + is_reply_message = getattr(message, "type", None) == discord.MessageType.reply + if auto_thread and not skip_thread and not is_voice_linked_channel and not is_reply_message: thread = await self._auto_create_thread(message) if thread: is_thread = True @@ -2312,7 +3263,7 @@ async def _handle_message(self, message: DiscordMessage) -> None: # Determine message type msg_type = MessageType.TEXT - if message.content.startswith("/"): + if normalized_content.startswith("/"): msg_type = MessageType.COMMAND elif message.attachments: # Check attachment types @@ -2363,6 +3314,7 @@ async def _handle_message(self, message: DiscordMessage) -> None: user_name=message.author.display_name, thread_id=thread_id, chat_topic=chat_topic, + is_bot=getattr(message.author, "bot", False), ) # Build media URLs -- download image attachments to local cache so the @@ -2378,7 +3330,7 @@ async def _handle_message(self, message: DiscordMessage) -> None: ext = "." + content_type.split("/")[-1].split(";")[0] if ext not in (".jpg", ".jpeg", ".png", ".gif", ".webp"): ext = ".jpg" - cached_path = await cache_image_from_url(att.url, ext=ext) + cached_path = await self._cache_discord_image(att, ext) media_urls.append(cached_path) media_types.append(content_type) print(f"[Discord] Cached user image: {cached_path}", flush=True) @@ -2392,7 +3344,7 @@ async def _handle_message(self, message: DiscordMessage) -> None: ext = "." + content_type.split("/")[-1].split(";")[0] if ext not in (".ogg", ".mp3", ".wav", ".webm", ".m4a"): ext = ".ogg" - cached_path = await cache_audio_from_url(att.url, ext=ext) + cached_path = await self._cache_discord_audio(att, ext) media_urls.append(cached_path) media_types.append(content_type) print(f"[Discord] Cached user audio: {cached_path}", flush=True) @@ -2423,19 +3375,7 @@ async def _handle_message(self, message: DiscordMessage) -> None: ) else: try: - import aiohttp - from gateway.platforms.base import resolve_proxy_url, proxy_kwargs_for_aiohttp - _proxy = resolve_proxy_url(platform_env_var="DISCORD_PROXY") - _sess_kw, _req_kw = proxy_kwargs_for_aiohttp(_proxy) - async with aiohttp.ClientSession(**_sess_kw) as session: - async with session.get( - att.url, - timeout=aiohttp.ClientTimeout(total=30), - **_req_kw, - ) as resp: - if resp.status != 200: - raise Exception(f"HTTP {resp.status}") - raw_bytes = await resp.read() + raw_bytes = await self._cache_discord_document(att, ext) cached_path = cache_document_from_bytes( raw_bytes, att.filename or f"document{ext}" ) @@ -2463,7 +3403,9 @@ async def _handle_message(self, message: DiscordMessage) -> None: att.filename, e, exc_info=True, ) - event_text = message.content + # Use normalized_content (saved before auto-threading) instead of message.content, + # to detect /slash commands in channel messages. + event_text = normalized_content if pending_text_injection: event_text = f"{pending_text_injection}\n\n{event_text}" if event_text else pending_text_injection @@ -2476,6 +3418,15 @@ async def _handle_message(self, message: DiscordMessage) -> None: _parent_id = str(getattr(_chan, "parent_id", "") or "") _chan_id = str(getattr(_chan, "id", "")) _skills = self._resolve_channel_skills(_chan_id, _parent_id or None) + _channel_prompt = self._resolve_channel_prompt(_chan_id, _parent_id or None) + + reply_to_id = None + reply_to_text = None + if message.reference: + reply_to_id = str(message.reference.message_id) + if message.reference.resolved: + reply_to_text = getattr(message.reference.resolved, "content", None) or None + event = MessageEvent( text=event_text, message_type=msg_type, @@ -2484,9 +3435,11 @@ async def _handle_message(self, message: DiscordMessage) -> None: message_id=str(message.id), media_urls=media_urls, media_types=media_types, - reply_to_message_id=str(message.reference.message_id) if message.reference else None, + reply_to_message_id=reply_to_id, + reply_to_text=reply_to_text, timestamp=message.created_at, auto_skill=_skills, + channel_prompt=_channel_prompt, ) # Track thread participation so the bot won't require @mention for @@ -2564,7 +3517,20 @@ async def _flush_text_batch(self, key: str) -> None: "[Discord] Flushing text batch %s (%d chars)", key, len(event.text or ""), ) - await self.handle_message(event) + # Shield the downstream dispatch so that a subsequent chunk + # arriving while handle_message is mid-flight cannot cancel + # the running agent turn. _enqueue_text_event always cancels + # the prior flush task when a new chunk lands; without this + # shield, CancelledError would propagate from our task down + # into handle_message → the agent's streaming request, + # aborting the response the user was waiting on. The new + # chunk is handled by the fresh flush task regardless. + await asyncio.shield(self.handle_message(event)) + except asyncio.CancelledError: + # Only reached if cancel landed before the pop — the shielded + # handle_message is unaffected either way. Let the task exit + # cleanly so the finally block cleans up. + pass finally: if self._pending_text_batch_tasks.get(key) is current_task: self._pending_text_batch_tasks.pop(key, None) diff --git a/gateway/platforms/email.py b/gateway/platforms/email.py index d4261ccfb817..2a38d699ec43 100644 --- a/gateway/platforms/email.py +++ b/gateway/platforms/email.py @@ -545,6 +545,7 @@ async def send_document( caption: Optional[str] = None, file_name: Optional[str] = None, reply_to: Optional[str] = None, + **kwargs, ) -> SendResult: """Send a file as an email attachment.""" try: diff --git a/gateway/platforms/feishu.py b/gateway/platforms/feishu.py index 7fce74def38c..718f01e9954d 100644 --- a/gateway/platforms/feishu.py +++ b/gateway/platforms/feishu.py @@ -8,11 +8,41 @@ - Gateway allowlist integration via FEISHU_ALLOWED_USERS - Persistent dedup state across restarts - Per-chat serial message processing (matches openclaw createChatQueue) -- Persistent ACK emoji reaction on inbound messages +- Processing status reactions: Typing while working, removed on success, + swapped for CrossMark on failure - Reaction events routed as synthetic text events (matches openclaw) - Interactive card button-click events routed as synthetic COMMAND events - Webhook anomaly tracking (matches openclaw createWebhookAnomalyTracker) - Verification token validation as second auth layer (matches openclaw) + +Feishu identity model +--------------------- +Feishu uses three user-ID tiers (official docs: +https://open.feishu.cn/document/home/user-identity-introduction/introduction): + + open_id (ou_xxx) — **App-scoped**. The same person gets a different + open_id under each Feishu app. Always available in + event payloads without extra permissions. + user_id (u_xxx) — **Tenant-scoped**. Stable within a company but + requires the ``contact:user.employee_id:readonly`` + scope. May not be present. + union_id (on_xxx) — **Developer-scoped**. Same across all apps owned by + one developer/ISV. Best cross-app stable ID. + +For bots specifically: + + app_id — The application's canonical credential identifier. + bot open_id — Returned by ``/bot/v3/info``. This is the bot's own + open_id *within its app context* and is what Feishu + puts in ``mentions[].id.open_id`` when someone + @-mentions the bot. Used for mention gating only. + +In single-bot mode (what Hermes currently supports), open_id works as a +de-facto unique user identifier since there is only one app context. + +Session-key participant isolation prefers ``union_id`` (via user_id_alt) +over ``open_id`` (via user_id) so that sessions stay stable if the same +user is seen through different apps in the future. """ from __future__ import annotations @@ -29,11 +59,12 @@ import threading import time import uuid +from collections import OrderedDict from dataclasses import dataclass, field from datetime import datetime from pathlib import Path from types import SimpleNamespace -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List, Optional, Sequence from urllib.error import HTTPError, URLError from urllib.parse import urlencode from urllib.request import Request, urlopen @@ -71,8 +102,13 @@ UpdateMessageRequest, UpdateMessageRequestBody, ) + from lark_oapi.core import AccessTokenType, HttpMethod from lark_oapi.core.const import FEISHU_DOMAIN, LARK_DOMAIN - from lark_oapi.event.callback.model.p2_card_action_trigger import P2CardActionTriggerResponse + from lark_oapi.core.model import BaseRequest + from lark_oapi.event.callback.model.p2_card_action_trigger import ( + CallBackCard, + P2CardActionTriggerResponse, + ) from lark_oapi.event.dispatcher_handler import EventDispatcherHandler from lark_oapi.ws import Client as FeishuWSClient @@ -80,6 +116,7 @@ except ImportError: FEISHU_AVAILABLE = False lark = None # type: ignore[assignment] + CallBackCard = None # type: ignore[assignment] P2CardActionTriggerResponse = None # type: ignore[assignment] EventDispatcherHandler = None # type: ignore[assignment] FeishuWSClient = None # type: ignore[assignment] @@ -94,6 +131,7 @@ BasePlatformAdapter, MessageEvent, MessageType, + ProcessingOutcome, SendResult, SUPPORTED_DOCUMENT_TYPES, cache_document_from_bytes, @@ -115,6 +153,8 @@ re.MULTILINE, ) _MARKDOWN_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)") +_MARKDOWN_FENCE_OPEN_RE = re.compile(r"^```([^\n`]*)\s*$") +_MARKDOWN_FENCE_CLOSE_RE = re.compile(r"^```\s*$") _MENTION_RE = re.compile(r"@_user_\d+") _MULTISPACE_RE = re.compile(r"[ \t]{2,}") _POST_CONTENT_INVALID_RE = re.compile(r"content format of the post type is incorrect", re.IGNORECASE) @@ -169,9 +209,32 @@ _FEISHU_WEBHOOK_ANOMALY_THRESHOLD = 25 # consecutive error responses before WARNING log _FEISHU_WEBHOOK_ANOMALY_TTL_SECONDS = 6 * 60 * 60 # anomaly tracker TTL (6 hours) — matches openclaw _FEISHU_CARD_ACTION_DEDUP_TTL_SECONDS = 15 * 60 # card action token dedup window (15 min) + +_APPROVAL_CHOICE_MAP: Dict[str, str] = { + "approve_once": "once", + "approve_session": "session", + "approve_always": "always", + "deny": "deny", +} +_APPROVAL_LABEL_MAP: Dict[str, str] = { + "once": "Approved once", + "session": "Approved for session", + "always": "Approved permanently", + "deny": "Denied", +} _FEISHU_BOT_MSG_TRACK_SIZE = 512 # LRU size for tracking sent message IDs _FEISHU_REPLY_FALLBACK_CODES = frozenset({230011, 231003}) # reply target withdrawn/missing → create fallback -_FEISHU_ACK_EMOJI = "OK" + +# Feishu reactions render as prominent badges, unlike Discord/Telegram's +# small footer emoji — a success badge on every message would add noise, so +# we only mark start (Typing) and failure (CrossMark); the reply itself is +# the success signal. +_FEISHU_REACTION_IN_PROGRESS = "Typing" +_FEISHU_REACTION_FAILURE = "CrossMark" +# Bound on the (message_id → reaction_id) handle cache. Happy-path entries +# drain on completion; the cap is a safeguard against unbounded growth from +# delete-failures, not a capacity plan. +_FEISHU_PROCESSING_REACTION_CACHE_SIZE = 1024 # QR onboarding constants _ONBOARD_ACCOUNTS_URLS = { @@ -202,6 +265,8 @@ _PREFERRED_LOCALES = ("zh_cn", "en_us") _MARKDOWN_SPECIAL_CHARS_RE = re.compile(r"([\\`*_{}\[\]()#+\-!|>~])") _MENTION_PLACEHOLDER_RE = re.compile(r"@_user_\d+") +_MENTION_BOUNDARY_CHARS = frozenset(" \t\n\r.,;:!?、，。；：！？()[]{}<>\"'`") +_TRAILING_TERMINAL_PUNCT = frozenset(" \t\n\r.!?。！？") _WHITESPACE_RE = re.compile(r"\s+") _SUPPORTED_CARD_TEXT_KEYS = ( "title", @@ -245,12 +310,36 @@ class FeishuPostMediaRef: resource_type: str = "file" +@dataclass(frozen=True) +class FeishuMentionRef: + name: str = "" + open_id: str = "" + is_all: bool = False + is_self: bool = False + + +@dataclass(frozen=True) +class _FeishuBotIdentity: + open_id: str = "" + user_id: str = "" + name: str = "" + + def matches(self, *, open_id: str, user_id: str, name: str) -> bool: + # Precedence: open_id > user_id > name. IDs are authoritative when both + # sides have them; the next tier is only considered when either side + # lacks the current one. + if open_id and self.open_id: + return open_id == self.open_id + if user_id and self.user_id: + return user_id == self.user_id + return bool(self.name) and name == self.name + + @dataclass(frozen=True) class FeishuPostParseResult: text_content: str image_keys: List[str] = field(default_factory=list) media_refs: List[FeishuPostMediaRef] = field(default_factory=list) - mentioned_ids: List[str] = field(default_factory=list) @dataclass(frozen=True) @@ -260,14 +349,14 @@ class FeishuNormalizedMessage: preferred_message_type: str = "text" image_keys: List[str] = field(default_factory=list) media_refs: List[FeishuPostMediaRef] = field(default_factory=list) - mentioned_ids: List[str] = field(default_factory=list) + mentions: List[FeishuMentionRef] = field(default_factory=list) relation_kind: str = "plain" metadata: Dict[str, Any] = field(default_factory=dict) @dataclass(frozen=True) class FeishuAdapterSettings: - app_id: str + app_id: str # Canonical bot/app identifier (credential, not from event payloads) app_secret: str domain_name: str connection_mode: str @@ -275,7 +364,11 @@ class FeishuAdapterSettings: verification_token: str group_policy: str allowed_group_users: frozenset[str] + # Bot's own open_id (app-scoped) — returned by /bot/v3/info. Used only for + # @mention matching: Feishu puts this value in mentions[].id.open_id when + # a user @-mentions the bot in a group chat. bot_open_id: str + # Bot's user_id (tenant-scoped) — optional, used as fallback mention match. bot_user_id: str bot_name: str dedup_cache_size: int @@ -413,39 +506,77 @@ def _coerce_required_int(value: Any, default: int, min_value: int = 0) -> int: def _build_markdown_post_payload(content: str) -> str: + rows = _build_markdown_post_rows(content) return json.dumps( { "zh_cn": { - "content": [ - [ - { - "tag": "md", - "text": content, - } - ] - ], + "content": rows, } }, ensure_ascii=False, ) -def parse_feishu_post_content(raw_content: str) -> FeishuPostParseResult: - try: - parsed = json.loads(raw_content) if raw_content else {} - except json.JSONDecodeError: - return FeishuPostParseResult(text_content=FALLBACK_POST_TEXT) - return parse_feishu_post_payload(parsed) +def _build_markdown_post_rows(content: str) -> List[List[Dict[str, str]]]: + """Build Feishu post rows while isolating fenced code blocks. + + Feishu's `md` renderer can swallow trailing content when a fenced code block + appears inside one large markdown element. Split the reply at real fence + lines so prose before/after the code block remains visible while code stays + in a dedicated row. + """ + if not content: + return [[{"tag": "md", "text": ""}]] + if "```" not in content: + return [[{"tag": "md", "text": content}]] + + rows: List[List[Dict[str, str]]] = [] + current: List[str] = [] + in_code_block = False + + def _flush_current() -> None: + nonlocal current + if not current: + return + segment = "\n".join(current) + if segment.strip(): + rows.append([{"tag": "md", "text": segment}]) + current = [] + + for raw_line in content.splitlines(): + stripped_line = raw_line.strip() + is_fence = bool( + _MARKDOWN_FENCE_CLOSE_RE.match(stripped_line) + if in_code_block + else _MARKDOWN_FENCE_OPEN_RE.match(stripped_line) + ) + + if is_fence: + if not in_code_block: + _flush_current() + current.append(raw_line) + in_code_block = not in_code_block + if not in_code_block: + _flush_current() + continue + current.append(raw_line) -def parse_feishu_post_payload(payload: Any) -> FeishuPostParseResult: + _flush_current() + return rows or [[{"tag": "md", "text": content}]] + + +def parse_feishu_post_payload( + payload: Any, + *, + mentions_map: Optional[Dict[str, FeishuMentionRef]] = None, +) -> FeishuPostParseResult: resolved = _resolve_post_payload(payload) if not resolved: return FeishuPostParseResult(text_content=FALLBACK_POST_TEXT) image_keys: List[str] = [] media_refs: List[FeishuPostMediaRef] = [] - mentioned_ids: List[str] = [] parts: List[str] = [] title = _normalize_feishu_text(str(resolved.get("title", "")).strip()) @@ -456,7 +587,10 @@ def parse_feishu_post_payload(payload: Any) -> FeishuPostParseResult: if not isinstance(row, list): continue row_text = _normalize_feishu_text( - "".join(_render_post_element(item, image_keys, media_refs, mentioned_ids) for item in row) + "".join( + _render_post_element(item, image_keys, media_refs, mentions_map) + for item in row + ) ) if row_text: parts.append(row_text) @@ -465,7 +599,6 @@ def parse_feishu_post_payload(payload: Any) -> FeishuPostParseResult: text_content="\n".join(parts).strip() or FALLBACK_POST_TEXT, image_keys=image_keys, media_refs=media_refs, - mentioned_ids=mentioned_ids, ) @@ -517,7 +650,7 @@ def _render_post_element( element: Any, image_keys: List[str], media_refs: List[FeishuPostMediaRef], - mentioned_ids: List[str], + mentions_map: Optional[Dict[str, FeishuMentionRef]] = None, ) -> str: if isinstance(element, str): return element @@ -535,19 +668,21 @@ def _render_post_element( escaped_label = _escape_markdown_text(label) return f"[{escaped_label}]({href})" if href else escaped_label if tag == "at": - mentioned_id = ( - str(element.get("open_id", "")).strip() - or str(element.get("user_id", "")).strip() - ) - if mentioned_id and mentioned_id not in mentioned_ids: - mentioned_ids.append(mentioned_id) - display_name = ( - str(element.get("user_name", "")).strip() - or str(element.get("name", "")).strip() - or str(element.get("text", "")).strip() - or mentioned_id - ) - return f"@{_escape_markdown_text(display_name)}" if display_name else "@" + # Post .user_id is a placeholder ("@_user_N" or "@_all"); look up + # the real ref in mentions_map for the display name. + placeholder = str(element.get("user_id", "")).strip() + if placeholder == "@_all": + # Feishu SDK sometimes omits @_all from the top-level mentions + # payload; record it here so the caller's mention list stays complete. + if mentions_map is not None and "@_all" not in mentions_map: + mentions_map["@_all"] = FeishuMentionRef(is_all=True) + return "@all" + ref = (mentions_map or {}).get(placeholder) + if ref is not None: + display_name = ref.name or ref.open_id or "user" + else: + display_name = str(element.get("user_name", "")).strip() or "user" + return f"@{_escape_markdown_text(display_name)}" if tag in {"img", "image"}: image_key = str(element.get("image_key", "")).strip() if image_key and image_key not in image_keys: @@ -585,8 +720,7 @@ def _render_post_element( nested_parts: List[str] = [] for key in ("text", "title", "content", "children", "elements"): - value = element.get(key) - extracted = _render_nested_post(value, image_keys, media_refs, mentioned_ids) + extracted = _render_nested_post(element.get(key), image_keys, media_refs, mentions_map) if extracted: nested_parts.append(extracted) return " ".join(part for part in nested_parts if part) @@ -596,7 +730,7 @@ def _render_nested_post( value: Any, image_keys: List[str], media_refs: List[FeishuPostMediaRef], - mentioned_ids: List[str], + mentions_map: Optional[Dict[str, FeishuMentionRef]] = None, ) -> str: if isinstance(value, str): return _escape_markdown_text(value) @@ -604,17 +738,17 @@ def _render_nested_post( return " ".join( part for item in value - for part in [_render_nested_post(item, image_keys, media_refs, mentioned_ids)] + for part in [_render_nested_post(item, image_keys, media_refs, mentions_map)] if part ) if isinstance(value, dict): - direct = _render_post_element(value, image_keys, media_refs, mentioned_ids) + direct = _render_post_element(value, image_keys, media_refs, mentions_map) if direct: return direct return " ".join( part for item in value.values() - for part in [_render_nested_post(item, image_keys, media_refs, mentioned_ids)] + for part in [_render_nested_post(item, image_keys, media_refs, mentions_map)] if part ) return "" @@ -625,31 +759,48 @@ def _render_nested_post( # --------------------------------------------------------------------------- -def normalize_feishu_message(*, message_type: str, raw_content: str) -> FeishuNormalizedMessage: +def normalize_feishu_message( + *, + message_type: str, + raw_content: str, + mentions: Optional[Sequence[Any]] = None, + bot: _FeishuBotIdentity = _FeishuBotIdentity(), +) -> FeishuNormalizedMessage: normalized_type = str(message_type or "").strip().lower() payload = _load_feishu_payload(raw_content) + mentions_map = _build_mentions_map(mentions, bot) if normalized_type == "text": + text = str(payload.get("text", "") or "") + # Feishu SDK sometimes omits @_all from the mentions payload even when + # the text literal contains it (confirmed via im.v1.message.get). + if "@_all" in text and "@_all" not in mentions_map: + mentions_map["@_all"] = FeishuMentionRef(is_all=True) return FeishuNormalizedMessage( raw_type=normalized_type, - text_content=_normalize_feishu_text(str(payload.get("text", "") or "")), + text_content=_normalize_feishu_text(text, mentions_map), + mentions=list(mentions_map.values()), ) if normalized_type == "post": - parsed_post = parse_feishu_post_payload(payload) + # The walker writes back to mentions_map if it encounters + # , so reading .values() after parsing is enough. + parsed_post = parse_feishu_post_payload(payload, mentions_map=mentions_map) return FeishuNormalizedMessage( raw_type=normalized_type, text_content=parsed_post.text_content, image_keys=list(parsed_post.image_keys), media_refs=list(parsed_post.media_refs), - mentioned_ids=list(parsed_post.mentioned_ids), + mentions=list(mentions_map.values()), relation_kind="post", ) + mention_refs = list(mentions_map.values()) if normalized_type == "image": image_key = str(payload.get("image_key", "") or "").strip() alt_text = _normalize_feishu_text( str(payload.get("text", "") or "") or str(payload.get("alt", "") or "") - or FALLBACK_IMAGE_TEXT + or FALLBACK_IMAGE_TEXT, + mentions_map, ) return FeishuNormalizedMessage( raw_type=normalized_type, @@ -657,6 +808,7 @@ def normalize_feishu_message(*, message_type: str, raw_content: str) -> FeishuNo preferred_message_type="photo", image_keys=[image_key] if image_key else [], relation_kind="image", + mentions=mention_refs, ) if normalized_type in {"file", "audio", "media"}: media_ref = _build_media_ref_from_payload(payload, resource_type=normalized_type) @@ -668,6 +820,7 @@ def normalize_feishu_message(*, message_type: str, raw_content: str) -> FeishuNo media_refs=[media_ref] if media_ref.file_key else [], relation_kind=normalized_type, metadata={"placeholder_text": placeholder}, + mentions=mention_refs, ) if normalized_type == "merge_forward": return _normalize_merge_forward_message(payload) @@ -942,8 +1095,20 @@ def _first_non_empty_text(*values: Any) -> str: # --------------------------------------------------------------------------- -def _normalize_feishu_text(text: str) -> str: - cleaned = _MENTION_PLACEHOLDER_RE.sub(" ", text or "") +def _normalize_feishu_text( + text: str, + mentions_map: Optional[Dict[str, FeishuMentionRef]] = None, +) -> str: + def _sub(match: "re.Match[str]") -> str: + key = match.group(0) + ref = (mentions_map or {}).get(key) + if ref is None: + return " " + name = ref.name or ref.open_id or "user" + return f"@{name}" + + cleaned = _MENTION_PLACEHOLDER_RE.sub(_sub, text or "") + cleaned = cleaned.replace("@_all", "@all") cleaned = cleaned.replace("\r\n", "\n").replace("\r", "\n") cleaned = "\n".join(_WHITESPACE_RE.sub(" ", line).strip() for line in cleaned.split("\n")) cleaned = "\n".join(line for line in cleaned.split("\n") if line) @@ -962,6 +1127,117 @@ def _unique_lines(lines: List[str]) -> List[str]: return unique +# --------------------------------------------------------------------------- +# Mention helpers +# --------------------------------------------------------------------------- + + +def _extract_mention_ids(mention: Any) -> tuple[str, str]: + # Returns (open_id, user_id). im.v1.message.get hands back id as a string + # plus id_type discriminator; event payloads hand back a nested UserId + # object carrying both fields. + mention_id = getattr(mention, "id", None) + if isinstance(mention_id, str): + id_type = str(getattr(mention, "id_type", "") or "").lower() + if id_type == "open_id": + return mention_id, "" + if id_type == "user_id": + return "", mention_id + return "", "" + if mention_id is None: + return "", "" + return ( + str(getattr(mention_id, "open_id", "") or ""), + str(getattr(mention_id, "user_id", "") or ""), + ) + + +def _build_mentions_map( + mentions: Optional[Sequence[Any]], + bot: _FeishuBotIdentity, +) -> Dict[str, FeishuMentionRef]: + result: Dict[str, FeishuMentionRef] = {} + for mention in mentions or []: + key = str(getattr(mention, "key", "") or "") + if not key: + continue + if key == "@_all": + result[key] = FeishuMentionRef(is_all=True) + continue + open_id, user_id = _extract_mention_ids(mention) + name = str(getattr(mention, "name", "") or "").strip() + result[key] = FeishuMentionRef( + name=name, + open_id=open_id, + is_self=bot.matches(open_id=open_id, user_id=user_id, name=name), + ) + return result + + +def _build_mention_hint(mentions: Sequence[FeishuMentionRef]) -> str: + parts: List[str] = [] + seen: set = set() + for ref in mentions: + if ref.is_self: + continue + signature = (ref.is_all, ref.open_id, ref.name) + if signature in seen: + continue + seen.add(signature) + if ref.is_all: + parts.append("@all") + elif ref.open_id: + parts.append(f"{ref.name or 'unknown'} (open_id={ref.open_id})") + else: + parts.append(ref.name or "unknown") + return f"[Mentioned: {', '.join(parts)}]" if parts else "" + + +def _strip_edge_self_mentions( + text: str, + mentions: Sequence[FeishuMentionRef], +) -> str: + # Leading: strip consecutive self-mentions unconditionally. + # Trailing: strip only when followed by whitespace/terminal punct, so + # mid-sentence references ("don't @Bot again") stay intact. + # Leading word-boundary prevents @Al from eating @Alice. + if not text: + return text + self_names = [ + f"@{ref.name or ref.open_id or 'user'}" + for ref in mentions + if ref.is_self + ] + if not self_names: + return text + + remaining = text.lstrip() + while True: + for nm in self_names: + if not remaining.startswith(nm): + continue + after = remaining[len(nm):] + if after and after[0] not in _MENTION_BOUNDARY_CHARS: + continue + remaining = after.lstrip() + break + else: + break + + while True: + i = len(remaining) + while i > 0 and remaining[i - 1] in _TRAILING_TERMINAL_PUNCT: + i -= 1 + body = remaining[:i] + tail = remaining[i:] + for nm in self_names: + if body.endswith(nm): + remaining = body[: -len(nm)].rstrip() + tail + break + else: + return remaining + + def _run_official_feishu_ws_client(ws_client: Any, adapter: Any) -> None: """Run the official Lark WS client in its own thread-local event loop.""" import lark_oapi.ws.client as ws_client_module @@ -1064,6 +1340,13 @@ def __init__(self, config: PlatformConfig): self._webhook_rate_counts: Dict[str, tuple[int, float]] = {} # rate_key → (count, window_start) self._webhook_anomaly_counts: Dict[str, tuple[int, str, float]] = {} # ip → (count, last_status, first_seen) self._card_action_tokens: Dict[str, float] = {} # token → first_seen_time + # Inbound events that arrived before the adapter loop was ready + # (e.g. during startup/restart or network-flap reconnect). A single + # drainer thread replays them as soon as the loop becomes available. + self._pending_inbound_events: List[Any] = [] + self._pending_inbound_lock = threading.Lock() + self._pending_drain_scheduled = False + self._pending_inbound_max_depth = 1000 # cap queue; drop oldest beyond self._chat_locks: Dict[str, asyncio.Lock] = {} # chat_id → lock (per-chat serial processing) self._sent_message_ids_to_chat: Dict[str, str] = {} # message_id → chat_id (for reaction routing) self._sent_message_id_order: List[str] = [] # LRU order for _sent_message_ids_to_chat @@ -1080,6 +1363,9 @@ def __init__(self, config: PlatformConfig): # Exec approval button state (approval_id → {session_key, message_id, chat_id}) self._approval_state: Dict[int, Dict[str, str]] = {} self._approval_counter = itertools.count(1) + # Feishu reaction deletion requires the opaque reaction_id returned + # by create, so we cache it per message_id. + self._pending_processing_reactions: "OrderedDict[str, str]" = OrderedDict() self._load_seen_message_ids() @staticmethod @@ -1210,6 +1496,12 @@ def _build_event_handler(self) -> Any: .register_p2_card_action_trigger(self._on_card_action_trigger) .register_p2_im_chat_member_bot_added_v1(self._on_bot_added_to_chat) .register_p2_im_chat_member_bot_deleted_v1(self._on_bot_removed_from_chat) + .register_p2_im_chat_access_event_bot_p2p_chat_entered_v1(self._on_p2p_chat_entered) + .register_p2_im_message_recalled_v1(self._on_message_recalled) + .register_p2_customized_event( + "drive.notice.comment_add_v1", + self._on_drive_comment_event, + ) .build() ) @@ -1401,11 +1693,14 @@ async def edit_message( chat_id: str, message_id: str, content: str, + *, + finalize: bool = False, ) -> SendResult: """Edit a previously sent Feishu text/post message.""" if not self._client: return SendResult(success=False, error="Not connected") + content = self.format_message(content) try: msg_type, payload = self._build_outbound_payload(content) body = self._build_update_message_body(msg_type=msg_type, content=payload) @@ -1498,14 +1793,12 @@ def _btn(label: str, action_name: str, btn_type: str = "default") -> dict: logger.warning("[Feishu] send_exec_approval failed: %s", exc) return SendResult(success=False, error=str(exc)) - async def _update_approval_card( - self, message_id: str, label: str, user_name: str, choice: str, - ) -> None: - """Replace the approval card with a resolved status card.""" - if not self._client or not message_id: - return + @staticmethod + def _build_resolved_approval_card(*, choice: str, user_name: str) -> Dict[str, Any]: + """Build raw card JSON for a resolved approval action.""" icon = "❌" if choice == "deny" else "✅" - card = { + label = _APPROVAL_LABEL_MAP.get(choice, "Resolved") + return { "config": {"wide_screen_mode": True}, "header": { "title": {"content": f"{icon} {label}", "tag": "plain_text"}, @@ -1518,13 +1811,6 @@ async def _update_approval_card( }, ], } - try: - payload = json.dumps(card, ensure_ascii=False) - body = self._build_update_message_body(msg_type="interactive", content=payload) - request = self._build_update_message_request(message_id=message_id, request_body=body) - await asyncio.to_thread(self._client.im.v1.message.update, request) - except Exception as exc: - logger.warning("[Feishu] Failed to update approval card %s: %s", message_id, exc) async def send_voice( self, @@ -1757,10 +2043,22 @@ def format_message(self, content: str) -> str: # ========================================================================= def _on_message_event(self, data: Any) -> None: - """Normalize Feishu inbound events into MessageEvent.""" + """Normalize Feishu inbound events into MessageEvent. + + Called by the lark_oapi SDK's event dispatcher on a background thread. + If the adapter loop is not currently accepting callbacks (brief window + during startup/restart or network-flap reconnect), the event is queued + for replay instead of dropped. + """ loop = self._loop - if loop is None or bool(getattr(loop, "is_closed", lambda: False)()): - logger.warning("[Feishu] Dropping inbound message before adapter loop is ready") + if not self._loop_accepts_callbacks(loop): + start_drainer = self._enqueue_pending_inbound_event(data) + if start_drainer: + threading.Thread( + target=self._drain_pending_inbound_events, + name="feishu-pending-inbound-drainer", + daemon=True, + ).start() return future = asyncio.run_coroutine_threadsafe( self._handle_message_event_data(data), @@ -1768,6 +2066,124 @@ def _on_message_event(self, data: Any) -> None: ) future.add_done_callback(self._log_background_failure) + def _enqueue_pending_inbound_event(self, data: Any) -> bool: + """Append an event to the pending-inbound queue. + + Returns True if the caller should spawn a drainer thread (no drainer + currently scheduled), False if a drainer is already running and will + pick up the new event on its next pass. + """ + with self._pending_inbound_lock: + if len(self._pending_inbound_events) >= self._pending_inbound_max_depth: + # Queue full — drop the oldest to make room. This happens only + # if the loop stays unavailable for an extended period AND the + # WS keeps firing callbacks. Still better than silent drops. + dropped = self._pending_inbound_events.pop(0) + try: + event = getattr(dropped, "event", None) + message = getattr(event, "message", None) + message_id = str(getattr(message, "message_id", "") or "unknown") + except Exception: + message_id = "unknown" + logger.error( + "[Feishu] Pending-inbound queue full (%d); dropped oldest event %s", + self._pending_inbound_max_depth, + message_id, + ) + self._pending_inbound_events.append(data) + depth = len(self._pending_inbound_events) + should_start = not self._pending_drain_scheduled + if should_start: + self._pending_drain_scheduled = True + logger.warning( + "[Feishu] Queued inbound event for replay (loop not ready, queue depth=%d)", + depth, + ) + return should_start + + def _drain_pending_inbound_events(self) -> None: + """Replay queued inbound events once the adapter loop is ready. + + Runs in a dedicated daemon thread. Polls ``_running`` and + ``_loop_accepts_callbacks`` until events can be dispatched or the + adapter shuts down. A single drainer handles the entire queue; + concurrent ``_on_message_event`` calls just append. + """ + poll_interval = 0.25 + max_wait_seconds = 120.0 # safety cap: drop queue after 2 minutes + waited = 0.0 + try: + while True: + if not getattr(self, "_running", True): + # Adapter shutting down — drop queued events rather than + # holding them against a closed loop. + with self._pending_inbound_lock: + dropped = len(self._pending_inbound_events) + self._pending_inbound_events.clear() + if dropped: + logger.warning( + "[Feishu] Dropped %d queued inbound event(s) during shutdown", + dropped, + ) + return + loop = self._loop + if self._loop_accepts_callbacks(loop): + with self._pending_inbound_lock: + batch = self._pending_inbound_events[:] + self._pending_inbound_events.clear() + if not batch: + # Queue emptied between check and grab; done. + with self._pending_inbound_lock: + if not self._pending_inbound_events: + return + continue + dispatched = 0 + requeue: List[Any] = [] + for event in batch: + try: + fut = asyncio.run_coroutine_threadsafe( + self._handle_message_event_data(event), + loop, + ) + fut.add_done_callback(self._log_background_failure) + dispatched += 1 + except RuntimeError: + # Loop closed between check and submit — requeue + # and poll again. + requeue.append(event) + if requeue: + with self._pending_inbound_lock: + self._pending_inbound_events[:0] = requeue + if dispatched: + logger.info( + "[Feishu] Replayed %d queued inbound event(s)", + dispatched, + ) + if not requeue: + # Successfully drained; check if more arrived while + # we were dispatching and exit if not. + with self._pending_inbound_lock: + if not self._pending_inbound_events: + return + # More events queued or requeue pending — loop again. + continue + if waited >= max_wait_seconds: + with self._pending_inbound_lock: + dropped = len(self._pending_inbound_events) + self._pending_inbound_events.clear() + logger.error( + "[Feishu] Adapter loop unavailable for %.0fs; " + "dropped %d queued inbound event(s)", + max_wait_seconds, + dropped, + ) + return + time.sleep(poll_interval) + waited += poll_interval + finally: + with self._pending_inbound_lock: + self._pending_drain_scheduled = False + async def _handle_message_event_data(self, data: Any) -> None: """Shared inbound message handling for websocket and webhook transports.""" event = getattr(data, "event", None) @@ -1782,8 +2198,8 @@ async def _handle_message_event_data(self, data: Any) -> None: if not message_id or self._is_duplicate(message_id): logger.debug("[Feishu] Dropping duplicate/missing message_id: %s", message_id) return - if getattr(sender, "sender_type", "") == "bot": - logger.debug("[Feishu] Dropping bot-originated event: %s", message_id) + if self._is_self_sent_bot_message(event): + logger.debug("[Feishu] Dropping self-sent bot event: %s", message_id) return chat_type = getattr(message, "chat_type", "p2p") @@ -1820,6 +2236,31 @@ def _on_bot_removed_from_chat(self, data: Any) -> None: logger.info("[Feishu] Bot removed from chat: %s", chat_id) self._chat_info_cache.pop(chat_id, None) + def _on_p2p_chat_entered(self, data: Any) -> None: + logger.debug("[Feishu] User entered P2P chat with bot") + + def _on_message_recalled(self, data: Any) -> None: + logger.debug("[Feishu] Message recalled by user") + + def _on_drive_comment_event(self, data: Any) -> None: + """Handle drive document comment notification (drive.notice.comment_add_v1). + + Delegates to :mod:`gateway.platforms.feishu_comment` for parsing, + logging, and reaction. Scheduling follows the same + ``run_coroutine_threadsafe`` pattern used by ``_on_message_event``. + """ + from gateway.platforms.feishu_comment import handle_drive_comment_event + + loop = self._loop + if not self._loop_accepts_callbacks(loop): + logger.warning("[Feishu] Dropping drive comment event before adapter loop is ready") + return + future = asyncio.run_coroutine_threadsafe( + handle_drive_comment_event(self._client, data, self_open_id=self._bot_open_id), + loop, + ) + future.add_done_callback(self._log_background_failure) + def _on_reaction_event(self, event_type: str, data: Any) -> None: """Route user reactions on bot messages as synthetic text events.""" event = getattr(data, "event", None) @@ -1835,12 +2276,12 @@ def _on_reaction_event(self, event_type: str, data: Any) -> None: operator_type, emoji_type, ) - # Only process reactions from real users. Ignore app/bot-generated reactions - # and Hermes' own ACK emoji to avoid feedback loops. + # Drop bot/app-origin reactions to break the feedback loop from our + # own lifecycle reactions. A human reacting with the same emoji (e.g. + # clicking Typing on a bot message) is still routed through. loop = self._loop if ( operator_type in {"bot", "app"} - or emoji_type == _FEISHU_ACK_EMOJI or not message_id or loop is None or bool(getattr(loop, "is_closed", lambda: False)()) @@ -1853,20 +2294,82 @@ def _on_reaction_event(self, event_type: str, data: Any) -> None: future.add_done_callback(self._log_background_failure) def _on_card_action_trigger(self, data: Any) -> Any: - """Schedule Feishu card actions on the adapter loop and acknowledge immediately.""" + """Handle card-action callback from the Feishu SDK (synchronous). + + For approval actions: parses the event once, returns the resolved card + inline (the only reliable way to sync all clients), and schedules a + lightweight async method to actually unblock the agent. + + For other card actions: delegates to ``_handle_card_action_event``. + """ loop = self._loop - if loop is None or bool(getattr(loop, "is_closed", lambda: False)()): + if not self._loop_accepts_callbacks(loop): logger.warning("[Feishu] Dropping card action before adapter loop is ready") - else: - future = asyncio.run_coroutine_threadsafe( - self._handle_card_action_event(data), - loop, - ) - future.add_done_callback(self._log_background_failure) + return P2CardActionTriggerResponse() if P2CardActionTriggerResponse else None + + event = getattr(data, "event", None) + action = getattr(event, "action", None) + action_value = getattr(action, "value", {}) or {} + hermes_action = action_value.get("hermes_action") if isinstance(action_value, dict) else None + + if hermes_action: + return self._handle_approval_card_action(event=event, action_value=action_value, loop=loop) + + self._submit_on_loop(loop, self._handle_card_action_event(data)) if P2CardActionTriggerResponse is None: return None return P2CardActionTriggerResponse() + @staticmethod + def _loop_accepts_callbacks(loop: Any) -> bool: + """Return True when the adapter loop can accept thread-safe submissions.""" + return loop is not None and not bool(getattr(loop, "is_closed", lambda: False)()) + + def _submit_on_loop(self, loop: Any, coro: Any) -> None: + """Schedule background work on the adapter loop with shared failure logging.""" + future = asyncio.run_coroutine_threadsafe(coro, loop) + future.add_done_callback(self._log_background_failure) + + def _handle_approval_card_action(self, *, event: Any, action_value: Dict[str, Any], loop: Any) -> Any: + """Schedule approval resolution and build the synchronous callback response.""" + approval_id = action_value.get("approval_id") + if approval_id is None: + logger.debug("[Feishu] Card action missing approval_id, ignoring") + return P2CardActionTriggerResponse() if P2CardActionTriggerResponse else None + choice = _APPROVAL_CHOICE_MAP.get(action_value.get("hermes_action"), "deny") + + operator = getattr(event, "operator", None) + open_id = str(getattr(operator, "open_id", "") or "") + user_name = self._get_cached_sender_name(open_id) or open_id + + self._submit_on_loop(loop, self._resolve_approval(approval_id, choice, user_name)) + + if P2CardActionTriggerResponse is None: + return None + response = P2CardActionTriggerResponse() + if CallBackCard is not None: + card = CallBackCard() + card.type = "raw" + card.data = self._build_resolved_approval_card(choice=choice, user_name=user_name) + response.card = card + return response + + async def _resolve_approval(self, approval_id: Any, choice: str, user_name: str) -> None: + """Pop approval state and unblock the waiting agent thread.""" + state = self._approval_state.pop(approval_id, None) + if not state: + logger.debug("[Feishu] Approval %s already resolved or unknown", approval_id) + return + try: + from tools.approval import resolve_gateway_approval + count = resolve_gateway_approval(state["session_key"], choice) + logger.info( + "Feishu button resolved %d approval(s) for session %s (choice=%s, user=%s)", + count, state["session_key"], choice, user_name, + ) + except Exception as exc: + logger.error("Failed to resolve gateway approval from Feishu button: %s", exc) + async def _handle_reaction_event(self, event_type: str, data: Any) -> None: """Fetch the reacted-to message; if it was sent by this bot, emit a synthetic text event.""" if not self._client: @@ -1958,51 +2461,6 @@ async def _handle_card_action_event(self, data: Any) -> None: action_tag = str(getattr(action, "tag", "") or "button") action_value = getattr(action, "value", {}) or {} - # --- Exec approval button intercept --- - hermes_action = action_value.get("hermes_action") if isinstance(action_value, dict) else None - if hermes_action: - approval_id = action_value.get("approval_id") - state = self._approval_state.pop(approval_id, None) - if not state: - logger.debug("[Feishu] Approval %s already resolved or unknown", approval_id) - return - - choice_map = { - "approve_once": "once", - "approve_session": "session", - "approve_always": "always", - "deny": "deny", - } - choice = choice_map.get(hermes_action, "deny") - - label_map = { - "once": "Approved once", - "session": "Approved for session", - "always": "Approved permanently", - "deny": "Denied", - } - label = label_map.get(choice, "Resolved") - - # Resolve sender name for the status card - sender_id = SimpleNamespace(open_id=open_id, user_id=None, union_id=None) - sender_profile = await self._resolve_sender_profile(sender_id) - user_name = sender_profile.get("user_name") or open_id - - # Resolve the approval — unblocks the agent thread - try: - from tools.approval import resolve_gateway_approval - count = resolve_gateway_approval(state["session_key"], choice) - logger.info( - "Feishu button resolved %d approval(s) for session %s (choice=%s, user=%s)", - count, state["session_key"], choice, user_name, - ) - except Exception as exc: - logger.error("Failed to resolve gateway approval from Feishu button: %s", exc) - - # Update the card to show the decision - await self._update_approval_card(state.get("message_id", ""), label, user_name, choice) - return - synthetic_text = f"/card {action_tag}" if action_value: try: @@ -2047,33 +2505,35 @@ def _get_chat_lock(self, chat_id: str) -> asyncio.Lock: async def _handle_message_with_guards(self, event: MessageEvent) -> None: """Dispatch a single event through the agent pipeline with per-chat serialization - and a persistent ACK emoji reaction before processing starts. + before handing the event off to the agent. - - Per-chat lock: ensures messages in the same chat are processed one at a time - (matches openclaw's createChatQueue serial queue behaviour). - - ACK indicator: adds a CHECK reaction to the triggering message before handing - off to the agent and leaves it in place as a receipt marker. + Per-chat lock ensures messages in the same chat are processed one at a + time (matches openclaw's createChatQueue serial queue behaviour). """ chat_id = getattr(event.source, "chat_id", "") or "" if event.source else "" chat_lock = self._get_chat_lock(chat_id) async with chat_lock: - message_id = event.message_id - if message_id: - await self._add_ack_reaction(message_id) await self.handle_message(event) - async def _add_ack_reaction(self, message_id: str) -> Optional[str]: - """Add a persistent ACK emoji reaction to signal the message was received.""" - if not self._client or not message_id: + # ========================================================================= + # Processing status reactions + # ========================================================================= + + def _reactions_enabled(self) -> bool: + return os.getenv("FEISHU_REACTIONS", "true").strip().lower() not in ("false", "0", "no") + + async def _add_reaction(self, message_id: str, emoji_type: str) -> Optional[str]: + """Return the reaction_id on success, else None. The id is needed later for deletion.""" + if not self._client or not message_id or not emoji_type: return None try: - from lark_oapi.api.im.v1 import ( # lazy import — keeps optional dep optional + from lark_oapi.api.im.v1 import ( CreateMessageReactionRequest, CreateMessageReactionRequestBody, ) body = ( CreateMessageReactionRequestBody.builder() - .reaction_type({"emoji_type": _FEISHU_ACK_EMOJI}) + .reaction_type({"emoji_type": emoji_type}) .build() ) request = ( @@ -2086,16 +2546,93 @@ async def _add_ack_reaction(self, message_id: str) -> Optional[str]: if response and getattr(response, "success", lambda: False)(): data = getattr(response, "data", None) return getattr(data, "reaction_id", None) - logger.warning( - "[Feishu] Failed to add ack reaction to %s: code=%s msg=%s", + logger.debug( + "[Feishu] Add reaction %s on %s rejected: code=%s msg=%s", + emoji_type, message_id, getattr(response, "code", None), getattr(response, "msg", None), ) except Exception: - logger.warning("[Feishu] Failed to add ack reaction to %s", message_id, exc_info=True) + logger.warning( + "[Feishu] Add reaction %s on %s raised", + emoji_type, + message_id, + exc_info=True, + ) return None + async def _remove_reaction(self, message_id: str, reaction_id: str) -> bool: + if not self._client or not message_id or not reaction_id: + return False + try: + from lark_oapi.api.im.v1 import DeleteMessageReactionRequest + request = ( + DeleteMessageReactionRequest.builder() + .message_id(message_id) + .reaction_id(reaction_id) + .build() + ) + response = await asyncio.to_thread(self._client.im.v1.message_reaction.delete, request) + if response and getattr(response, "success", lambda: False)(): + return True + logger.debug( + "[Feishu] Remove reaction %s on %s rejected: code=%s msg=%s", + reaction_id, + message_id, + getattr(response, "code", None), + getattr(response, "msg", None), + ) + except Exception: + logger.warning( + "[Feishu] Remove reaction %s on %s raised", + reaction_id, + message_id, + exc_info=True, + ) + return False + + def _remember_processing_reaction(self, message_id: str, reaction_id: str) -> None: + cache = self._pending_processing_reactions + cache[message_id] = reaction_id + cache.move_to_end(message_id) + while len(cache) > _FEISHU_PROCESSING_REACTION_CACHE_SIZE: + cache.popitem(last=False) + + def _pop_processing_reaction(self, message_id: str) -> Optional[str]: + return self._pending_processing_reactions.pop(message_id, None) + + async def on_processing_start(self, event: MessageEvent) -> None: + if not self._reactions_enabled(): + return + message_id = event.message_id + if not message_id or message_id in self._pending_processing_reactions: + return + reaction_id = await self._add_reaction(message_id, _FEISHU_REACTION_IN_PROGRESS) + if reaction_id: + self._remember_processing_reaction(message_id, reaction_id) + + async def on_processing_complete( + self, event: MessageEvent, outcome: ProcessingOutcome + ) -> None: + if not self._reactions_enabled(): + return + message_id = event.message_id + if not message_id: + return + + start_reaction_id = self._pending_processing_reactions.get(message_id) + if start_reaction_id: + if not await self._remove_reaction(message_id, start_reaction_id): + # Don't stack a second badge on top of a Typing we couldn't + # remove — UI would read as both "working" and "done/failed" + # simultaneously. Keep the handle so LRU eventually evicts it. + return + self._pop_processing_reaction(message_id) + + if outcome is ProcessingOutcome.FAILURE: + await self._add_reaction(message_id, _FEISHU_REACTION_FAILURE) + # ========================================================================= # Webhook server and security # ========================================================================= @@ -2143,13 +2680,22 @@ async def _process_inbound_message( chat_type: str, message_id: str, ) -> None: - text, inbound_type, media_urls, media_types = await self._extract_message_content(message) + text, inbound_type, media_urls, media_types, mentions = await self._extract_message_content(message) + + if inbound_type == MessageType.TEXT: + text = _strip_edge_self_mentions(text, mentions) + if text.startswith("/"): + inbound_type = MessageType.COMMAND + + # Guard runs post-strip so a pure "@Bot" message (stripped to "") is dropped. if inbound_type == MessageType.TEXT and not text and not media_urls: - logger.debug("[Feishu] Ignoring unsupported or empty message type: %s", getattr(message, "message_type", "")) + logger.debug("[Feishu] Ignoring empty text message id=%s", message_id) return - if inbound_type == MessageType.TEXT and text.startswith("/"): - inbound_type = MessageType.COMMAND + if inbound_type != MessageType.COMMAND: + hint = _build_mention_hint(mentions) + if hint: + text = f"{hint}\n\n{text}" if text else hint reply_to_message_id = ( getattr(message, "parent_id", None) @@ -2428,6 +2974,8 @@ async def _handle_webhook_request(self, request: Any) -> Any: self._on_reaction_event(event_type, data) elif event_type == "card.action.trigger": self._on_card_action_trigger(data) + elif event_type == "drive.notice.comment_add_v1": + self._on_drive_comment_event(data) else: logger.debug("[Feishu] Ignoring webhook event type: %s", event_type or "unknown") return web.json_response({"code": 0, "msg": "ok"}) @@ -2606,14 +3154,20 @@ async def _flush_text_batch_now(self, key: str) -> None: # Message content extraction and resource download # ========================================================================= - async def _extract_message_content(self, message: Any) -> tuple[str, MessageType, List[str], List[str]]: - """Extract text and cached media from a normalized Feishu message.""" + async def _extract_message_content( + self, message: Any + ) -> tuple[str, MessageType, List[str], List[str], List[FeishuMentionRef]]: raw_content = getattr(message, "content", "") or "" raw_type = getattr(message, "message_type", "") or "" message_id = str(getattr(message, "message_id", "") or "") logger.info("[Feishu] Received raw message type=%s message_id=%s", raw_type, message_id) - normalized = normalize_feishu_message(message_type=raw_type, raw_content=raw_content) + normalized = normalize_feishu_message( + message_type=raw_type, + raw_content=raw_content, + mentions=getattr(message, "mentions", None), + bot=self._bot_identity(), + ) media_urls, media_types = await self._download_feishu_message_resources( message_id=message_id, normalized=normalized, @@ -2630,7 +3184,7 @@ async def _extract_message_content(self, message: Any) -> tuple[str, MessageType if injected: text = injected - return text, inbound_type, media_urls, media_types + return text, inbound_type, media_urls, media_types, list(normalized.mentions) async def _download_feishu_message_resources( self, @@ -2688,12 +3242,6 @@ def _resolve_normalized_message_type( return self._resolve_media_message_type(media_types[0] if media_types else "", default=MessageType.DOCUMENT) return MessageType.TEXT - def _normalize_inbound_text(self, text: str) -> str: - """Strip Feishu mention placeholders from inbound text.""" - text = _MENTION_RE.sub(" ", text or "") - text = _MULTISPACE_RE.sub(" ", text) - return text.strip() - async def _maybe_extract_text_document(self, cached_path: str, media_type: str) -> str: if not cached_path or not media_type.startswith("text/"): return "" @@ -2900,10 +3448,22 @@ def _resolve_source_chat_type(*, chat_info: Dict[str, Any], event_chat_type: str return "group" async def _resolve_sender_profile(self, sender_id: Any) -> Dict[str, Optional[str]]: + """Map Feishu's three-tier user IDs onto Hermes' SessionSource fields. + + Preference order for the primary ``user_id`` field: + 1. user_id (tenant-scoped, most stable — requires permission scope) + 2. open_id (app-scoped, always available — different per bot app) + + ``user_id_alt`` carries the union_id (developer-scoped, stable across + all apps by the same developer). Session-key generation prefers + user_id_alt when present, so participant isolation stays stable even + if the primary ID is the app-scoped open_id. + """ open_id = getattr(sender_id, "open_id", None) or None user_id = getattr(sender_id, "user_id", None) or None union_id = getattr(sender_id, "union_id", None) or None - primary_id = open_id or user_id + # Prefer tenant-scoped user_id; fall back to app-scoped open_id. + primary_id = user_id or open_id display_name = await self._resolve_sender_name_from_api(primary_id or union_id) return { "user_id": primary_id, @@ -2911,6 +3471,19 @@ async def _resolve_sender_profile(self, sender_id: Any) -> Dict[str, Optional[st "user_id_alt": union_id, } + def _get_cached_sender_name(self, sender_id: Optional[str]) -> Optional[str]: + """Return a cached sender name only while its TTL is still valid.""" + if not sender_id: + return None + cached = self._sender_name_cache.get(sender_id) + if cached is None: + return None + name, expire_at = cached + if time.time() < expire_at: + return name + self._sender_name_cache.pop(sender_id, None) + return None + async def _resolve_sender_name_from_api(self, sender_id: Optional[str]) -> Optional[str]: """Fetch the sender's display name from the Feishu contact API with a 10-minute cache. @@ -2923,11 +3496,9 @@ async def _resolve_sender_name_from_api(self, sender_id: Optional[str]) -> Optio if not trimmed: return None now = time.time() - cached = self._sender_name_cache.get(trimmed) - if cached is not None: - name, expire_at = cached - if now < expire_at: - return name + cached_name = self._get_cached_sender_name(trimmed) + if cached_name is not None: + return cached_name try: from lark_oapi.api.contact.v3 import GetUserRequest # lazy import if trimmed.startswith("ou_"): @@ -2974,15 +3545,31 @@ async def _fetch_message_text(self, message_id: str) -> Optional[str]: body = getattr(parent, "body", None) msg_type = getattr(parent, "msg_type", "") or "" raw_content = getattr(body, "content", "") or "" - text = self._extract_text_from_raw_content(msg_type=msg_type, raw_content=raw_content) + parent_mentions = getattr(parent, "mentions", None) if parent else None + text = self._extract_text_from_raw_content( + msg_type=msg_type, + raw_content=raw_content, + mentions=parent_mentions, + ) self._message_text_cache[message_id] = text return text except Exception: logger.warning("[Feishu] Failed to fetch parent message %s", message_id, exc_info=True) return None - def _extract_text_from_raw_content(self, *, msg_type: str, raw_content: str) -> Optional[str]: - normalized = normalize_feishu_message(message_type=msg_type, raw_content=raw_content) + def _extract_text_from_raw_content( + self, + *, + msg_type: str, + raw_content: str, + mentions: Optional[Sequence[Any]] = None, + ) -> Optional[str]: + normalized = normalize_feishu_message( + message_type=msg_type, + raw_content=raw_content, + mentions=mentions, + bot=self._bot_identity(), + ) if normalized.text_content: return normalized.text_content placeholder = normalized.metadata.get("placeholder_text") if isinstance(normalized.metadata, dict) else None @@ -3052,42 +3639,112 @@ def _should_accept_group_message(self, message: Any, sender_id: Any, chat_id: st normalized = normalize_feishu_message( message_type=getattr(message, "message_type", "") or "", raw_content=raw_content, + mentions=getattr(message, "mentions", None), + bot=self._bot_identity(), ) - if normalized.mentioned_ids: - return self._post_mentions_bot(normalized.mentioned_ids) + return self._post_mentions_bot(normalized.mentions) + + def _is_self_sent_bot_message(self, event: Any) -> bool: + """Return True only for Feishu events emitted by this Hermes bot.""" + sender = getattr(event, "sender", None) + sender_type = str(getattr(sender, "sender_type", "") or "").strip().lower() + if sender_type not in {"bot", "app"}: + return False + + sender_id = getattr(sender, "sender_id", None) + sender_open_id = str(getattr(sender_id, "open_id", "") or "").strip() + sender_user_id = str(getattr(sender_id, "user_id", "") or "").strip() + + if self._bot_open_id and sender_open_id == self._bot_open_id: + return True + if self._bot_user_id and sender_user_id == self._bot_user_id: + return True return False def _message_mentions_bot(self, mentions: List[Any]) -> bool: - """Check whether any mention targets the configured or inferred bot identity.""" + # IDs trump names: when both sides have open_id (or both user_id), + # match requires equal IDs. Name fallback only when either side + # lacks an ID. for mention in mentions: mention_id = getattr(mention, "id", None) - mention_open_id = getattr(mention_id, "open_id", None) - mention_user_id = getattr(mention_id, "user_id", None) + mention_open_id = (getattr(mention_id, "open_id", None) or "").strip() + mention_user_id = (getattr(mention_id, "user_id", None) or "").strip() mention_name = (getattr(mention, "name", None) or "").strip() - if self._bot_open_id and mention_open_id == self._bot_open_id: - return True - if self._bot_user_id and mention_user_id == self._bot_user_id: - return True + if mention_open_id and self._bot_open_id: + if mention_open_id == self._bot_open_id: + return True + continue # IDs differ — not the bot; skip name fallback. + if mention_user_id and self._bot_user_id: + if mention_user_id == self._bot_user_id: + return True + continue if self._bot_name and mention_name == self._bot_name: return True return False - def _post_mentions_bot(self, mentioned_ids: List[str]) -> bool: - if not mentioned_ids: - return False - if self._bot_open_id and self._bot_open_id in mentioned_ids: - return True - if self._bot_user_id and self._bot_user_id in mentioned_ids: - return True - return False + def _post_mentions_bot(self, mentions: List[FeishuMentionRef]) -> bool: + return any(m.is_self for m in mentions) + + def _bot_identity(self) -> _FeishuBotIdentity: + return _FeishuBotIdentity( + open_id=self._bot_open_id, + user_id=self._bot_user_id, + name=self._bot_name, + ) async def _hydrate_bot_identity(self) -> None: - """Best-effort discovery of bot identity for precise group mention gating.""" + """Best-effort discovery of bot identity for precise group mention gating + and self-sent bot event filtering. + + Populates ``_bot_open_id`` and ``_bot_name`` from /open-apis/bot/v3/info + (no extra scopes required beyond the tenant access token). Falls back to + the application info endpoint for ``_bot_name`` only when the first probe + doesn't return it. Each field is hydrated independently — a value already + supplied via env vars (FEISHU_BOT_OPEN_ID / FEISHU_BOT_USER_ID / + FEISHU_BOT_NAME) is preserved and skips its probe. + """ if not self._client: return - if any((self._bot_open_id, self._bot_user_id, self._bot_name)): + if self._bot_open_id and self._bot_name: + # Everything the self-send filter and precise mention gate need is + # already in place; nothing to probe. + return + + # Primary probe: /open-apis/bot/v3/info — returns bot_name + open_id, no + # extra scopes required. This is the same endpoint the onboarding wizard + # uses via probe_bot(). + if not self._bot_open_id or not self._bot_name: + try: + req = ( + BaseRequest.builder() + .http_method(HttpMethod.GET) + .uri("/open-apis/bot/v3/info") + .token_types({AccessTokenType.TENANT}) + .build() + ) + resp = await asyncio.to_thread(self._client.request, req) + content = getattr(getattr(resp, "raw", None), "content", None) + if content: + payload = json.loads(content) + parsed = _parse_bot_response(payload) or {} + open_id = (parsed.get("bot_open_id") or "").strip() + bot_name = (parsed.get("bot_name") or "").strip() + if open_id and not self._bot_open_id: + self._bot_open_id = open_id + if bot_name and not self._bot_name: + self._bot_name = bot_name + except Exception: + logger.debug( + "[Feishu] /bot/v3/info probe failed during hydration", + exc_info=True, + ) + + # Fallback probe for _bot_name only: application info endpoint. Needs + # admin:app.info:readonly or application:application:self_manage scope, + # so it's best-effort. + if self._bot_name: return try: request = self._build_get_application_request(app_id=self._app_id, lang="en_us") @@ -3096,17 +3753,17 @@ async def _hydrate_bot_identity(self) -> None: code = getattr(response, "code", None) if code == 99991672: logger.warning( - "[Feishu] Unable to hydrate bot identity from application info. " + "[Feishu] Unable to hydrate bot name from application info. " "Grant admin:app.info:readonly or application:application:self_manage " "so group @mention gating can resolve the bot name precisely." ) return app = getattr(getattr(response, "data", None), "app", None) app_name = (getattr(app, "app_name", None) or "").strip() - if app_name: + if app_name and not self._bot_name: self._bot_name = app_name except Exception: - logger.debug("[Feishu] Failed to hydrate bot identity", exc_info=True) + logger.debug("[Feishu] Failed to hydrate bot name from application info", exc_info=True) # ========================================================================= # Deduplication — seen message ID cache (persistent) @@ -3816,6 +4473,9 @@ def probe_bot(app_id: str, app_secret: str, domain: str) -> Optional[dict]: Uses lark_oapi SDK when available, falls back to raw HTTP otherwise. Returns {"bot_name": ..., "bot_open_id": ...} on success, None on failure. + + Note: ``bot_open_id`` here is the bot's app-scoped open_id — the same ID + that Feishu puts in @mention payloads. It is NOT the app_id. """ if FEISHU_AVAILABLE: return _probe_bot_sdk(app_id, app_secret, domain) @@ -3836,12 +4496,12 @@ def _build_onboard_client(app_id: str, app_secret: str, domain: str) -> Any: def _parse_bot_response(data: dict) -> Optional[dict]: - """Extract bot_name and bot_open_id from a /bot/v3/info response.""" + # /bot/v3/info returns bot.app_name; legacy paths used bot_name — accept both. if data.get("code") != 0: return None bot = data.get("bot") or data.get("data", {}).get("bot") or {} return { - "bot_name": bot.get("bot_name"), + "bot_name": bot.get("app_name") or bot.get("bot_name"), "bot_open_id": bot.get("open_id"), } @@ -3850,13 +4510,18 @@ def _probe_bot_sdk(app_id: str, app_secret: str, domain: str) -> Optional[dict]: """Probe bot info using lark_oapi SDK.""" try: client = _build_onboard_client(app_id, app_secret, domain) - resp = client.request( - method="GET", - url="/open-apis/bot/v3/info", - body=None, - raw_response=True, + req = ( + BaseRequest.builder() + .http_method(HttpMethod.GET) + .uri("/open-apis/bot/v3/info") + .token_types({AccessTokenType.TENANT}) + .build() ) - return _parse_bot_response(json.loads(resp.content)) + resp = client.request(req) + content = getattr(getattr(resp, "raw", None), "content", None) + if content is None: + return None + return _parse_bot_response(json.loads(content)) except Exception as exc: logger.debug("[Feishu onboard] SDK probe failed: %s", exc) return None diff --git a/gateway/platforms/feishu_comment.py b/gateway/platforms/feishu_comment.py new file mode 100644 index 000000000000..46807630ce3b --- /dev/null +++ b/gateway/platforms/feishu_comment.py @@ -0,0 +1,1383 @@ +""" +Feishu/Lark drive document comment handling. + +Processes ``drive.notice.comment_add_v1`` events and interacts with the +Drive v2 comment reaction API. Kept in a separate module so that the +main ``feishu.py`` adapter does not grow further and comment-related +logic can evolve independently. + +Flow: + 1. Parse event -> extract file_token, comment_id, reply_id, etc. + 2. Add OK reaction + 3. Parallel fetch: doc meta + comment details (batch_query) + 4. Branch on is_whole: + Whole -> list whole comments timeline + Local -> list comment thread replies + 5. Build prompt (local or whole) + 6. Create AIAgent with feishu_doc + feishu_drive tools -> agent generates reply + 7. Route reply: + Whole -> add_whole_comment + Local -> reply_to_comment (fallback to add_whole_comment on 1069302) +""" + +from __future__ import annotations + +import asyncio +import json +import logging +from typing import Any, Dict, List, Optional, Tuple + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Lark SDK helpers (lazy-imported) +# --------------------------------------------------------------------------- + + +def _build_request(method: str, uri: str, paths=None, queries=None, body=None): + """Build a lark_oapi BaseRequest.""" + from lark_oapi import AccessTokenType + from lark_oapi.core.enum import HttpMethod + from lark_oapi.core.model.base_request import BaseRequest + + http_method = HttpMethod.GET if method == "GET" else HttpMethod.POST + + builder = ( + BaseRequest.builder() + .http_method(http_method) + .uri(uri) + .token_types({AccessTokenType.TENANT}) + ) + if paths: + builder = builder.paths(paths) + if queries: + builder = builder.queries(queries) + if body is not None: + builder = builder.body(body) + return builder.build() + + +async def _exec_request(client, method, uri, paths=None, queries=None, body=None): + """Execute a lark API request and return (code, msg, data_dict).""" + logger.info("[Feishu-Comment] API >>> %s %s paths=%s queries=%s body=%s", + method, uri, paths, queries, + json.dumps(body, ensure_ascii=False)[:500] if body else None) + request = _build_request(method, uri, paths, queries, body) + response = await asyncio.to_thread(client.request, request) + + code = getattr(response, "code", None) + msg = getattr(response, "msg", "") + + data: dict = {} + raw = getattr(response, "raw", None) + if raw and hasattr(raw, "content"): + try: + body_json = json.loads(raw.content) + data = body_json.get("data", {}) + except (json.JSONDecodeError, AttributeError): + pass + if not data: + resp_data = getattr(response, "data", None) + if isinstance(resp_data, dict): + data = resp_data + elif resp_data and hasattr(resp_data, "__dict__"): + data = vars(resp_data) + + logger.info("[Feishu-Comment] API <<< %s %s code=%s msg=%s data_keys=%s", + method, uri, code, msg, list(data.keys()) if data else "empty") + if code != 0: + # Log raw response for debugging failed API calls + raw = getattr(response, "raw", None) + raw_content = "" + if raw and hasattr(raw, "content"): + raw_content = raw.content[:500] if isinstance(raw.content, (str, bytes)) else str(raw.content)[:500] + logger.warning("[Feishu-Comment] API FAIL raw response: %s", raw_content) + return code, msg, data + + +# --------------------------------------------------------------------------- +# Event parsing +# --------------------------------------------------------------------------- + + +def parse_drive_comment_event(data: Any) -> Optional[Dict[str, Any]]: + """Extract structured fields from a ``drive.notice.comment_add_v1`` payload. + + *data* may be a ``CustomizedEvent`` (WebSocket) whose ``.event`` is a dict, + or a ``SimpleNamespace`` (Webhook) built from the full JSON body. + + Returns a flat dict with the relevant fields, or ``None`` when the + payload is malformed. + """ + logger.debug("[Feishu-Comment] parse_drive_comment_event: data type=%s", type(data).__name__) + event = getattr(data, "event", None) + if event is None: + logger.debug("[Feishu-Comment] parse_drive_comment_event: no .event attribute, returning None") + return None + + evt: dict = event if isinstance(event, dict) else ( + vars(event) if hasattr(event, "__dict__") else {} + ) + logger.debug("[Feishu-Comment] parse_drive_comment_event: evt keys=%s", list(evt.keys())) + + notice_meta = evt.get("notice_meta") or {} + if not isinstance(notice_meta, dict): + notice_meta = vars(notice_meta) if hasattr(notice_meta, "__dict__") else {} + + from_user = notice_meta.get("from_user_id") or {} + if not isinstance(from_user, dict): + from_user = vars(from_user) if hasattr(from_user, "__dict__") else {} + + to_user = notice_meta.get("to_user_id") or {} + if not isinstance(to_user, dict): + to_user = vars(to_user) if hasattr(to_user, "__dict__") else {} + + return { + "event_id": str(evt.get("event_id") or ""), + "comment_id": str(evt.get("comment_id") or ""), + "reply_id": str(evt.get("reply_id") or ""), + "is_mentioned": bool(evt.get("is_mentioned")), + "timestamp": str(evt.get("timestamp") or ""), + "file_token": str(notice_meta.get("file_token") or ""), + "file_type": str(notice_meta.get("file_type") or ""), + "notice_type": str(notice_meta.get("notice_type") or ""), + "from_open_id": str(from_user.get("open_id") or ""), + "to_open_id": str(to_user.get("open_id") or ""), + } + + +# --------------------------------------------------------------------------- +# Comment reaction API +# --------------------------------------------------------------------------- + +_REACTION_URI = "/open-apis/drive/v2/files/:file_token/comments/reaction" + + +async def add_comment_reaction( + client: Any, + *, + file_token: str, + file_type: str, + reply_id: str, + reaction_type: str = "OK", +) -> bool: + """Add an emoji reaction to a document comment reply. + + Uses the Drive v2 ``update_reaction`` endpoint:: + + POST /open-apis/drive/v2/files/{file_token}/comments/reaction?file_type=... + + Returns ``True`` on success, ``False`` on failure (errors are logged). + """ + try: + from lark_oapi import AccessTokenType # noqa: F401 + except ImportError: + logger.error("[Feishu-Comment] lark_oapi not available") + return False + + body = { + "action": "add", + "reply_id": reply_id, + "reaction_type": reaction_type, + } + + code, msg, _ = await _exec_request( + client, "POST", _REACTION_URI, + paths={"file_token": file_token}, + queries=[("file_type", file_type)], + body=body, + ) + + succeeded = code == 0 + if succeeded: + logger.info( + "[Feishu-Comment] Reaction '%s' added: file=%s:%s reply=%s", + reaction_type, file_type, file_token, reply_id, + ) + else: + logger.warning( + "[Feishu-Comment] Reaction API failed: code=%s msg=%s " + "file=%s:%s reply=%s", + code, msg, file_type, file_token, reply_id, + ) + return succeeded + + +async def delete_comment_reaction( + client: Any, + *, + file_token: str, + file_type: str, + reply_id: str, + reaction_type: str = "OK", +) -> bool: + """Remove an emoji reaction from a document comment reply. + + Best-effort — errors are logged but not raised. + """ + body = { + "action": "delete", + "reply_id": reply_id, + "reaction_type": reaction_type, + } + + code, msg, _ = await _exec_request( + client, "POST", _REACTION_URI, + paths={"file_token": file_token}, + queries=[("file_type", file_type)], + body=body, + ) + + succeeded = code == 0 + if succeeded: + logger.info( + "[Feishu-Comment] Reaction '%s' deleted: file=%s:%s reply=%s", + reaction_type, file_type, file_token, reply_id, + ) + else: + logger.warning( + "[Feishu-Comment] Reaction API failed: code=%s msg=%s " + "file=%s:%s reply=%s", + code, msg, file_type, file_token, reply_id, + ) + return succeeded + + +# --------------------------------------------------------------------------- +# API call layer +# --------------------------------------------------------------------------- + +_BATCH_QUERY_META_URI = "/open-apis/drive/v1/metas/batch_query" +_BATCH_QUERY_COMMENT_URI = "/open-apis/drive/v1/files/:file_token/comments/batch_query" +_LIST_COMMENTS_URI = "/open-apis/drive/v1/files/:file_token/comments" +_LIST_REPLIES_URI = "/open-apis/drive/v1/files/:file_token/comments/:comment_id/replies" +_REPLY_COMMENT_URI = "/open-apis/drive/v1/files/:file_token/comments/:comment_id/replies" +_ADD_COMMENT_URI = "/open-apis/drive/v1/files/:file_token/new_comments" + + +async def query_document_meta( + client: Any, file_token: str, file_type: str, +) -> Dict[str, Any]: + """Fetch document title and URL via batch_query meta API. + + Returns ``{"title": "...", "url": "...", "doc_type": "..."}`` or empty dict. + """ + body = { + "request_docs": [{"doc_token": file_token, "doc_type": file_type}], + "with_url": True, + } + logger.debug("[Feishu-Comment] query_document_meta: file_token=%s file_type=%s", file_token, file_type) + code, msg, data = await _exec_request( + client, "POST", _BATCH_QUERY_META_URI, body=body, + ) + if code != 0: + logger.warning("[Feishu-Comment] Meta batch_query failed: code=%s msg=%s", code, msg) + return {} + + metas = data.get("metas", []) + logger.debug("[Feishu-Comment] query_document_meta: raw metas type=%s value=%s", + type(metas).__name__, str(metas)[:300]) + if not metas: + # Try alternate response shape: metas may be a dict keyed by token + if isinstance(data.get("metas"), dict): + meta = data["metas"].get(file_token, {}) + else: + logger.debug("[Feishu-Comment] query_document_meta: no metas found") + return {} + else: + meta = metas[0] if isinstance(metas, list) else {} + + result = { + "title": meta.get("title", ""), + "url": meta.get("url", ""), + "doc_type": meta.get("doc_type", file_type), + } + logger.info("[Feishu-Comment] query_document_meta: title=%s url=%s", + result["title"], result["url"][:80] if result["url"] else "") + return result + + +_COMMENT_RETRY_LIMIT = 6 +_COMMENT_RETRY_DELAY_S = 1.0 + + +async def batch_query_comment( + client: Any, file_token: str, file_type: str, comment_id: str, +) -> Dict[str, Any]: + """Fetch comment details via batch_query comment API. + + Retries up to 6 times on failure (handles eventual consistency). + + Returns the comment dict with fields like ``is_whole``, ``quote``, + ``reply_list``, etc. Empty dict on failure. + """ + logger.debug("[Feishu-Comment] batch_query_comment: file_token=%s comment_id=%s", file_token, comment_id) + + for attempt in range(_COMMENT_RETRY_LIMIT): + code, msg, data = await _exec_request( + client, "POST", _BATCH_QUERY_COMMENT_URI, + paths={"file_token": file_token}, + queries=[ + ("file_type", file_type), + ("user_id_type", "open_id"), + ], + body={"comment_ids": [comment_id]}, + ) + if code == 0: + break + if attempt < _COMMENT_RETRY_LIMIT - 1: + logger.info( + "[Feishu-Comment] batch_query_comment retry %d/%d: code=%s msg=%s", + attempt + 1, _COMMENT_RETRY_LIMIT, code, msg, + ) + await asyncio.sleep(_COMMENT_RETRY_DELAY_S) + else: + logger.warning( + "[Feishu-Comment] batch_query_comment failed after %d attempts: code=%s msg=%s", + _COMMENT_RETRY_LIMIT, code, msg, + ) + return {} + + # Response: {"items": [{"comment_id": "...", ...}]} + items = data.get("items", []) + logger.debug("[Feishu-Comment] batch_query_comment: got %d items", len(items) if isinstance(items, list) else 0) + if items and isinstance(items, list): + item = items[0] + logger.info("[Feishu-Comment] batch_query_comment: is_whole=%s quote=%s reply_count=%s", + item.get("is_whole"), + (item.get("quote", "") or "")[:60], + len(item.get("reply_list", {}).get("replies", [])) if isinstance(item.get("reply_list"), dict) else "?") + return item + logger.warning("[Feishu-Comment] batch_query_comment: empty items, raw data keys=%s", list(data.keys())) + return {} + + +async def list_whole_comments( + client: Any, file_token: str, file_type: str, +) -> List[Dict[str, Any]]: + """List all whole-document comments (paginated, up to 500).""" + logger.debug("[Feishu-Comment] list_whole_comments: file_token=%s", file_token) + all_comments: List[Dict[str, Any]] = [] + page_token = "" + + for _ in range(5): # max 5 pages + queries = [ + ("file_type", file_type), + ("is_whole", "true"), + ("page_size", "100"), + ("user_id_type", "open_id"), + ] + if page_token: + queries.append(("page_token", page_token)) + + code, msg, data = await _exec_request( + client, "GET", _LIST_COMMENTS_URI, + paths={"file_token": file_token}, + queries=queries, + ) + if code != 0: + logger.warning("[Feishu-Comment] List whole comments failed: code=%s msg=%s", code, msg) + break + + items = data.get("items", []) + if isinstance(items, list): + all_comments.extend(items) + logger.debug("[Feishu-Comment] list_whole_comments: page got %d items, total=%d", + len(items), len(all_comments)) + + if not data.get("has_more"): + break + page_token = data.get("page_token", "") + if not page_token: + break + + logger.info("[Feishu-Comment] list_whole_comments: total %d whole comments fetched", len(all_comments)) + return all_comments + + +async def list_comment_replies( + client: Any, file_token: str, file_type: str, comment_id: str, + *, expect_reply_id: str = "", +) -> List[Dict[str, Any]]: + """List all replies in a comment thread (paginated, up to 500). + + If *expect_reply_id* is set and not found in the first fetch, + retries up to 6 times (handles eventual consistency). + """ + logger.debug("[Feishu-Comment] list_comment_replies: file_token=%s comment_id=%s", file_token, comment_id) + + for attempt in range(_COMMENT_RETRY_LIMIT): + all_replies: List[Dict[str, Any]] = [] + page_token = "" + fetch_ok = True + + for _ in range(5): # max 5 pages + queries = [ + ("file_type", file_type), + ("page_size", "100"), + ("user_id_type", "open_id"), + ] + if page_token: + queries.append(("page_token", page_token)) + + code, msg, data = await _exec_request( + client, "GET", _LIST_REPLIES_URI, + paths={"file_token": file_token, "comment_id": comment_id}, + queries=queries, + ) + if code != 0: + logger.warning("[Feishu-Comment] List replies failed: code=%s msg=%s", code, msg) + fetch_ok = False + break + + items = data.get("items", []) + if isinstance(items, list): + all_replies.extend(items) + + if not data.get("has_more"): + break + page_token = data.get("page_token", "") + if not page_token: + break + + # If we don't need a specific reply, or we found it, return + if not expect_reply_id or not fetch_ok: + break + found = any(r.get("reply_id") == expect_reply_id for r in all_replies) + if found: + break + if attempt < _COMMENT_RETRY_LIMIT - 1: + logger.info( + "[Feishu-Comment] list_comment_replies: reply_id=%s not found, retry %d/%d", + expect_reply_id, attempt + 1, _COMMENT_RETRY_LIMIT, + ) + await asyncio.sleep(_COMMENT_RETRY_DELAY_S) + else: + logger.warning( + "[Feishu-Comment] list_comment_replies: reply_id=%s not found after %d attempts", + expect_reply_id, _COMMENT_RETRY_LIMIT, + ) + + logger.info("[Feishu-Comment] list_comment_replies: total %d replies fetched", len(all_replies)) + return all_replies + + +def _sanitize_comment_text(text: str) -> str: + """Escape characters not allowed in Feishu comment text_run content.""" + return text.replace("&", "&").replace("<", "<").replace(">", ">") + + +async def reply_to_comment( + client: Any, file_token: str, file_type: str, comment_id: str, text: str, +) -> Tuple[bool, int]: + """Post a reply to a local comment thread. + + Returns ``(success, code)``. + """ + text = _sanitize_comment_text(text) + logger.info("[Feishu-Comment] reply_to_comment: comment_id=%s text=%s", + comment_id, text[:100]) + body = { + "content": { + "elements": [ + {"type": "text_run", "text_run": {"text": text}}, + ] + } + } + + code, msg, _ = await _exec_request( + client, "POST", _REPLY_COMMENT_URI, + paths={"file_token": file_token, "comment_id": comment_id}, + queries=[("file_type", file_type)], + body=body, + ) + if code != 0: + logger.warning( + "[Feishu-Comment] reply_to_comment FAILED: code=%s msg=%s comment_id=%s", + code, msg, comment_id, + ) + else: + logger.info("[Feishu-Comment] reply_to_comment OK: comment_id=%s", comment_id) + return code == 0, code + + +async def add_whole_comment( + client: Any, file_token: str, file_type: str, text: str, +) -> bool: + """Add a new whole-document comment. + + Returns ``True`` on success. + """ + text = _sanitize_comment_text(text) + logger.info("[Feishu-Comment] add_whole_comment: file_token=%s text=%s", + file_token, text[:100]) + body = { + "file_type": file_type, + "reply_elements": [ + {"type": "text", "text": text}, + ], + } + + code, msg, _ = await _exec_request( + client, "POST", _ADD_COMMENT_URI, + paths={"file_token": file_token}, + body=body, + ) + if code != 0: + logger.warning("[Feishu-Comment] add_whole_comment FAILED: code=%s msg=%s", code, msg) + else: + logger.info("[Feishu-Comment] add_whole_comment OK") + return code == 0 + + +_REPLY_CHUNK_SIZE = 4000 + + +def _chunk_text(text: str, limit: int = _REPLY_CHUNK_SIZE) -> List[str]: + """Split text into chunks for delivery, preferring line breaks.""" + if len(text) <= limit: + return [text] + chunks = [] + while text: + if len(text) <= limit: + chunks.append(text) + break + # Find last newline within limit + cut = text.rfind("\n", 0, limit) + if cut <= 0: + cut = limit + chunks.append(text[:cut]) + text = text[cut:].lstrip("\n") + return chunks + + +async def deliver_comment_reply( + client: Any, + file_token: str, + file_type: str, + comment_id: str, + text: str, + is_whole: bool, +) -> bool: + """Route agent reply to the correct API, chunking long text. + + - Whole comment -> add_whole_comment + - Local comment -> reply_to_comment, fallback to add_whole_comment on 1069302 + """ + chunks = _chunk_text(text) + logger.info("[Feishu-Comment] deliver_comment_reply: is_whole=%s comment_id=%s text_len=%d chunks=%d", + is_whole, comment_id, len(text), len(chunks)) + + all_ok = True + for i, chunk in enumerate(chunks): + if len(chunks) > 1: + logger.info("[Feishu-Comment] deliver_comment_reply: sending chunk %d/%d (%d chars)", + i + 1, len(chunks), len(chunk)) + + if is_whole: + ok = await add_whole_comment(client, file_token, file_type, chunk) + else: + success, code = await reply_to_comment(client, file_token, file_type, comment_id, chunk) + if success: + ok = True + elif code == 1069302: + logger.info("[Feishu-Comment] Reply not allowed (1069302), falling back to add_whole_comment") + ok = await add_whole_comment(client, file_token, file_type, chunk) + is_whole = True # subsequent chunks also use add_comment + else: + ok = False + + if not ok: + all_ok = False + break + + return all_ok + + +# --------------------------------------------------------------------------- +# Comment content extraction helpers +# --------------------------------------------------------------------------- + + +def _extract_reply_text(reply: Dict[str, Any]) -> str: + """Extract plain text from a comment reply's content structure.""" + content = reply.get("content", {}) + if isinstance(content, str): + try: + content = json.loads(content) + except (json.JSONDecodeError, TypeError): + return content + + elements = content.get("elements", []) + parts = [] + for elem in elements: + if elem.get("type") == "text_run": + text_run = elem.get("text_run", {}) + parts.append(text_run.get("text", "")) + elif elem.get("type") == "docs_link": + docs_link = elem.get("docs_link", {}) + parts.append(docs_link.get("url", "")) + elif elem.get("type") == "person": + person = elem.get("person", {}) + parts.append(f"@{person.get('user_id', 'unknown')}") + return "".join(parts) + + +def _get_reply_user_id(reply: Dict[str, Any]) -> str: + """Extract user_id from a reply dict.""" + user_id = reply.get("user_id", "") + if isinstance(user_id, dict): + return user_id.get("open_id", "") or user_id.get("user_id", "") + return str(user_id) + + +def _extract_semantic_text(reply: Dict[str, Any], self_open_id: str = "") -> str: + """Extract semantic text from a reply, stripping self @mentions and extra whitespace.""" + content = reply.get("content", {}) + if isinstance(content, str): + try: + content = json.loads(content) + except (json.JSONDecodeError, TypeError): + return content + + elements = content.get("elements", []) + parts = [] + for elem in elements: + if elem.get("type") == "person": + person = elem.get("person", {}) + uid = person.get("user_id", "") + # Skip self @mention (it's routing, not content) + if self_open_id and uid == self_open_id: + continue + parts.append(f"@{uid}") + elif elem.get("type") == "text_run": + text_run = elem.get("text_run", {}) + parts.append(text_run.get("text", "")) + elif elem.get("type") == "docs_link": + docs_link = elem.get("docs_link", {}) + parts.append(docs_link.get("url", "")) + return " ".join("".join(parts).split()).strip() + + +# --------------------------------------------------------------------------- +# Document link parsing and wiki resolution +# --------------------------------------------------------------------------- + +import re as _re + +# Matches feishu/lark document URLs and extracts doc_type + token +_FEISHU_DOC_URL_RE = _re.compile( + r"(?:feishu\.cn|larkoffice\.com|larksuite\.com|lark\.suite\.com)" + r"/(?Pwiki|doc|docx|sheet|sheets|slides|mindnote|bitable|base|file)" + r"/(?P[A-Za-z0-9_-]{10,40})" +) + +_WIKI_GET_NODE_URI = "/open-apis/wiki/v2/spaces/get_node" + + +def _extract_docs_links(replies: List[Dict[str, Any]]) -> List[Dict[str, str]]: + """Extract unique document links from a list of comment replies. + + Returns list of ``{"url": "...", "doc_type": "...", "token": "..."}`` dicts. + """ + seen_tokens = set() + links = [] + for reply in replies: + content = reply.get("content", {}) + if isinstance(content, str): + try: + content = json.loads(content) + except (json.JSONDecodeError, TypeError): + continue + for elem in content.get("elements", []): + if elem.get("type") not in ("docs_link", "link"): + continue + link_data = elem.get("docs_link") or elem.get("link") or {} + url = link_data.get("url", "") + if not url: + continue + m = _FEISHU_DOC_URL_RE.search(url) + if not m: + continue + doc_type = m.group("doc_type") + token = m.group("token") + if token in seen_tokens: + continue + seen_tokens.add(token) + links.append({"url": url, "doc_type": doc_type, "token": token}) + return links + + +async def _reverse_lookup_wiki_token( + client: Any, obj_type: str, obj_token: str, +) -> Optional[str]: + """Reverse-lookup: given an obj_token, find its wiki node_token. + + Returns the wiki_token if the document belongs to a wiki space, + or None if it doesn't or the API call fails. + """ + code, msg, data = await _exec_request( + client, "GET", _WIKI_GET_NODE_URI, + queries=[("token", obj_token), ("obj_type", obj_type)], + ) + if code == 0: + node = data.get("node", {}) + wiki_token = node.get("node_token", "") + return wiki_token if wiki_token else None + # code != 0: either not a wiki doc or service error — log and return None + logger.warning("[Feishu-Comment] Wiki reverse lookup failed: code=%s msg=%s obj=%s:%s", code, msg, obj_type, obj_token) + return None + + +async def _resolve_wiki_nodes( + client: Any, + links: List[Dict[str, str]], +) -> List[Dict[str, str]]: + """Resolve wiki links to their underlying document type and token. + + Mutates entries in *links* in-place: replaces ``doc_type`` and ``token`` + with the resolved values for wiki links. Non-wiki links are unchanged. + """ + wiki_links = [l for l in links if l["doc_type"] == "wiki"] + if not wiki_links: + return links + + for link in wiki_links: + wiki_token = link["token"] + code, msg, data = await _exec_request( + client, "GET", _WIKI_GET_NODE_URI, + queries=[("token", wiki_token)], + ) + if code == 0: + node = data.get("node", {}) + resolved_type = node.get("obj_type", "") + resolved_token = node.get("obj_token", "") + if resolved_type and resolved_token: + logger.info( + "[Feishu-Comment] Wiki resolved: %s -> %s:%s", + wiki_token, resolved_type, resolved_token, + ) + link["resolved_type"] = resolved_type + link["resolved_token"] = resolved_token + else: + logger.warning("[Feishu-Comment] Wiki resolve returned empty: %s", wiki_token) + else: + logger.warning("[Feishu-Comment] Wiki resolve failed: code=%s msg=%s token=%s", code, msg, wiki_token) + + return links + + +def _format_referenced_docs( + links: List[Dict[str, str]], current_file_token: str = "", +) -> str: + """Format resolved document links for prompt embedding.""" + if not links: + return "" + lines = ["", "Referenced documents in comments:"] + for link in links: + rtype = link.get("resolved_type", link["doc_type"]) + rtoken = link.get("resolved_token", link["token"]) + is_current = rtoken == current_file_token + suffix = " (same as current document)" if is_current else "" + lines.append(f"- {rtype}:{rtoken}{suffix} ({link['url'][:80]})") + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Prompt construction +# --------------------------------------------------------------------------- + +_PROMPT_TEXT_LIMIT = 220 +_LOCAL_TIMELINE_LIMIT = 20 +_WHOLE_TIMELINE_LIMIT = 12 + + +def _truncate(text: str, limit: int = _PROMPT_TEXT_LIMIT) -> str: + """Truncate text for prompt embedding.""" + if len(text) <= limit: + return text + return text[:limit] + "..." + + +def _select_local_timeline( + timeline: List[Tuple[str, str, bool]], + target_index: int, +) -> List[Tuple[str, str, bool]]: + """Select up to _LOCAL_TIMELINE_LIMIT entries centered on target_index. + + Always keeps first, target, and last entries. + """ + if len(timeline) <= _LOCAL_TIMELINE_LIMIT: + return timeline + n = len(timeline) + selected = set() + selected.add(0) # first + selected.add(n - 1) # last + if 0 <= target_index < n: + selected.add(target_index) # current + # Expand outward from target + budget = _LOCAL_TIMELINE_LIMIT - len(selected) + lo, hi = target_index - 1, target_index + 1 + while budget > 0 and (lo >= 0 or hi < n): + if lo >= 0 and lo not in selected: + selected.add(lo) + budget -= 1 + lo -= 1 + if budget > 0 and hi < n and hi not in selected: + selected.add(hi) + budget -= 1 + hi += 1 + return [timeline[i] for i in sorted(selected)] + + +def _select_whole_timeline( + timeline: List[Tuple[str, str, bool]], + current_index: int, + nearest_self_index: int, +) -> List[Tuple[str, str, bool]]: + """Select up to _WHOLE_TIMELINE_LIMIT entries for whole-doc comments. + + Prioritizes current entry and nearest self reply. + """ + if len(timeline) <= _WHOLE_TIMELINE_LIMIT: + return timeline + n = len(timeline) + selected = set() + if 0 <= current_index < n: + selected.add(current_index) + if 0 <= nearest_self_index < n: + selected.add(nearest_self_index) + # Expand outward from current + budget = _WHOLE_TIMELINE_LIMIT - len(selected) + lo, hi = current_index - 1, current_index + 1 + while budget > 0 and (lo >= 0 or hi < n): + if lo >= 0 and lo not in selected: + selected.add(lo) + budget -= 1 + lo -= 1 + if budget > 0 and hi < n and hi not in selected: + selected.add(hi) + budget -= 1 + hi += 1 + if not selected: + # Fallback: take last N entries + return timeline[-_WHOLE_TIMELINE_LIMIT:] + return [timeline[i] for i in sorted(selected)] + + +_COMMON_INSTRUCTIONS = """ +This is a Feishu document comment thread, not an IM chat. +Do NOT call feishu_drive_add_comment or feishu_drive_reply_comment yourself. +Your reply will be posted automatically. Just output the reply text. +Use the thread timeline above as the main context. +If the quoted content is not enough, use feishu_doc_read to read nearby context. +The quoted content is your primary anchor — insert/summarize/explain requests are about it. +Do not guess document content you haven't read. +Reply in the same language as the user's comment unless they request otherwise. +Use plain text only. Do not use Markdown, headings, bullet lists, tables, or code blocks. +Do not show your reasoning process. Do not start with "I will", "Let me", or "I'll first". +Output only the final user-facing reply. +If no reply is needed, output exactly NO_REPLY. +""".strip() + + +def build_local_comment_prompt( + *, + doc_title: str, + doc_url: str, + file_token: str, + file_type: str, + comment_id: str, + quote_text: str, + root_comment_text: str, + target_reply_text: str, + timeline: List[Tuple[str, str, bool]], # [(user_id, text, is_self)] + self_open_id: str, + target_index: int = -1, + referenced_docs: str = "", +) -> str: + """Build the prompt for a local (quoted-text) comment.""" + selected = _select_local_timeline(timeline, target_index) + + lines = [ + f'The user added a reply in "{doc_title}".', + f'Current user comment text: "{_truncate(target_reply_text)}"', + f'Original comment text: "{_truncate(root_comment_text)}"', + f'Quoted content: "{_truncate(quote_text, 500)}"', + "This comment mentioned you (@mention is for routing, not task content).", + f"Document link: {doc_url}", + "Current commented document:", + f"- file_type={file_type}", + f"- file_token={file_token}", + f"- comment_id={comment_id}", + "", + f"Current comment card timeline ({len(selected)}/{len(timeline)} entries):", + ] + + for user_id, text, is_self in selected: + marker = " <-- YOU" if is_self else "" + lines.append(f"[{user_id}] {_truncate(text)}{marker}") + + if referenced_docs: + lines.append(referenced_docs) + + lines.append("") + lines.append(_COMMON_INSTRUCTIONS) + return "\n".join(lines) + + +def build_whole_comment_prompt( + *, + doc_title: str, + doc_url: str, + file_token: str, + file_type: str, + comment_text: str, + timeline: List[Tuple[str, str, bool]], # [(user_id, text, is_self)] + self_open_id: str, + current_index: int = -1, + nearest_self_index: int = -1, + referenced_docs: str = "", +) -> str: + """Build the prompt for a whole-document comment.""" + selected = _select_whole_timeline(timeline, current_index, nearest_self_index) + + lines = [ + f'The user added a comment in "{doc_title}".', + f'Current user comment text: "{_truncate(comment_text)}"', + "This is a whole-document comment.", + "This comment mentioned you (@mention is for routing, not task content).", + f"Document link: {doc_url}", + "Current commented document:", + f"- file_type={file_type}", + f"- file_token={file_token}", + "", + f"Whole-document comment timeline ({len(selected)}/{len(timeline)} entries):", + ] + + for user_id, text, is_self in selected: + marker = " <-- YOU" if is_self else "" + lines.append(f"[{user_id}] {_truncate(text)}{marker}") + + if referenced_docs: + lines.append(referenced_docs) + + lines.append("") + lines.append(_COMMON_INSTRUCTIONS) + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Agent execution +# --------------------------------------------------------------------------- + + +def _resolve_model_and_runtime() -> Tuple[str, dict]: + """Resolve model and provider credentials, same as gateway message handling.""" + import os + from gateway.run import _load_gateway_config, _resolve_gateway_model + + user_config = _load_gateway_config() + model = _resolve_gateway_model(user_config) + + from gateway.run import _resolve_runtime_agent_kwargs + runtime_kwargs = _resolve_runtime_agent_kwargs() + + # Fall back to provider's default model if none configured + if not model and runtime_kwargs.get("provider"): + try: + from hermes_cli.models import get_default_model_for_provider + model = get_default_model_for_provider(runtime_kwargs["provider"]) + except Exception: + pass + + return model, runtime_kwargs + + +# --------------------------------------------------------------------------- +# Session cache for cross-card memory within the same document +# --------------------------------------------------------------------------- + +import threading +import time as _time + +_SESSION_MAX_MESSAGES = 50 # keep last N messages per document session +_SESSION_TTL_S = 3600 # expire sessions after 1 hour of inactivity + +_session_cache_lock = threading.Lock() +_session_cache: Dict[str, Dict] = {} # key -> {"messages": [...], "last_access": float} + + +def _session_key(file_type: str, file_token: str) -> str: + return f"comment-doc:{file_type}:{file_token}" + + +def _load_session_history(key: str) -> List[Dict[str, Any]]: + """Load conversation history for a document session.""" + with _session_cache_lock: + entry = _session_cache.get(key) + if entry is None: + return [] + # Check TTL + if _time.time() - entry["last_access"] > _SESSION_TTL_S: + del _session_cache[key] + logger.info("[Feishu-Comment] Session expired: %s", key) + return [] + entry["last_access"] = _time.time() + return list(entry["messages"]) + + +def _save_session_history(key: str, messages: List[Dict[str, Any]]) -> None: + """Save conversation history for a document session (keeps last N messages).""" + # Only keep user/assistant messages (strip system messages and tool internals) + cleaned = [ + m for m in messages + if m.get("role") in ("user", "assistant") and m.get("content") + ] + # Keep last N + if len(cleaned) > _SESSION_MAX_MESSAGES: + cleaned = cleaned[-_SESSION_MAX_MESSAGES:] + with _session_cache_lock: + _session_cache[key] = { + "messages": cleaned, + "last_access": _time.time(), + } + logger.info("[Feishu-Comment] Session saved: %s (%d messages)", key, len(cleaned)) + + +def _run_comment_agent(prompt: str, client: Any, session_key: str = "") -> str: + """Create an AIAgent with feishu tools and run the prompt. + + If *session_key* is provided, loads/saves conversation history for + cross-card memory within the same document. + + Returns the agent's final response text, or empty string on failure. + """ + from run_agent import AIAgent + + logger.info("[Feishu-Comment] _run_comment_agent: injecting lark client into tool thread-locals") + from tools.feishu_doc_tool import set_client as set_doc_client + from tools.feishu_drive_tool import set_client as set_drive_client + set_doc_client(client) + set_drive_client(client) + + try: + model, runtime_kwargs = _resolve_model_and_runtime() + logger.info("[Feishu-Comment] _run_comment_agent: model=%s provider=%s base_url=%s", + model, runtime_kwargs.get("provider"), (runtime_kwargs.get("base_url") or "")[:50]) + + # Load session history for cross-card memory + history = _load_session_history(session_key) if session_key else [] + if history: + logger.info("[Feishu-Comment] _run_comment_agent: loaded %d history messages from session %s", + len(history), session_key) + + agent = AIAgent( + model=model, + base_url=runtime_kwargs.get("base_url"), + api_key=runtime_kwargs.get("api_key"), + provider=runtime_kwargs.get("provider"), + api_mode=runtime_kwargs.get("api_mode"), + credential_pool=runtime_kwargs.get("credential_pool"), + quiet_mode=True, + skip_context_files=True, + skip_memory=True, + max_iterations=15, + enabled_toolsets=["feishu_doc", "feishu_drive"], + ) + logger.info("[Feishu-Comment] _run_comment_agent: calling run_conversation (prompt=%d chars, history=%d)", + len(prompt), len(history)) + result = agent.run_conversation(prompt, conversation_history=history or None) + response = (result.get("final_response") or "").strip() + api_calls = result.get("api_calls", 0) + logger.info("[Feishu-Comment] _run_comment_agent: done api_calls=%d response_len=%d response=%s", + api_calls, len(response), response[:200]) + + # Save updated history + if session_key: + new_messages = result.get("messages", []) + if new_messages: + _save_session_history(session_key, new_messages) + + return response + except Exception as e: + logger.exception("[Feishu-Comment] _run_comment_agent: agent failed: %s", e) + return "" + finally: + set_doc_client(None) + set_drive_client(None) + + +# --------------------------------------------------------------------------- +# Event handler entry point +# --------------------------------------------------------------------------- + +_NO_REPLY_SENTINEL = "NO_REPLY" + + +_ALLOWED_NOTICE_TYPES = {"add_comment", "add_reply"} + + +async def handle_drive_comment_event( + client: Any, data: Any, *, self_open_id: str = "", +) -> None: + """Full orchestration for a drive comment event. + + 1. Parse event + filter (self-reply, notice_type) + 2. Add OK reaction + 3. Fetch doc meta + comment details in parallel + 4. Branch on is_whole: build timeline + 5. Build prompt, run agent + 6. Deliver reply + """ + logger.info("[Feishu-Comment] ========== handle_drive_comment_event START ==========") + parsed = parse_drive_comment_event(data) + if parsed is None: + logger.warning("[Feishu-Comment] Dropping malformed drive comment event") + return + logger.info("[Feishu-Comment] [Step 0/5] Event parsed successfully") + + file_token = parsed["file_token"] + file_type = parsed["file_type"] + comment_id = parsed["comment_id"] + reply_id = parsed["reply_id"] + from_open_id = parsed["from_open_id"] + to_open_id = parsed["to_open_id"] + notice_type = parsed["notice_type"] + + # Filter: self-reply, receiver check, notice_type + if from_open_id and self_open_id and from_open_id == self_open_id: + logger.debug("[Feishu-Comment] Skipping self-authored event: from=%s", from_open_id) + return + if not to_open_id or (self_open_id and to_open_id != self_open_id): + logger.debug("[Feishu-Comment] Skipping event not addressed to self: to=%s", to_open_id or "(empty)") + return + if notice_type and notice_type not in _ALLOWED_NOTICE_TYPES: + logger.debug("[Feishu-Comment] Skipping notice_type=%s", notice_type) + return + if not file_token or not file_type or not comment_id: + logger.warning("[Feishu-Comment] Missing required fields, skipping") + return + + logger.info( + "[Feishu-Comment] Event: notice=%s file=%s:%s comment=%s from=%s", + notice_type, file_type, file_token, comment_id, from_open_id, + ) + + # Access control + from gateway.platforms.feishu_comment_rules import load_config, resolve_rule, is_user_allowed, has_wiki_keys + + comments_cfg = load_config() + rule = resolve_rule(comments_cfg, file_type, file_token) + + # If no exact match and config has wiki keys, try reverse-lookup + if rule.match_source in ("wildcard", "top") and has_wiki_keys(comments_cfg): + wiki_token = await _reverse_lookup_wiki_token(client, file_type, file_token) + if wiki_token: + rule = resolve_rule(comments_cfg, file_type, file_token, wiki_token=wiki_token) + + if not rule.enabled: + logger.info("[Feishu-Comment] Comments disabled for %s:%s, skipping", file_type, file_token) + return + if not is_user_allowed(rule, from_open_id): + logger.info("[Feishu-Comment] User %s denied (policy=%s, rule=%s)", from_open_id, rule.policy, rule.match_source) + return + + logger.info("[Feishu-Comment] Access granted: user=%s policy=%s rule=%s", from_open_id, rule.policy, rule.match_source) + if reply_id: + asyncio.ensure_future( + add_comment_reaction( + client, + file_token=file_token, + file_type=file_type, + reply_id=reply_id, + reaction_type="OK", + ) + ) + + # Step 2: Parallel fetch -- doc meta + comment details + logger.info("[Feishu-Comment] [Step 2/5] Parallel fetch: doc meta + comment batch_query") + meta_task = asyncio.ensure_future( + query_document_meta(client, file_token, file_type) + ) + comment_task = asyncio.ensure_future( + batch_query_comment(client, file_token, file_type, comment_id) + ) + doc_meta, comment_detail = await asyncio.gather(meta_task, comment_task) + + doc_title = doc_meta.get("title", "Untitled") + doc_url = doc_meta.get("url", "") + is_whole = bool(comment_detail.get("is_whole")) + + logger.info( + "[Feishu-Comment] Comment context: title=%s is_whole=%s", + doc_title, is_whole, + ) + + # Step 3: Build timeline based on comment type + logger.info("[Feishu-Comment] [Step 3/5] Building timeline (is_whole=%s)", is_whole) + if is_whole: + # Whole-document comment: fetch all whole comments as timeline + logger.info("[Feishu-Comment] Fetching whole-document comments for timeline...") + whole_comments = await list_whole_comments(client, file_token, file_type) + + timeline: List[Tuple[str, str, bool]] = [] + current_text = "" + current_index = -1 + nearest_self_index = -1 + for wc in whole_comments: + reply_list = wc.get("reply_list", {}) + if isinstance(reply_list, str): + try: + reply_list = json.loads(reply_list) + except (json.JSONDecodeError, TypeError): + reply_list = {} + replies = reply_list.get("replies", []) + for r in replies: + uid = _get_reply_user_id(r) + text = _extract_reply_text(r) + is_self = (uid == self_open_id) if self_open_id else False + idx = len(timeline) + timeline.append((uid, text, is_self)) + if uid == from_open_id: + current_text = _extract_semantic_text(r, self_open_id) + current_index = idx + if is_self: + nearest_self_index = idx + + if not current_text: + for i, (uid, text, is_self) in reversed(list(enumerate(timeline))): + if not is_self: + current_text = text + current_index = i + break + + logger.info("[Feishu-Comment] Whole timeline: %d entries, current_idx=%d, self_idx=%d, text=%s", + len(timeline), current_index, nearest_self_index, + current_text[:80] if current_text else "(empty)") + + # Extract and resolve document links from all replies + all_raw_replies = [] + for wc in whole_comments: + rl = wc.get("reply_list", {}) + if isinstance(rl, str): + try: + rl = json.loads(rl) + except (json.JSONDecodeError, TypeError): + rl = {} + all_raw_replies.extend(rl.get("replies", [])) + doc_links = _extract_docs_links(all_raw_replies) + if doc_links: + doc_links = await _resolve_wiki_nodes(client, doc_links) + ref_docs_text = _format_referenced_docs(doc_links, file_token) + + prompt = build_whole_comment_prompt( + doc_title=doc_title, + doc_url=doc_url, + file_token=file_token, + file_type=file_type, + comment_text=current_text, + timeline=timeline, + self_open_id=self_open_id, + current_index=current_index, + nearest_self_index=nearest_self_index, + referenced_docs=ref_docs_text, + ) + + else: + # Local comment: fetch the comment thread replies + logger.info("[Feishu-Comment] Fetching comment thread replies...") + replies = await list_comment_replies( + client, file_token, file_type, comment_id, + expect_reply_id=reply_id, + ) + + quote_text = comment_detail.get("quote", "") + + timeline = [] + root_text = "" + target_text = "" + target_index = -1 + for i, r in enumerate(replies): + uid = _get_reply_user_id(r) + text = _extract_reply_text(r) + is_self = (uid == self_open_id) if self_open_id else False + timeline.append((uid, text, is_self)) + if i == 0: + root_text = _extract_semantic_text(r, self_open_id) + rid = r.get("reply_id", "") + if rid and rid == reply_id: + target_text = _extract_semantic_text(r, self_open_id) + target_index = i + + if not target_text and timeline: + for i, (uid, text, is_self) in reversed(list(enumerate(timeline))): + if uid == from_open_id: + target_text = text + target_index = i + break + + logger.info("[Feishu-Comment] Local timeline: %d entries, target_idx=%d, quote=%s root=%s target=%s", + len(timeline), target_index, + quote_text[:60] if quote_text else "(empty)", + root_text[:60] if root_text else "(empty)", + target_text[:60] if target_text else "(empty)") + + # Extract and resolve document links from replies + doc_links = _extract_docs_links(replies) + if doc_links: + doc_links = await _resolve_wiki_nodes(client, doc_links) + ref_docs_text = _format_referenced_docs(doc_links, file_token) + + prompt = build_local_comment_prompt( + doc_title=doc_title, + doc_url=doc_url, + file_token=file_token, + file_type=file_type, + comment_id=comment_id, + quote_text=quote_text, + root_comment_text=root_text, + target_reply_text=target_text, + timeline=timeline, + self_open_id=self_open_id, + target_index=target_index, + referenced_docs=ref_docs_text, + ) + + logger.info("[Feishu-Comment] [Step 4/5] Prompt built (%d chars), running agent...", len(prompt)) + logger.debug("[Feishu-Comment] Full prompt:\n%s", prompt) + + # Step 4: Run agent in a thread (run_conversation is synchronous) + # Session key groups all comment cards on the same document + sess_key = _session_key(file_type, file_token) + loop = asyncio.get_running_loop() + response = await loop.run_in_executor( + None, _run_comment_agent, prompt, client, sess_key, + ) + + if not response or _NO_REPLY_SENTINEL in response: + logger.info("[Feishu-Comment] Agent returned NO_REPLY, skipping delivery") + else: + logger.info("[Feishu-Comment] Agent response (%d chars): %s", len(response), response[:200]) + + # Step 5: Deliver reply + logger.info("[Feishu-Comment] [Step 5/5] Delivering reply (is_whole=%s, comment_id=%s)", is_whole, comment_id) + success = await deliver_comment_reply( + client, file_token, file_type, comment_id, response, is_whole, + ) + if success: + logger.info("[Feishu-Comment] Reply delivered successfully") + else: + logger.error("[Feishu-Comment] Failed to deliver reply") + + # Cleanup: remove OK reaction (best-effort, non-blocking) + if reply_id: + await delete_comment_reaction( + client, + file_token=file_token, + file_type=file_type, + reply_id=reply_id, + reaction_type="OK", + ) + + logger.info("[Feishu-Comment] ========== handle_drive_comment_event END ==========") diff --git a/gateway/platforms/feishu_comment_rules.py b/gateway/platforms/feishu_comment_rules.py new file mode 100644 index 000000000000..054ef9569898 --- /dev/null +++ b/gateway/platforms/feishu_comment_rules.py @@ -0,0 +1,429 @@ +""" +Feishu document comment access-control rules. + +3-tier rule resolution: exact doc > wildcard "*" > top-level > code defaults. +Each field (enabled/policy/allow_from) falls back independently. +Config: ~/.hermes/feishu_comment_rules.json (mtime-cached, hot-reload). +Pairing store: ~/.hermes/feishu_comment_pairing.json. +""" + +from __future__ import annotations + +import json +import logging +import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, Optional + +from hermes_constants import get_hermes_home + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Paths +# --------------------------------------------------------------------------- +# +# Uses the canonical ``get_hermes_home()`` helper (HERMES_HOME-aware and +# profile-safe). Resolved at import time; this module is lazy-imported by +# the Feishu comment event handler, which runs long after profile overrides +# have been applied, so freezing paths here is safe. + +RULES_FILE = get_hermes_home() / "feishu_comment_rules.json" +PAIRING_FILE = get_hermes_home() / "feishu_comment_pairing.json" + +# --------------------------------------------------------------------------- +# Data models +# --------------------------------------------------------------------------- + +_VALID_POLICIES = ("allowlist", "pairing") + + +@dataclass(frozen=True) +class CommentDocumentRule: + """Per-document rule. ``None`` means 'inherit from lower tier'.""" + enabled: Optional[bool] = None + policy: Optional[str] = None + allow_from: Optional[frozenset] = None + + +@dataclass(frozen=True) +class CommentsConfig: + """Top-level comment access config.""" + enabled: bool = True + policy: str = "pairing" + allow_from: frozenset = field(default_factory=frozenset) + documents: Dict[str, CommentDocumentRule] = field(default_factory=dict) + + +@dataclass(frozen=True) +class ResolvedCommentRule: + """Fully resolved rule after field-by-field fallback.""" + enabled: bool + policy: str + allow_from: frozenset + match_source: str # e.g. "exact:docx:xxx" | "wildcard" | "top" | "default" + + +# --------------------------------------------------------------------------- +# Mtime-cached file loading +# --------------------------------------------------------------------------- + +class _MtimeCache: + """Generic mtime-based file cache. ``stat()`` per access, re-read only on change.""" + + def __init__(self, path: Path): + self._path = path + self._mtime: float = 0.0 + self._data: Optional[dict] = None + + def load(self) -> dict: + try: + st = self._path.stat() + mtime = st.st_mtime + except FileNotFoundError: + self._mtime = 0.0 + self._data = {} + return {} + + if mtime == self._mtime and self._data is not None: + return self._data + + try: + with open(self._path, "r", encoding="utf-8") as f: + data = json.load(f) + if not isinstance(data, dict): + data = {} + except (json.JSONDecodeError, OSError): + logger.warning("[Feishu-Rules] Failed to read %s, using empty config", self._path) + data = {} + + self._mtime = mtime + self._data = data + return data + + +_rules_cache = _MtimeCache(RULES_FILE) +_pairing_cache = _MtimeCache(PAIRING_FILE) + + +# --------------------------------------------------------------------------- +# Config parsing +# --------------------------------------------------------------------------- + +def _parse_frozenset(raw: Any) -> Optional[frozenset]: + """Parse a list of strings into a frozenset; return None if key absent.""" + if raw is None: + return None + if isinstance(raw, (list, tuple)): + return frozenset(str(u).strip() for u in raw if str(u).strip()) + return None + + +def _parse_document_rule(raw: dict) -> CommentDocumentRule: + enabled = raw.get("enabled") + if enabled is not None: + enabled = bool(enabled) + policy = raw.get("policy") + if policy is not None: + policy = str(policy).strip().lower() + if policy not in _VALID_POLICIES: + policy = None + allow_from = _parse_frozenset(raw.get("allow_from")) + return CommentDocumentRule(enabled=enabled, policy=policy, allow_from=allow_from) + + +def load_config() -> CommentsConfig: + """Load comment rules from disk (mtime-cached).""" + raw = _rules_cache.load() + if not raw: + return CommentsConfig() + + documents: Dict[str, CommentDocumentRule] = {} + raw_docs = raw.get("documents", {}) + if isinstance(raw_docs, dict): + for key, rule_raw in raw_docs.items(): + if isinstance(rule_raw, dict): + documents[str(key)] = _parse_document_rule(rule_raw) + + policy = str(raw.get("policy", "pairing")).strip().lower() + if policy not in _VALID_POLICIES: + policy = "pairing" + + return CommentsConfig( + enabled=raw.get("enabled", True), + policy=policy, + allow_from=_parse_frozenset(raw.get("allow_from")) or frozenset(), + documents=documents, + ) + + +# --------------------------------------------------------------------------- +# Rule resolution (§8.4 field-by-field fallback) +# --------------------------------------------------------------------------- + +def has_wiki_keys(cfg: CommentsConfig) -> bool: + """Check if any document rule key starts with 'wiki:'.""" + return any(k.startswith("wiki:") for k in cfg.documents) + + +def resolve_rule( + cfg: CommentsConfig, + file_type: str, + file_token: str, + wiki_token: str = "", +) -> ResolvedCommentRule: + """Resolve effective rule: exact doc → wiki key → wildcard → top-level → defaults.""" + exact_key = f"{file_type}:{file_token}" + + exact = cfg.documents.get(exact_key) + exact_src = f"exact:{exact_key}" + if exact is None and wiki_token: + wiki_key = f"wiki:{wiki_token}" + exact = cfg.documents.get(wiki_key) + exact_src = f"exact:{wiki_key}" + + wildcard = cfg.documents.get("*") + + layers = [] + if exact is not None: + layers.append((exact, exact_src)) + if wildcard is not None: + layers.append((wildcard, "wildcard")) + + def _pick(field_name: str): + for layer, source in layers: + val = getattr(layer, field_name) + if val is not None: + return val, source + return getattr(cfg, field_name), "top" + + enabled, en_src = _pick("enabled") + policy, pol_src = _pick("policy") + allow_from, _ = _pick("allow_from") + + # match_source = highest-priority tier that contributed any field + priority_order = {"exact": 0, "wildcard": 1, "top": 2} + best_src = min( + [en_src, pol_src], + key=lambda s: priority_order.get(s.split(":")[0], 3), + ) + + return ResolvedCommentRule( + enabled=enabled, + policy=policy, + allow_from=allow_from, + match_source=best_src, + ) + + +# --------------------------------------------------------------------------- +# Pairing store +# --------------------------------------------------------------------------- + +def _load_pairing_approved() -> set: + """Return set of approved user open_ids (mtime-cached).""" + data = _pairing_cache.load() + approved = data.get("approved", {}) + if isinstance(approved, dict): + return set(approved.keys()) + if isinstance(approved, list): + return set(str(u) for u in approved if u) + return set() + + +def _save_pairing(data: dict) -> None: + PAIRING_FILE.parent.mkdir(parents=True, exist_ok=True) + tmp = PAIRING_FILE.with_suffix(".tmp") + with open(tmp, "w", encoding="utf-8") as f: + json.dump(data, f, indent=2, ensure_ascii=False) + tmp.replace(PAIRING_FILE) + # Invalidate cache so next load picks up change + _pairing_cache._mtime = 0.0 + _pairing_cache._data = None + + +def pairing_add(user_open_id: str) -> bool: + """Add a user to the pairing-approved list. Returns True if newly added.""" + data = _pairing_cache.load() + approved = data.get("approved", {}) + if not isinstance(approved, dict): + approved = {} + if user_open_id in approved: + return False + approved[user_open_id] = {"approved_at": time.time()} + data["approved"] = approved + _save_pairing(data) + return True + + +def pairing_remove(user_open_id: str) -> bool: + """Remove a user from the pairing-approved list. Returns True if removed.""" + data = _pairing_cache.load() + approved = data.get("approved", {}) + if not isinstance(approved, dict): + return False + if user_open_id not in approved: + return False + del approved[user_open_id] + data["approved"] = approved + _save_pairing(data) + return True + + +def pairing_list() -> Dict[str, Any]: + """Return the approved dict {user_open_id: {approved_at: ...}}.""" + data = _pairing_cache.load() + approved = data.get("approved", {}) + return dict(approved) if isinstance(approved, dict) else {} + + +# --------------------------------------------------------------------------- +# Access check (public API for feishu_comment.py) +# --------------------------------------------------------------------------- + +def is_user_allowed(rule: ResolvedCommentRule, user_open_id: str) -> bool: + """Check if user passes the resolved rule's policy gate.""" + if user_open_id in rule.allow_from: + return True + if rule.policy == "pairing": + return user_open_id in _load_pairing_approved() + return False + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + +def _print_status() -> None: + cfg = load_config() + print(f"Rules file: {RULES_FILE}") + print(f" exists: {RULES_FILE.exists()}") + print(f"Pairing file: {PAIRING_FILE}") + print(f" exists: {PAIRING_FILE.exists()}") + print() + print(f"Top-level:") + print(f" enabled: {cfg.enabled}") + print(f" policy: {cfg.policy}") + print(f" allow_from: {sorted(cfg.allow_from) if cfg.allow_from else '[]'}") + print() + if cfg.documents: + print(f"Document rules ({len(cfg.documents)}):") + for key, rule in sorted(cfg.documents.items()): + parts = [] + if rule.enabled is not None: + parts.append(f"enabled={rule.enabled}") + if rule.policy is not None: + parts.append(f"policy={rule.policy}") + if rule.allow_from is not None: + parts.append(f"allow_from={sorted(rule.allow_from)}") + print(f" [{key}] {', '.join(parts) if parts else '(empty — inherits all)'}") + else: + print("Document rules: (none)") + print() + approved = pairing_list() + print(f"Pairing approved ({len(approved)}):") + for uid, meta in sorted(approved.items()): + ts = meta.get("approved_at", 0) + print(f" {uid} (approved_at={ts})") + + +def _do_check(doc_key: str, user_open_id: str) -> None: + cfg = load_config() + parts = doc_key.split(":", 1) + if len(parts) != 2: + print(f"Error: doc_key must be 'fileType:fileToken', got '{doc_key}'") + return + file_type, file_token = parts + rule = resolve_rule(cfg, file_type, file_token) + allowed = is_user_allowed(rule, user_open_id) + print(f"Document: {doc_key}") + print(f"User: {user_open_id}") + print(f"Resolved rule:") + print(f" enabled: {rule.enabled}") + print(f" policy: {rule.policy}") + print(f" allow_from: {sorted(rule.allow_from) if rule.allow_from else '[]'}") + print(f" match_source: {rule.match_source}") + print(f"Result: {'ALLOWED' if allowed else 'DENIED'}") + + +def _main() -> int: + import sys + + try: + from hermes_cli.env_loader import load_hermes_dotenv + load_hermes_dotenv() + except Exception: + pass + + usage = ( + "Usage: python -m gateway.platforms.feishu_comment_rules [args]\n" + "\n" + "Commands:\n" + " status Show rules config and pairing state\n" + " check Simulate access check\n" + " pairing add Add user to pairing-approved list\n" + " pairing remove Remove user from pairing-approved list\n" + " pairing list List pairing-approved users\n" + "\n" + f"Rules config file: {RULES_FILE}\n" + " Edit this JSON file directly to configure policies and document rules.\n" + " Changes take effect on the next comment event (no restart needed).\n" + ) + + args = sys.argv[1:] + if not args: + print(usage) + return 1 + + cmd = args[0] + + if cmd == "status": + _print_status() + + elif cmd == "check": + if len(args) < 3: + print("Usage: check ") + return 1 + _do_check(args[1], args[2]) + + elif cmd == "pairing": + if len(args) < 2: + print("Usage: pairing [args]") + return 1 + sub = args[1] + if sub == "add": + if len(args) < 3: + print("Usage: pairing add ") + return 1 + if pairing_add(args[2]): + print(f"Added: {args[2]}") + else: + print(f"Already approved: {args[2]}") + elif sub == "remove": + if len(args) < 3: + print("Usage: pairing remove ") + return 1 + if pairing_remove(args[2]): + print(f"Removed: {args[2]}") + else: + print(f"Not in approved list: {args[2]}") + elif sub == "list": + approved = pairing_list() + if not approved: + print("(no approved users)") + for uid, meta in sorted(approved.items()): + print(f" {uid} approved_at={meta.get('approved_at', '?')}") + else: + print(f"Unknown pairing subcommand: {sub}") + return 1 + else: + print(f"Unknown command: {cmd}\n") + print(usage) + return 1 + return 0 + + +if __name__ == "__main__": + import sys + sys.exit(_main()) diff --git a/gateway/platforms/helpers.py b/gateway/platforms/helpers.py index c834dd89ca01..18d97fcb7a16 100644 --- a/gateway/platforms/helpers.py +++ b/gateway/platforms/helpers.py @@ -49,7 +49,10 @@ def is_duplicate(self, msg_id: str) -> bool: return False now = time.time() if msg_id in self._seen: - return True + if now - self._seen[msg_id] < self._ttl: + return True + # Entry has expired — remove it and treat as new + del self._seen[msg_id] self._seen[msg_id] = now if len(self._seen) > self._max_size: cutoff = now - self._ttl diff --git a/gateway/platforms/matrix.py b/gateway/platforms/matrix.py index 654d77070ed7..a5f9352b555c 100644 --- a/gateway/platforms/matrix.py +++ b/gateway/platforms/matrix.py @@ -25,17 +25,15 @@ from __future__ import annotations import asyncio -import json import logging import mimetypes import os import re import time +from html import escape as _html_escape from pathlib import Path from typing import Any, Dict, Optional, Set -from html import escape as _html_escape - try: from mautrix.types import ( ContentURI, @@ -61,28 +59,33 @@ class _EventTypeStub: # type: ignore[no-redef] REACTION = "m.reaction" ROOM_ENCRYPTED = "m.room.encrypted" ROOM_NAME = "m.room.name" + EventType = _EventTypeStub # type: ignore[misc,assignment] class _PaginationDirectionStub: # type: ignore[no-redef] BACKWARD = "b" FORWARD = "f" + PaginationDirection = _PaginationDirectionStub # type: ignore[misc,assignment] class _PresenceStateStub: # type: ignore[no-redef] ONLINE = "online" OFFLINE = "offline" UNAVAILABLE = "unavailable" + PresenceState = _PresenceStateStub # type: ignore[misc,assignment] class _RoomCreatePresetStub: # type: ignore[no-redef] PRIVATE = "private_chat" PUBLIC = "public_chat" TRUSTED_PRIVATE = "trusted_private_chat" + RoomCreatePreset = _RoomCreatePresetStub # type: ignore[misc,assignment] class _TrustStateStub: # type: ignore[no-redef] UNVERIFIED = 0 VERIFIED = 1 + TrustState = _TrustStateStub # type: ignore[misc,assignment] from gateway.config import Platform, PlatformConfig @@ -104,20 +107,16 @@ class _TrustStateStub: # type: ignore[no-redef] # Store directory for E2EE keys and sync state. # Uses get_hermes_home() so each profile gets its own Matrix store. from hermes_constants import get_hermes_dir as _get_hermes_dir + _STORE_DIR = _get_hermes_dir("platforms/matrix/store", "matrix/store") _CRYPTO_DB_PATH = _STORE_DIR / "crypto.db" # Grace period: ignore messages older than this many seconds before startup. _STARTUP_GRACE_SECONDS = 5 -# Pending undecrypted events: cap and TTL for retry buffer. -_MAX_PENDING_EVENTS = 100 -_PENDING_EVENT_TTL = 300 # seconds — stop retrying after 5 min - _E2EE_INSTALL_HINT = ( - "Install with: pip install 'mautrix[encryption]' " - "(requires libolm C library)" + "Install with: pip install 'mautrix[encryption]' (requires libolm C library)" ) @@ -125,6 +124,7 @@ def _check_e2ee_deps() -> bool: """Return True if mautrix E2EE dependencies (python-olm) are available.""" try: from mautrix.crypto import OlmMachine # noqa: F401 + return True except (ImportError, AttributeError): return False @@ -146,14 +146,17 @@ def check_matrix_requirements() -> bool: import mautrix # noqa: F401 except ImportError: logger.warning( - "Matrix: mautrix not installed. " - "Run: pip install 'mautrix[encryption]'" + "Matrix: mautrix not installed. Run: pip install 'mautrix[encryption]'" ) return False # If encryption is requested, verify E2EE deps are available at startup # rather than silently degrading to plaintext-only at connect time. - encryption_requested = os.getenv("MATRIX_ENCRYPTION", "").lower() in ("true", "1", "yes") + encryption_requested = os.getenv("MATRIX_ENCRYPTION", "").lower() in ( + "true", + "1", + "yes", + ) if encryption_requested and not _check_e2ee_deps(): logger.error( "Matrix: MATRIX_ENCRYPTION=true but E2EE dependencies are missing. %s. " @@ -205,25 +208,21 @@ def __init__(self, config: PlatformConfig): super().__init__(config, Platform.MATRIX) self._homeserver: str = ( - config.extra.get("homeserver", "") - or os.getenv("MATRIX_HOMESERVER", "") + config.extra.get("homeserver", "") or os.getenv("MATRIX_HOMESERVER", "") ).rstrip("/") self._access_token: str = config.token or os.getenv("MATRIX_ACCESS_TOKEN", "") - self._user_id: str = ( - config.extra.get("user_id", "") - or os.getenv("MATRIX_USER_ID", "") + self._user_id: str = config.extra.get("user_id", "") or os.getenv( + "MATRIX_USER_ID", "" ) - self._password: str = ( - config.extra.get("password", "") - or os.getenv("MATRIX_PASSWORD", "") + self._password: str = config.extra.get("password", "") or os.getenv( + "MATRIX_PASSWORD", "" ) self._encryption: bool = config.extra.get( "encryption", os.getenv("MATRIX_ENCRYPTION", "").lower() in ("true", "1", "yes"), ) - self._device_id: str = ( - config.extra.get("device_id", "") - or os.getenv("MATRIX_DEVICE_ID", "") + self._device_id: str = config.extra.get("device_id", "") or os.getenv( + "MATRIX_DEVICE_ID", "" ) self._client: Any = None # mautrix.client.Client @@ -238,22 +237,32 @@ def __init__(self, config: PlatformConfig): self._joined_rooms: Set[str] = set() # Event deduplication (bounded deque keeps newest entries) from collections import deque + self._processed_events: deque = deque(maxlen=1000) self._processed_events_set: set = set() # Buffer for undecrypted events pending key receipt. # Each entry: (room_id, event, timestamp) - self._pending_megolm: list = [] # Thread participation tracking (for require_mention bypass) self._threads = ThreadParticipationTracker("matrix") # Mention/thread gating — parsed once from env vars. - self._require_mention: bool = os.getenv("MATRIX_REQUIRE_MENTION", "true").lower() not in ("false", "0", "no") + self._require_mention: bool = os.getenv( + "MATRIX_REQUIRE_MENTION", "true" + ).lower() not in ("false", "0", "no") free_rooms_raw = os.getenv("MATRIX_FREE_RESPONSE_ROOMS", "") - self._free_rooms: Set[str] = {r.strip() for r in free_rooms_raw.split(",") if r.strip()} - self._auto_thread: bool = os.getenv("MATRIX_AUTO_THREAD", "true").lower() in ("true", "1", "yes") - self._dm_mention_threads: bool = os.getenv("MATRIX_DM_MENTION_THREADS", "false").lower() in ("true", "1", "yes") + self._free_rooms: Set[str] = { + r.strip() for r in free_rooms_raw.split(",") if r.strip() + } + self._auto_thread: bool = os.getenv("MATRIX_AUTO_THREAD", "true").lower() in ( + "true", + "1", + "yes", + ) + self._dm_mention_threads: bool = os.getenv( + "MATRIX_DM_MENTION_THREADS", "false" + ).lower() in ("true", "1", "yes") # Reactions: configurable via MATRIX_REACTIONS (default: true). self._reactions_enabled: bool = os.getenv( @@ -263,8 +272,12 @@ def __init__(self, config: PlatformConfig): # Text batching: merge rapid successive messages (Telegram-style). # Matrix clients split long messages around 4000 chars. - self._text_batch_delay_seconds = float(os.getenv("HERMES_MATRIX_TEXT_BATCH_DELAY_SECONDS", "0.6")) - self._text_batch_split_delay_seconds = float(os.getenv("HERMES_MATRIX_TEXT_BATCH_SPLIT_DELAY_SECONDS", "2.0")) + self._text_batch_delay_seconds = float( + os.getenv("HERMES_MATRIX_TEXT_BATCH_DELAY_SECONDS", "0.6") + ) + self._text_batch_split_delay_seconds = float( + os.getenv("HERMES_MATRIX_TEXT_BATCH_SPLIT_DELAY_SECONDS", "2.0") + ) self._pending_text_batches: Dict[str, MessageEvent] = {} self._pending_text_batch_tasks: Dict[str, asyncio.Task] = {} @@ -285,6 +298,38 @@ def _is_duplicate_event(self, event_id) -> bool: # E2EE helpers # ------------------------------------------------------------------ + @staticmethod + def _extract_server_ed25519(device_keys_obj: Any) -> Optional[str]: + """Extract the ed25519 identity key from a DeviceKeys object.""" + for kid, kval in (getattr(device_keys_obj, "keys", {}) or {}).items(): + if str(kid).startswith("ed25519:"): + return str(kval) + return None + + async def _reverify_keys_after_upload( + self, client: Any, local_ed25519: str + ) -> bool: + """Re-query the server after share_keys() and verify our ed25519 key matches.""" + try: + resp = await client.query_keys({client.mxid: [client.device_id]}) + dk = getattr(resp, "device_keys", {}) or {} + ud = dk.get(str(client.mxid)) or {} + dev = ud.get(str(client.device_id)) + if dev: + server_ed = self._extract_server_ed25519(dev) + if server_ed != local_ed25519: + logger.error( + "Matrix: device %s has immutable identity keys that " + "don't match this installation. Generate a new access " + "token with a fresh device.", + client.device_id, + ) + return False + except Exception as exc: + logger.error("Matrix: post-upload key verification failed: %s", exc) + return False + return True + async def _verify_device_keys_on_server(self, client: Any, olm: Any) -> bool: """Verify our device keys are on the homeserver after loading crypto state. @@ -295,15 +340,15 @@ async def _verify_device_keys_on_server(self, client: Any, olm: Any) -> bool: resp = await client.query_keys({client.mxid: [client.device_id]}) except Exception as exc: logger.error( - "Matrix: cannot verify device keys on server: %s — refusing E2EE", exc, + "Matrix: cannot verify device keys on server: %s — refusing E2EE", + exc, ) return False - # query_keys returns typed objects (QueryKeysResponse, DeviceKeys - # with KeyID keys). Normalise to plain strings for comparison. device_keys_map = getattr(resp, "device_keys", {}) or {} our_user_devices = device_keys_map.get(str(client.mxid)) or {} our_keys = our_user_devices.get(str(client.device_id)) + local_ed25519 = olm.account.identity_keys.get("ed25519") if not our_keys: logger.warning("Matrix: device keys missing from server — re-uploading") @@ -313,21 +358,12 @@ async def _verify_device_keys_on_server(self, client: Any, olm: Any) -> bool: except Exception as exc: logger.error("Matrix: failed to re-upload device keys: %s", exc) return False - return True + return await self._reverify_keys_after_upload(client, local_ed25519) - # DeviceKeys.keys is a dict[KeyID, str]. Iterate to find the - # ed25519 key rather than constructing a KeyID for lookup. - server_ed25519 = None - keys_dict = getattr(our_keys, "keys", {}) or {} - for key_id, key_value in keys_dict.items(): - if str(key_id).startswith("ed25519:"): - server_ed25519 = str(key_value) - break - local_ed25519 = olm.account.identity_keys.get("ed25519") + server_ed25519 = self._extract_server_ed25519(our_keys) if server_ed25519 != local_ed25519: if olm.account.shared: - # Restored account from DB but server has different keys — corrupted state. logger.error( "Matrix: server has different identity keys for device %s — " "local crypto state is stale. Delete %s and restart.", @@ -336,8 +372,6 @@ async def _verify_device_keys_on_server(self, client: Any, olm: Any) -> bool: ) return False - # Fresh account (never uploaded). Server has stale keys from a - # previous installation. Try to delete the old device and re-upload. logger.warning( "Matrix: server has stale keys for device %s — attempting re-upload", client.device_id, @@ -349,10 +383,10 @@ async def _verify_device_keys_on_server(self, client: Any, olm: Any) -> bool: else "DELETE", f"/_matrix/client/v3/devices/{client.device_id}", ) - logger.info("Matrix: deleted stale device %s from server", client.device_id) + logger.info( + "Matrix: deleted stale device %s from server", client.device_id + ) except Exception: - # Device deletion often requires UIA or may simply not be - # permitted — that's fine, share_keys will try to overwrite. pass try: await olm.share_keys() @@ -364,6 +398,7 @@ async def _verify_device_keys_on_server(self, client: Any, olm: Any) -> bool: exc, ) return False + return await self._reverify_keys_after_upload(client, local_ed25519) return True @@ -449,7 +484,9 @@ async def connect(self) -> bool: await api.session.close() return False else: - logger.error("Matrix: need MATRIX_ACCESS_TOKEN or MATRIX_USER_ID + MATRIX_PASSWORD") + logger.error( + "Matrix: need MATRIX_ACCESS_TOKEN or MATRIX_USER_ID + MATRIX_PASSWORD" + ) await api.session.close() return False @@ -473,7 +510,9 @@ async def connect(self) -> bool: # Remove legacy pickle file from pre-SQLite era. legacy_pickle = _STORE_DIR / "crypto_store.pickle" if legacy_pickle.exists(): - logger.info("Matrix: removing legacy crypto_store.pickle (migrated to SQLite)") + logger.info( + "Matrix: removing legacy crypto_store.pickle (migrated to SQLite)" + ) legacy_pickle.unlink() # Open SQLite-backed crypto store. @@ -509,6 +548,37 @@ async def connect(self) -> bool: await api.session.close() return False + # Proactively flush one-time keys to detect stale OTK + # conflicts early. When crypto state is wiped but the + # same device ID is reused, the server may still hold OTKs + # signed with the old ed25519 key. Identity key re-upload + # succeeds but OTK uploads fail ("already exists" with + # mismatched signature). Peers then cannot establish Olm + # sessions and all new messages are undecryptable. + try: + await olm.share_keys() + except Exception as exc: + exc_str = str(exc) + if "already exists" in exc_str: + logger.error( + "Matrix: device %s has stale one-time keys on the " + "server signed with a previous identity key. " + "Peers cannot establish new Olm sessions with " + "this device. Delete the device from the " + "homeserver and restart, or generate a new " + "access token to get a fresh device ID.", + client.device_id, + ) + await crypto_db.stop() + await api.session.close() + return False + # Non-OTK errors are transient (network, etc.) — log + # but allow startup to continue. + logger.warning( + "Matrix: share_keys() warning during startup: %s", + exc, + ) + # Import cross-signing private keys from SSSS and self-sign # the current device. Required after any device-key rotation # (fresh crypto.db, share_keys re-upload) — otherwise the @@ -520,7 +590,9 @@ async def connect(self) -> bool: await olm.verify_with_recovery_key(recovery_key) logger.info("Matrix: cross-signing verified via recovery key") except Exception as exc: - logger.warning("Matrix: recovery key verification failed: %s", exc) + logger.warning( + "Matrix: recovery key verification failed: %s", exc + ) client.crypto = olm logger.info( @@ -531,21 +603,23 @@ async def connect(self) -> bool: except Exception as exc: logger.error( "Matrix: failed to create E2EE client: %s. %s", - exc, _E2EE_INSTALL_HINT, + exc, + _E2EE_INSTALL_HINT, ) await api.session.close() return False # Register event handlers. from mautrix.client import InternalEventType as IntEvt + from mautrix.client.dispatcher import MembershipEventDispatcher + + # Without this the INVITE handler below never fires. + client.add_dispatcher(MembershipEventDispatcher) client.add_event_handler(EventType.ROOM_MESSAGE, self._on_room_message) client.add_event_handler(EventType.REACTION, self._on_reaction) client.add_event_handler(IntEvt.INVITE, self._on_invite) - if self._encryption and getattr(client, "crypto", None): - client.add_event_handler(EventType.ROOM_ENCRYPTED, self._on_encrypted_event) - # Initial sync to catch up, then start background sync. self._startup_ts = time.time() self._closing = False @@ -554,7 +628,8 @@ async def connect(self) -> bool: sync_data = await client.sync(timeout=10000, full_state=True) if isinstance(sync_data, dict): rooms_join = sync_data.get("rooms", {}).get("join", {}) - self._joined_rooms = set(rooms_join.keys()) + self._joined_rooms.clear() + self._joined_rooms.update(rooms_join.keys()) # Store the next_batch token so incremental syncs start # from where the initial sync left off. nb = sync_data.get("next_batch") @@ -576,7 +651,10 @@ async def connect(self) -> bool: except Exception as exc: logger.warning("Matrix: initial sync event dispatch error: %s", exc) else: - logger.warning("Matrix: initial sync returned unexpected type %s", type(sync_data).__name__) + logger.warning( + "Matrix: initial sync returned unexpected type %s", + type(sync_data).__name__, + ) except Exception as exc: logger.warning("Matrix: initial sync error: %s", exc) @@ -649,9 +727,7 @@ async def send( # Reply-to support. if reply_to: - msg_content["m.relates_to"] = { - "m.in_reply_to": {"event_id": reply_to} - } + msg_content["m.relates_to"] = {"m.in_reply_to": {"event_id": reply_to}} # Thread support: if metadata has thread_id, send as threaded reply. thread_id = (metadata or {}).get("thread_id") @@ -689,10 +765,18 @@ async def send( timeout=45, ) last_event_id = str(event_id) - logger.info("Matrix: sent event %s to %s (after key share)", last_event_id, chat_id) + logger.info( + "Matrix: sent event %s to %s (after key share)", + last_event_id, + chat_id, + ) continue except Exception as retry_exc: - logger.error("Matrix: failed to send to %s after retry: %s", chat_id, retry_exc) + logger.error( + "Matrix: failed to send to %s after retry: %s", + chat_id, + retry_exc, + ) return SendResult(success=False, error=str(retry_exc)) logger.error("Matrix: failed to send to %s: %s", chat_id, exc) return SendResult(success=False, error=str(exc)) @@ -707,7 +791,8 @@ async def get_chat_info(self, chat_id: str) -> Dict[str, Any]: if self._client: try: name_evt = await self._client.get_state_event( - RoomID(chat_id), EventType.ROOM_NAME, + RoomID(chat_id), + EventType.ROOM_NAME, ) if name_evt and hasattr(name_evt, "name") and name_evt.name: name = name_evt.name @@ -730,8 +815,17 @@ async def send_typing( except Exception: pass + async def stop_typing(self, chat_id: str) -> None: + """Clear the typing indicator.""" + if self._client: + try: + await self._client.set_typing(RoomID(chat_id), timeout=0) + except Exception: + pass + + async def edit_message( - self, chat_id: str, message_id: str, content: str + self, chat_id: str, message_id: str, content: str, *, finalize: bool = False ) -> SendResult: """Edit an existing message (via m.replace).""" @@ -758,7 +852,9 @@ async def edit_message( try: event_id = await self._client.send_message_event( - RoomID(chat_id), EventType.ROOM_MESSAGE, msg_content, + RoomID(chat_id), + EventType.ROOM_MESSAGE, + msg_content, ) return SendResult(success=True, message_id=str(event_id)) except Exception as exc: @@ -774,22 +870,31 @@ async def send_image( ) -> SendResult: """Download an image URL and upload it to Matrix.""" from tools.url_safety import is_safe_url + if not is_safe_url(image_url): logger.warning("Matrix: blocked unsafe image URL (SSRF protection)") - return await super().send_image(chat_id, image_url, caption, reply_to, metadata=metadata) + return await super().send_image( + chat_id, image_url, caption, reply_to, metadata=metadata + ) try: # Try aiohttp first (always available), fall back to httpx try: import aiohttp as _aiohttp + async with _aiohttp.ClientSession(trust_env=True) as http: - async with http.get(image_url, timeout=_aiohttp.ClientTimeout(total=30)) as resp: + async with http.get( + image_url, timeout=_aiohttp.ClientTimeout(total=30) + ) as resp: resp.raise_for_status() data = await resp.read() ct = resp.content_type or "image/png" - fname = image_url.rsplit("/", 1)[-1].split("?")[0] or "image.png" + fname = ( + image_url.rsplit("/", 1)[-1].split("?")[0] or "image.png" + ) except ImportError: import httpx + async with httpx.AsyncClient() as http: resp = await http.get(image_url, follow_redirects=True, timeout=30) resp.raise_for_status() @@ -798,9 +903,13 @@ async def send_image( fname = image_url.rsplit("/", 1)[-1].split("?")[0] or "image.png" except Exception as exc: logger.warning("Matrix: failed to download image %s: %s", image_url, exc) - return await self.send(chat_id, f"{caption or ''}\n{image_url}".strip(), reply_to) + return await self.send( + chat_id, f"{caption or ''}\n{image_url}".strip(), reply_to + ) - return await self._upload_and_send(chat_id, data, fname, ct, "m.image", caption, reply_to, metadata) + return await self._upload_and_send( + chat_id, data, fname, ct, "m.image", caption, reply_to, metadata + ) async def send_image_file( self, @@ -811,7 +920,9 @@ async def send_image_file( metadata: Optional[Dict[str, Any]] = None, ) -> SendResult: """Upload a local image file to Matrix.""" - return await self._send_local_file(chat_id, image_path, "m.image", caption, reply_to, metadata=metadata) + return await self._send_local_file( + chat_id, image_path, "m.image", caption, reply_to, metadata=metadata + ) async def send_document( self, @@ -823,7 +934,9 @@ async def send_document( metadata: Optional[Dict[str, Any]] = None, ) -> SendResult: """Upload a local file as a document.""" - return await self._send_local_file(chat_id, file_path, "m.file", caption, reply_to, file_name, metadata) + return await self._send_local_file( + chat_id, file_path, "m.file", caption, reply_to, file_name, metadata + ) async def send_voice( self, @@ -835,8 +948,13 @@ async def send_voice( ) -> SendResult: """Upload an audio file as a voice message (MSC3245 native voice).""" return await self._send_local_file( - chat_id, audio_path, "m.audio", caption, reply_to, - metadata=metadata, is_voice=True + chat_id, + audio_path, + "m.audio", + caption, + reply_to, + metadata=metadata, + is_voice=True, ) async def send_video( @@ -848,7 +966,9 @@ async def send_video( metadata: Optional[Dict[str, Any]] = None, ) -> SendResult: """Upload a video file.""" - return await self._send_local_file(chat_id, video_path, "m.video", caption, reply_to, metadata=metadata) + return await self._send_local_file( + chat_id, video_path, "m.video", caption, reply_to, metadata=metadata + ) def format_message(self, content: str) -> str: """Pass-through — Matrix supports standard Markdown natively.""" @@ -874,12 +994,30 @@ async def _upload_and_send( ) -> SendResult: """Upload bytes to Matrix and send as a media message.""" + upload_data = data + encrypted_file = None + if self._encryption and getattr(self._client, "crypto", None): + state_store = getattr(self._client, "state_store", None) + if state_store: + try: + room_encrypted = bool(await state_store.is_encrypted(RoomID(room_id))) + except Exception: + room_encrypted = False + if room_encrypted: + try: + from mautrix.crypto.attachments import encrypt_attachment + upload_data, encrypted_file = encrypt_attachment(data) + except Exception as exc: + logger.error("Matrix: attachment encryption failed: %s", exc) + return SendResult(success=False, error=str(exc)) + # Upload to homeserver. try: mxc_url = await self._client.upload_media( - data, + upload_data, mime_type=content_type, filename=filename, + size=len(upload_data), ) except Exception as exc: logger.error("Matrix: upload failed: %s", exc) @@ -889,21 +1027,24 @@ async def _upload_and_send( msg_content: Dict[str, Any] = { "msgtype": msgtype, "body": caption or filename, - "url": str(mxc_url), "info": { "mimetype": content_type, "size": len(data), }, } + if encrypted_file is not None: + file_payload = encrypted_file.serialize() + file_payload["url"] = str(mxc_url) + msg_content["file"] = file_payload + else: + msg_content["url"] = str(mxc_url) # Add MSC3245 voice flag for native voice messages. if is_voice: msg_content["org.matrix.msc3245.voice"] = {} if reply_to: - msg_content["m.relates_to"] = { - "m.in_reply_to": {"event_id": reply_to} - } + msg_content["m.relates_to"] = {"m.in_reply_to": {"event_id": reply_to}} thread_id = (metadata or {}).get("thread_id") if thread_id: @@ -915,7 +1056,9 @@ async def _upload_and_send( try: event_id = await self._client.send_message_event( - RoomID(room_id), EventType.ROOM_MESSAGE, msg_content, + RoomID(room_id), + EventType.ROOM_MESSAGE, + msg_content, ) return SendResult(success=True, message_id=str(event_id)) except Exception as exc: @@ -933,7 +1076,7 @@ async def _send_local_file( is_voice: bool = False, ) -> SendResult: """Read a local file and upload it.""" - p = Path(file_path) + p = Path(file_path).expanduser() if not p.exists(): return await self.send( room_id, f"{caption or ''}\n(file not found: {file_path})", reply_to @@ -943,7 +1086,9 @@ async def _send_local_file( ct = mimetypes.guess_type(fname)[0] or "application/octet-stream" data = p.read_bytes() - return await self._upload_and_send(room_id, data, fname, ct, msgtype, caption, reply_to, metadata, is_voice) + return await self._upload_and_send( + room_id, data, fname, ct, msgtype, caption, reply_to, metadata, is_voice + ) # ------------------------------------------------------------------ # Sync loop @@ -957,8 +1102,22 @@ async def _sync_loop(self) -> None: while not self._closing: try: sync_data = await client.sync( - since=next_batch, timeout=30000, + since=next_batch, + timeout=30000, ) + + # nio returns SyncError objects (not exceptions) for auth + # failures like M_UNKNOWN_TOKEN. Detect and stop immediately. + _sync_msg = getattr(sync_data, "message", None) + if _sync_msg and isinstance(_sync_msg, str): + _lower = _sync_msg.lower() + if "m_unknown_token" in _lower or "unknown_token" in _lower: + logger.error( + "Matrix: permanent auth error from sync: %s — stopping", + _sync_msg, + ) + return + if isinstance(sync_data, dict): # Update joined rooms from sync response. rooms_join = sync_data.get("rooms", {}).get("join", {}) @@ -981,10 +1140,6 @@ async def _sync_loop(self) -> None: except Exception as exc: logger.warning("Matrix: sync event dispatch error: %s", exc) - # Retry any buffered undecrypted events. - if self._pending_megolm: - await self._retry_pending_decryptions() - except asyncio.CancelledError: return except Exception as exc: @@ -992,64 +1147,19 @@ async def _sync_loop(self) -> None: return # Detect permanent auth/permission failures. err_str = str(exc).lower() - if "401" in err_str or "403" in err_str or "unauthorized" in err_str or "forbidden" in err_str: - logger.error("Matrix: permanent auth error: %s — stopping sync", exc) + if ( + "401" in err_str + or "403" in err_str + or "unauthorized" in err_str + or "forbidden" in err_str + ): + logger.error( + "Matrix: permanent auth error: %s — stopping sync", exc + ) return logger.warning("Matrix: sync error: %s — retrying in 5s", exc) await asyncio.sleep(5) - async def _retry_pending_decryptions(self) -> None: - """Retry decrypting buffered encrypted events after new keys arrive.""" - client = self._client - if not client or not self._pending_megolm: - return - crypto = getattr(client, "crypto", None) - if not crypto: - return - - now = time.time() - still_pending: list = [] - - for room_id, event, ts in self._pending_megolm: - # Drop events that have aged past the TTL. - if now - ts > _PENDING_EVENT_TTL: - logger.debug( - "Matrix: dropping expired pending event %s (age %.0fs)", - getattr(event, "event_id", "?"), now - ts, - ) - continue - - try: - decrypted = await crypto.decrypt_megolm_event(event) - except Exception: - still_pending.append((room_id, event, ts)) - continue - - if decrypted is None or decrypted is event: - still_pending.append((room_id, event, ts)) - continue - - logger.info( - "Matrix: decrypted buffered event %s", - getattr(event, "event_id", "?"), - ) - - # Route to the appropriate handler. - # Remove from dedup set so _on_room_message doesn't drop it - # (the encrypted event ID was already registered by _on_encrypted_event). - decrypted_id = str(getattr(decrypted, "event_id", getattr(event, "event_id", ""))) - if decrypted_id: - self._processed_events_set.discard(decrypted_id) - try: - await self._on_room_message(decrypted) - except Exception as exc: - logger.warning( - "Matrix: error processing decrypted event %s: %s", - getattr(event, "event_id", "?"), exc, - ) - - self._pending_megolm = still_pending - # ------------------------------------------------------------------ # Event callbacks # ------------------------------------------------------------------ @@ -1069,7 +1179,11 @@ async def _on_room_message(self, event: Any) -> None: return # Startup grace: ignore old messages from initial sync. - raw_ts = getattr(event, "timestamp", None) or getattr(event, "server_timestamp", None) or 0 + raw_ts = ( + getattr(event, "timestamp", None) + or getattr(event, "server_timestamp", None) + or 0 + ) event_ts = raw_ts / 1000.0 if raw_ts else 0.0 if event_ts and event_ts < self._startup_ts - _STARTUP_GRACE_SECONDS: return @@ -1109,9 +1223,13 @@ async def _on_room_message(self, event: Any) -> None: # Dispatch by msgtype. media_msgtypes = ("m.image", "m.audio", "m.video", "m.file") if msgtype in media_msgtypes: - await self._handle_media_message(room_id, sender, event_id, event_ts, source_content, relates_to, msgtype) + await self._handle_media_message( + room_id, sender, event_id, event_ts, source_content, relates_to, msgtype + ) elif msgtype == "m.text": - await self._handle_text_message(room_id, sender, event_id, event_ts, source_content, relates_to) + await self._handle_text_message( + room_id, sender, event_id, event_ts, source_content, relates_to + ) async def _resolve_message_context( self, @@ -1137,7 +1255,9 @@ async def _resolve_message_context( formatted_body = source_content.get("formatted_body") # m.mentions.user_ids (MSC3952 / Matrix v1.7) — authoritative mention signal. mentions_block = source_content.get("m.mentions") or {} - mention_user_ids = mentions_block.get("user_ids") if isinstance(mentions_block, dict) else None + mention_user_ids = ( + mentions_block.get("user_ids") if isinstance(mentions_block, dict) else None + ) is_mentioned = self._is_bot_mentioned(body, formatted_body, mention_user_ids) # Require-mention gating. @@ -1153,8 +1273,8 @@ async def _resolve_message_context( thread_id = event_id self._threads.mark(thread_id) - # Strip mention from body. - if is_mentioned: + # Strip mention from body (only when mention-gating is active). + if is_mentioned and self._require_mention: body = self._strip_mention(body) # Auto-thread. @@ -1193,7 +1313,12 @@ async def _handle_text_message( return ctx = await self._resolve_message_context( - room_id, sender, event_id, body, source_content, relates_to, + room_id, + sender, + event_id, + body, + source_content, + relates_to, ) if ctx is None: return @@ -1271,7 +1396,9 @@ async def _handle_media_message( if url and url.startswith("mxc://"): http_url = self._mxc_to_http(url) - is_encrypted_media = bool(file_content and isinstance(file_content, dict) and file_content.get("url")) + is_encrypted_media = bool( + file_content and isinstance(file_content, dict) and file_content.get("url") + ) media_type = "application/octet-stream" msg_type = MessageType.DOCUMENT @@ -1295,9 +1422,9 @@ async def _handle_media_message( # Cache media locally when downstream tools need a real file path. cached_path = None - should_cache_locally = ( - msg_type == MessageType.PHOTO or is_voice_message or is_encrypted_media - ) + should_cache_locally = msg_type in ( + MessageType.PHOTO, MessageType.AUDIO, MessageType.VIDEO, MessageType.DOCUMENT, + ) or is_voice_message or is_encrypted_media if should_cache_locally and url: try: file_bytes = await self._client.download_media(ContentURI(url)) @@ -1305,17 +1432,35 @@ async def _handle_media_message( if is_encrypted_media: from mautrix.crypto.attachments import decrypt_attachment - hashes_value = file_content.get("hashes") if isinstance(file_content, dict) else None - hash_value = hashes_value.get("sha256") if isinstance(hashes_value, dict) else None + hashes_value = ( + file_content.get("hashes") + if isinstance(file_content, dict) + else None + ) + hash_value = ( + hashes_value.get("sha256") + if isinstance(hashes_value, dict) + else None + ) - key_value = file_content.get("key") if isinstance(file_content, dict) else None + key_value = ( + file_content.get("key") + if isinstance(file_content, dict) + else None + ) if isinstance(key_value, dict): key_value = key_value.get("k") - iv_value = file_content.get("iv") if isinstance(file_content, dict) else None + iv_value = ( + file_content.get("iv") + if isinstance(file_content, dict) + else None + ) if key_value and hash_value and iv_value: - file_bytes = decrypt_attachment(file_bytes, key_value, hash_value, iv_value) + file_bytes = decrypt_attachment( + file_bytes, key_value, hash_value, iv_value + ) else: logger.warning( "[Matrix] Encrypted media event missing decryption metadata for %s", @@ -1341,25 +1486,46 @@ async def _handle_media_message( cached_path = cache_image_from_bytes(file_bytes, ext=ext) logger.info("[Matrix] Cached user image at %s", cached_path) elif msg_type in (MessageType.AUDIO, MessageType.VOICE): - ext = Path(body or ("voice.ogg" if is_voice_message else "audio.ogg")).suffix or ".ogg" + ext = ( + Path( + body + or ( + "voice.ogg" if is_voice_message else "audio.ogg" + ) + ).suffix + or ".ogg" + ) cached_path = cache_audio_from_bytes(file_bytes, ext=ext) else: filename = body or ( - "video.mp4" if msg_type == MessageType.VIDEO else "document" + "video.mp4" + if msg_type == MessageType.VIDEO + else "document" + ) + cached_path = cache_document_from_bytes( + file_bytes, filename ) - cached_path = cache_document_from_bytes(file_bytes, filename) except Exception as e: logger.warning("[Matrix] Failed to cache media: %s", e) ctx = await self._resolve_message_context( - room_id, sender, event_id, body, source_content, relates_to, + room_id, + sender, + event_id, + body, + source_content, + relates_to, ) if ctx is None: return body, is_dm, chat_type, thread_id, display_name, source = ctx allow_http_fallback = bool(http_url) and not is_encrypted_media - media_urls = [cached_path] if cached_path else ([http_url] if allow_http_fallback else None) + media_urls = ( + [cached_path] + if cached_path + else ([http_url] if allow_http_fallback else None) + ) media_types = [media_type] if media_urls else None msg_event = MessageEvent( @@ -1374,23 +1540,6 @@ async def _handle_media_message( await self.handle_message(msg_event) - async def _on_encrypted_event(self, event: Any) -> None: - """Handle encrypted events that could not be auto-decrypted.""" - room_id = str(getattr(event, "room_id", "")) - event_id = str(getattr(event, "event_id", "")) - - if self._is_duplicate_event(event_id): - return - - logger.warning( - "Matrix: could not decrypt event %s in %s — buffering for retry", - event_id, room_id, - ) - - self._pending_megolm.append((room_id, event, time.time())) - if len(self._pending_megolm) > _MAX_PENDING_EVENTS: - self._pending_megolm = self._pending_megolm[-_MAX_PENDING_EVENTS:] - async def _on_invite(self, event: Any) -> None: """Auto-join rooms when invited.""" @@ -1413,7 +1562,10 @@ async def _on_invite(self, event: Any) -> None: # ------------------------------------------------------------------ async def _send_reaction( - self, room_id: str, event_id: str, emoji: str, + self, + room_id: str, + event_id: str, + emoji: str, ) -> Optional[str]: """Send an emoji reaction to a message in a room. Returns the reaction event_id on success, None on failure. @@ -1430,7 +1582,9 @@ async def _send_reaction( } try: resp_event_id = await self._client.send_message_event( - RoomID(room_id), EventType.REACTION, content, + RoomID(room_id), + EventType.REACTION, + content, ) logger.debug("Matrix: sent reaction %s to %s", emoji, event_id) return str(resp_event_id) @@ -1439,7 +1593,10 @@ async def _send_reaction( return None async def _redact_reaction( - self, room_id: str, reaction_event_id: str, reason: str = "", + self, + room_id: str, + reaction_event_id: str, + reason: str = "", ) -> bool: """Remove a reaction by redacting its event.""" return await self.redact_message(room_id, reaction_event_id, reason) @@ -1456,7 +1613,9 @@ async def on_processing_start(self, event: MessageEvent) -> None: self._pending_reactions[(room_id, msg_id)] = reaction_event_id async def on_processing_complete( - self, event: MessageEvent, outcome: ProcessingOutcome, + self, + event: MessageEvent, + outcome: ProcessingOutcome, ) -> None: """Replace eyes with checkmark (success) or cross (failure).""" if not self._reactions_enabled: @@ -1490,7 +1649,11 @@ async def _on_reaction(self, event: Any) -> None: room_id = str(getattr(event, "room_id", "")) content = getattr(event, "content", None) if content: - relates_to = content.get("m.relates_to", {}) if isinstance(content, dict) else getattr(content, "relates_to", {}) + relates_to = ( + content.get("m.relates_to", {}) + if isinstance(content, dict) + else getattr(content, "relates_to", {}) + ) reacts_to = "" key = "" if isinstance(relates_to, dict): @@ -1501,7 +1664,10 @@ async def _on_reaction(self, event: Any) -> None: key = str(getattr(relates_to, "key", "")) logger.info( "Matrix: reaction %s from %s on %s in %s", - key, sender, reacts_to, room_id, + key, + sender, + reacts_to, + room_id, ) # ------------------------------------------------------------------ @@ -1511,10 +1677,15 @@ async def _on_reaction(self, event: Any) -> None: def _text_batch_key(self, event: MessageEvent) -> str: """Session-scoped key for text message batching.""" from gateway.session import build_session_key + return build_session_key( event.source, - group_sessions_per_user=self.config.extra.get("group_sessions_per_user", True), - thread_sessions_per_user=self.config.extra.get("thread_sessions_per_user", False), + group_sessions_per_user=self.config.extra.get( + "group_sessions_per_user", True + ), + thread_sessions_per_user=self.config.extra.get( + "thread_sessions_per_user", False + ), ) def _enqueue_text_event(self, event: MessageEvent) -> None: @@ -1527,7 +1698,9 @@ def _enqueue_text_event(self, event: MessageEvent) -> None: self._pending_text_batches[key] = event else: if event.text: - existing.text = f"{existing.text}\n{event.text}" if existing.text else event.text + existing.text = ( + f"{existing.text}\n{event.text}" if existing.text else event.text + ) existing._last_chunk_len = chunk_len # type: ignore[attr-defined] if event.media_urls: existing.media_urls.extend(event.media_urls) @@ -1556,7 +1729,8 @@ async def _flush_text_batch(self, key: str) -> None: return logger.info( "[Matrix] Flushing text batch %s (%d chars)", - key, len(event.text or ""), + key, + len(event.text or ""), ) await self.handle_message(event) finally: @@ -1569,11 +1743,13 @@ async def _flush_text_batch(self, key: str) -> None: def _background_read_receipt(self, room_id: str, event_id: str) -> None: """Fire-and-forget read receipt with error logging.""" + async def _send() -> None: try: await self.send_read_receipt(room_id, event_id) except Exception as exc: # pragma: no cover — defensive logger.debug("Matrix: background read receipt failed: %s", exc) + asyncio.ensure_future(_send()) async def send_read_receipt(self, room_id: str, event_id: str) -> bool: @@ -1581,11 +1757,21 @@ async def send_read_receipt(self, room_id: str, event_id: str) -> bool: if not self._client: return False try: - await self._client.set_read_markers( - RoomID(room_id), - fully_read_event=EventID(event_id), - read_receipt=EventID(event_id), - ) + room = RoomID(room_id) + event = EventID(event_id) + if hasattr(self._client, "set_fully_read_marker"): + await self._client.set_fully_read_marker(room, event, event) + elif hasattr(self._client, "send_receipt"): + await self._client.send_receipt(room, event) + elif hasattr(self._client, "set_read_markers"): + await self._client.set_read_markers( + room, + fully_read_event=event, + read_receipt=event, + ) + else: + logger.debug("Matrix: client has no read receipt method") + return False logger.debug("Matrix: sent read receipt for %s in %s", event_id, room_id) return True except Exception as exc: @@ -1597,14 +1783,19 @@ async def send_read_receipt(self, room_id: str, event_id: str) -> bool: # ------------------------------------------------------------------ async def redact_message( - self, room_id: str, event_id: str, reason: str = "", + self, + room_id: str, + event_id: str, + reason: str = "", ) -> bool: """Redact (delete) a message or event from a room.""" if not self._client: return False try: await self._client.redact( - RoomID(room_id), EventID(event_id), reason=reason or None, + RoomID(room_id), + EventID(event_id), + reason=reason or None, ) logger.info("Matrix: redacted %s in %s", event_id, room_id) return True @@ -1612,52 +1803,6 @@ async def redact_message( logger.warning("Matrix: redact error: %s", exc) return False - # ------------------------------------------------------------------ - # Room history - # ------------------------------------------------------------------ - - async def fetch_room_history( - self, - room_id: str, - limit: int = 50, - start: str = "", - ) -> list: - """Fetch recent messages from a room.""" - if not self._client: - return [] - try: - resp = await self._client.get_messages( - RoomID(room_id), - direction=PaginationDirection.BACKWARD, - from_token=SyncToken(start) if start else None, - limit=limit, - ) - except Exception as exc: - logger.warning("Matrix: get_messages failed for %s: %s", room_id, exc) - return [] - - if not resp: - return [] - - events = getattr(resp, "chunk", []) or (resp.get("chunk", []) if isinstance(resp, dict) else []) - messages = [] - for event in reversed(events): - body = "" - content = getattr(event, "content", None) - if content: - if hasattr(content, "body"): - body = content.body or "" - elif isinstance(content, dict): - body = content.get("body", "") - messages.append({ - "event_id": str(getattr(event, "event_id", "")), - "sender": str(getattr(event, "sender", "")), - "body": body, - "timestamp": getattr(event, "timestamp", 0) or getattr(event, "server_timestamp", 0), - "type": type(event).__name__, - }) - return messages - # ------------------------------------------------------------------ # Room creation & management # ------------------------------------------------------------------ @@ -1741,7 +1886,10 @@ async def set_presence(self, state: str = "online", status_msg: str = "") -> boo # ------------------------------------------------------------------ async def _send_simple_message( - self, chat_id: str, text: str, msgtype: str, + self, + chat_id: str, + text: str, + msgtype: str, ) -> SendResult: """Send a simple message (emote, notice) with optional HTML formatting.""" if not self._client or not text: @@ -1755,24 +1903,14 @@ async def _send_simple_message( try: event_id = await self._client.send_message_event( - RoomID(chat_id), EventType.ROOM_MESSAGE, msg_content, + RoomID(chat_id), + EventType.ROOM_MESSAGE, + msg_content, ) return SendResult(success=True, message_id=str(event_id)) except Exception as exc: return SendResult(success=False, error=str(exc)) - async def send_emote( - self, chat_id: str, text: str, metadata: Optional[Dict[str, Any]] = None, - ) -> SendResult: - """Send an emote message (/me style action).""" - return await self._send_simple_message(chat_id, text, "m.emote") - - async def send_notice( - self, chat_id: str, text: str, metadata: Optional[Dict[str, Any]] = None, - ) -> SendResult: - """Send a notice message (bot-appropriate, non-alerting).""" - return await self._send_simple_message(chat_id, text, "m.notice") - # ------------------------------------------------------------------ # Helpers # ------------------------------------------------------------------ @@ -1782,7 +1920,9 @@ async def _is_dm_room(self, room_id: str) -> bool: if self._dm_rooms.get(room_id, False): return True # Fallback: check member count via state store. - state_store = getattr(self._client, "state_store", None) if self._client else None + state_store = ( + getattr(self._client, "state_store", None) if self._client else None + ) if state_store: try: members = await state_store.get_members(room_id) @@ -1816,10 +1956,7 @@ async def _refresh_dm_cache(self) -> None: if isinstance(rooms, list): dm_room_ids.update(str(r) for r in rooms) - self._dm_rooms = { - rid: (rid in dm_room_ids) - for rid in self._joined_rooms - } + self._dm_rooms = {rid: (rid in dm_room_ids) for rid in self._joined_rooms} # ------------------------------------------------------------------ # Mention detection helpers @@ -1849,7 +1986,9 @@ def _is_bot_mentioned( return True if self._user_id and ":" in self._user_id: localpart = self._user_id.split(":")[0].lstrip("@") - if localpart and re.search(r'\b' + re.escape(localpart) + r'\b', body, re.IGNORECASE): + if localpart and re.search( + r"\b" + re.escape(localpart) + r"\b", body, re.IGNORECASE + ): return True if formatted_body and self._user_id: if f"matrix.to/#/{self._user_id}" in formatted_body: @@ -1857,18 +1996,20 @@ def _is_bot_mentioned( return False def _strip_mention(self, body: str) -> str: - """Remove bot mention from message body.""" + """Strip the bot's full MXID (``@user:server``) from *body*. + + The bare localpart is intentionally *not* stripped — it would + mangle file paths like ``/home/hermes/media/file.png``. + """ if self._user_id: body = body.replace(self._user_id, "") - if self._user_id and ":" in self._user_id: - localpart = self._user_id.split(":")[0].lstrip("@") - if localpart: - body = re.sub(r'\b' + re.escape(localpart) + r'\b', '', body, flags=re.IGNORECASE) return body.strip() async def _get_display_name(self, room_id: str, user_id: str) -> str: """Get a user's display name in a room, falling back to user_id.""" - state_store = getattr(self._client, "state_store", None) if self._client else None + state_store = ( + getattr(self._client, "state_store", None) if self._client else None + ) if state_store: try: member = await state_store.get_member(room_id, user_id) @@ -1956,9 +2097,7 @@ def _protect_html(html_fragment: str) -> str: # Inline code: `code` result = re.sub( r"`([^`\n]+)`", - lambda m: _protect_html( - f"{_html_escape(m.group(1))}" - ), + lambda m: _protect_html(f"{_html_escape(m.group(1))}"), result, ) @@ -2003,11 +2142,18 @@ def _protect_html(html_fragment: str) -> str: continue # Blockquote - if line.startswith("> ") or line == ">" or line.startswith("> ") or line == ">": + if ( + line.startswith("> ") + or line == ">" + or line.startswith("> ") + or line == ">" + ): bq_lines = [] while i < len(lines) and ( - lines[i].startswith("> ") or lines[i] == ">" - or lines[i].startswith("> ") or lines[i] == ">" + lines[i].startswith("> ") + or lines[i] == ">" + or lines[i].startswith("> ") + or lines[i] == ">" ): ln = lines[i] if ln.startswith("> "): @@ -2048,13 +2194,19 @@ def _protect_html(html_fragment: str) -> str: result = "\n".join(out_lines) # Inline transforms. - result = re.sub(r"\*\*(.+?)\*\*", r"\1", result, flags=re.DOTALL) + result = re.sub( + r"\*\*(.+?)\*\*", r"\1", result, flags=re.DOTALL + ) result = re.sub(r"__(.+?)__", r"\1", result, flags=re.DOTALL) result = re.sub(r"\*(.+?)\*", r"\1", result, flags=re.DOTALL) - result = re.sub(r"(?\1", result, flags=re.DOTALL) + result = re.sub( + r"(?\1", result, flags=re.DOTALL + ) result = re.sub(r"~~(.+?)~~", r"\1", result, flags=re.DOTALL) result = re.sub(r"\n", "
\n", result) - result = re.sub(r"
\n(\n()
", r"\1", result) # Restore protected regions. diff --git a/gateway/platforms/mattermost.py b/gateway/platforms/mattermost.py index 23a86f02b124..0e6c9631d73e 100644 --- a/gateway/platforms/mattermost.py +++ b/gateway/platforms/mattermost.py @@ -304,7 +304,7 @@ async def send_typing( ) async def edit_message( - self, chat_id: str, message_id: str, content: str + self, chat_id: str, message_id: str, content: str, *, finalize: bool = False ) -> SendResult: """Edit an existing post.""" formatted = self.format_message(content) @@ -410,7 +410,6 @@ async def _send_url_as_file( logger.warning("Mattermost: blocked unsafe URL (SSRF protection)") return await self.send(chat_id, f"{caption or ''}\n{url}".strip(), reply_to) - import asyncio import aiohttp last_exc = None @@ -718,6 +717,12 @@ async def _handle_ws_event(self, event: Dict[str, Any]) -> None: thread_id=thread_id, ) + # Per-channel ephemeral prompt + from gateway.platforms.base import resolve_channel_prompt + _channel_prompt = resolve_channel_prompt( + self.config.extra, channel_id, None, + ) + msg_event = MessageEvent( text=message_text, message_type=msg_type, @@ -726,6 +731,7 @@ async def _handle_ws_event(self, event: Dict[str, Any]) -> None: message_id=post_id, media_urls=media_urls if media_urls else None, media_types=media_types if media_types else None, + channel_prompt=_channel_prompt, ) await self.handle_message(msg_event) diff --git a/gateway/platforms/qqbot/__init__.py b/gateway/platforms/qqbot/__init__.py new file mode 100644 index 000000000000..130269b5f26f --- /dev/null +++ b/gateway/platforms/qqbot/__init__.py @@ -0,0 +1,55 @@ +""" +QQBot platform package. + +Re-exports the main adapter symbols from ``adapter.py`` (the original +``qqbot.py``) so that **all existing import paths remain unchanged**:: + + from gateway.platforms.qqbot import QQAdapter # works + from gateway.platforms.qqbot import check_qq_requirements # works + +New modules: + - ``constants`` — shared constants (API URLs, timeouts, message types) + - ``utils`` — User-Agent builder, config helpers + - ``crypto`` — AES-256-GCM key generation and decryption + - ``onboard`` — QR-code scan-to-configure flow +""" + +# -- Adapter (original qqbot.py) ------------------------------------------ +from .adapter import ( # noqa: F401 + QQAdapter, + QQCloseError, + check_qq_requirements, + _coerce_list, + _ssrf_redirect_guard, +) + +# -- Onboard (QR-code scan-to-configure) ----------------------------------- +from .onboard import ( # noqa: F401 + BindStatus, + build_connect_url, + qr_register, +) +from .crypto import decrypt_secret, generate_bind_key # noqa: F401 + +# -- Utils ----------------------------------------------------------------- +from .utils import build_user_agent, get_api_headers, coerce_list # noqa: F401 + +__all__ = [ + # adapter + "QQAdapter", + "QQCloseError", + "check_qq_requirements", + "_coerce_list", + "_ssrf_redirect_guard", + # onboard + "BindStatus", + "build_connect_url", + "qr_register", + # crypto + "decrypt_secret", + "generate_bind_key", + # utils + "build_user_agent", + "get_api_headers", + "coerce_list", +] diff --git a/gateway/platforms/qqbot/adapter.py b/gateway/platforms/qqbot/adapter.py new file mode 100644 index 000000000000..932846458412 --- /dev/null +++ b/gateway/platforms/qqbot/adapter.py @@ -0,0 +1,2380 @@ +""" +QQ Bot platform adapter using the Official QQ Bot API (v2). + +Connects to the QQ Bot WebSocket Gateway for inbound events and uses the +REST API (``api.sgroup.qq.com``) for outbound messages and media uploads. + +Configuration in config.yaml: + platforms: + qq: + enabled: true + extra: + app_id: "your-app-id" # or QQ_APP_ID env var + client_secret: "your-secret" # or QQ_CLIENT_SECRET env var + markdown_support: true # enable QQ markdown (msg_type 2) + dm_policy: "open" # open | allowlist | disabled + allow_from: ["openid_1"] + group_policy: "open" # open | allowlist | disabled + group_allow_from: ["group_openid_1"] + stt: # Voice-to-text config (optional) + provider: "zai" # zai (GLM-ASR), openai (Whisper), etc. + baseUrl: "https://open.bigmodel.cn/api/coding/paas/v4" + apiKey: "your-stt-api-key" # or set QQ_STT_API_KEY env var + model: "glm-asr" # glm-asr, whisper-1, etc. + + Voice transcription priority: + 1. QQ's built-in ``asr_refer_text`` (Tencent ASR — free, always tried first) + 2. Configured STT provider via ``stt`` config or ``QQ_STT_*`` env vars + +Reference: https://bot.q.qq.com/wiki/develop/api-v2/ +""" + +from __future__ import annotations + +import asyncio +import base64 +import json +import logging +import mimetypes +import os +import time +import uuid +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple +from urllib.parse import urlparse + +try: + import aiohttp + + AIOHTTP_AVAILABLE = True +except ImportError: + AIOHTTP_AVAILABLE = False + aiohttp = None # type: ignore[assignment] + +try: + import httpx + + HTTPX_AVAILABLE = True +except ImportError: + HTTPX_AVAILABLE = False + httpx = None # type: ignore[assignment] + +from gateway.config import Platform, PlatformConfig +from gateway.platforms.base import ( + BasePlatformAdapter, + MessageEvent, + MessageType, + SendResult, + _ssrf_redirect_guard, + cache_document_from_bytes, + cache_image_from_bytes, +) +from gateway.platforms.helpers import strip_markdown + +logger = logging.getLogger(__name__) + + +class QQCloseError(Exception): + """Raised when QQ WebSocket closes with a specific code. + + Carries the close code and reason for proper handling in the reconnect loop. + """ + + def __init__(self, code, reason=""): + self.code = int(code) if code else None + self.reason = str(reason) if reason else "" + super().__init__(f"WebSocket closed (code={self.code}, reason={self.reason})") + + +# --------------------------------------------------------------------------- +# Constants — imported from the shared constants module. +# --------------------------------------------------------------------------- + +from gateway.platforms.qqbot.constants import ( + API_BASE, + TOKEN_URL, + GATEWAY_URL_PATH, + DEFAULT_API_TIMEOUT, + FILE_UPLOAD_TIMEOUT, + CONNECT_TIMEOUT_SECONDS, + RECONNECT_BACKOFF, + MAX_RECONNECT_ATTEMPTS, + RATE_LIMIT_DELAY, + QUICK_DISCONNECT_THRESHOLD, + MAX_QUICK_DISCONNECT_COUNT, + MAX_MESSAGE_LENGTH, + DEDUP_WINDOW_SECONDS, + DEDUP_MAX_SIZE, + MSG_TYPE_TEXT, + MSG_TYPE_MARKDOWN, + MSG_TYPE_MEDIA, + MSG_TYPE_INPUT_NOTIFY, + MEDIA_TYPE_IMAGE, + MEDIA_TYPE_VIDEO, + MEDIA_TYPE_VOICE, + MEDIA_TYPE_FILE, +) +from gateway.platforms.qqbot.utils import ( + coerce_list as _coerce_list_impl, + build_user_agent, +) + + +def check_qq_requirements() -> bool: + """Check if QQ runtime dependencies are available.""" + return AIOHTTP_AVAILABLE and HTTPX_AVAILABLE + + +def _coerce_list(value: Any) -> List[str]: + """Coerce config values into a trimmed string list.""" + return _coerce_list_impl(value) + + +# --------------------------------------------------------------------------- +# QQAdapter +# --------------------------------------------------------------------------- + + +class QQAdapter(BasePlatformAdapter): + """QQ Bot adapter backed by the official QQ Bot WebSocket Gateway + REST API.""" + + # QQ Bot API does not support editing sent messages. + SUPPORTS_MESSAGE_EDITING = False + MAX_MESSAGE_LENGTH = MAX_MESSAGE_LENGTH + _TYPING_INPUT_SECONDS = 60 # input_notify duration reported to QQ + _TYPING_DEBOUNCE_SECONDS = 50 # refresh before it expires + + @property + def _log_tag(self) -> str: + """Log prefix including app_id for multi-instance disambiguation.""" + app_id = getattr(self, "_app_id", None) + if app_id: + return f"QQBot:{app_id}" + return "QQBot" + + def _fail_pending(self, reason: str) -> None: + """Fail all pending response futures.""" + for fut in self._pending_responses.values(): + if not fut.done(): + fut.set_exception(RuntimeError(reason)) + self._pending_responses.clear() + + def __init__(self, config: PlatformConfig): + super().__init__(config, Platform.QQBOT) + + extra = config.extra or {} + self._app_id = str(extra.get("app_id") or os.getenv("QQ_APP_ID", "")).strip() + self._client_secret = str( + extra.get("client_secret") or os.getenv("QQ_CLIENT_SECRET", "") + ).strip() + self._markdown_support = bool(extra.get("markdown_support", True)) + + # Auth/ACL policies + self._dm_policy = str(extra.get("dm_policy", "open")).strip().lower() + self._allow_from = _coerce_list( + extra.get("allow_from") or extra.get("allowFrom") + ) + self._group_policy = str(extra.get("group_policy", "open")).strip().lower() + self._group_allow_from = _coerce_list( + extra.get("group_allow_from") or extra.get("groupAllowFrom") + ) + + # Connection state + self._session: Optional[aiohttp.ClientSession] = None + self._ws: Optional[aiohttp.ClientWebSocketResponse] = None + self._http_client: Optional[httpx.AsyncClient] = None + self._listen_task: Optional[asyncio.Task] = None + self._heartbeat_task: Optional[asyncio.Task] = None + self._heartbeat_interval: float = 30.0 # seconds, updated by Hello + self._session_id: Optional[str] = None + self._last_seq: Optional[int] = None + self._chat_type_map: Dict[str, str] = {} # chat_id → "c2c"|"group"|"guild"|"dm" + + # Request/response correlation + self._pending_responses: Dict[str, asyncio.Future] = {} + self._seen_messages: Dict[str, float] = {} + + # Last inbound message ID per chat — used by send_typing + self._last_msg_id: Dict[str, str] = {} + # Typing debounce: chat_id → last send_typing timestamp + self._typing_sent_at: Dict[str, float] = {} + + # Token cache + self._access_token: Optional[str] = None + self._token_expires_at: float = 0.0 + self._token_lock = asyncio.Lock() + + # Upload cache: content_hash -> {file_info, file_uuid, expires_at} + self._upload_cache: Dict[str, Dict[str, Any]] = {} + + # ------------------------------------------------------------------ + # Properties + # ------------------------------------------------------------------ + + @property + def name(self) -> str: + return "QQBot" + + # ------------------------------------------------------------------ + # Connection lifecycle + # ------------------------------------------------------------------ + + async def connect(self) -> bool: + """Authenticate, obtain gateway URL, and open the WebSocket.""" + if not AIOHTTP_AVAILABLE: + message = "QQ startup failed: aiohttp not installed" + self._set_fatal_error("qq_missing_dependency", message, retryable=True) + logger.warning("[%s] %s. Run: pip install aiohttp", self._log_tag, message) + return False + if not HTTPX_AVAILABLE: + message = "QQ startup failed: httpx not installed" + self._set_fatal_error("qq_missing_dependency", message, retryable=True) + logger.warning("[%s] %s. Run: pip install httpx", self._log_tag, message) + return False + if not self._app_id or not self._client_secret: + message = "QQ startup failed: QQ_APP_ID and QQ_CLIENT_SECRET are required" + self._set_fatal_error("qq_missing_credentials", message, retryable=True) + logger.warning("[%s] %s", self._log_tag, message) + return False + + # Prevent duplicate connections with the same credentials + if not self._acquire_platform_lock("qqbot-appid", self._app_id, "QQBot app ID"): + return False + + try: + self._http_client = httpx.AsyncClient( + timeout=30.0, + follow_redirects=True, + event_hooks={"response": [_ssrf_redirect_guard]}, + ) + + # 1. Get access token + await self._ensure_token() + + # 2. Get WebSocket gateway URL + gateway_url = await self._get_gateway_url() + logger.info("[%s] Gateway URL: %s", self._log_tag, gateway_url) + + # 3. Open WebSocket + await self._open_ws(gateway_url) + + # 4. Start listeners + self._listen_task = asyncio.create_task(self._listen_loop()) + self._heartbeat_task = asyncio.create_task(self._heartbeat_loop()) + self._mark_connected() + logger.info("[%s] Connected", self._log_tag) + return True + except Exception as exc: + message = f"QQ startup failed: {exc}" + self._set_fatal_error("qq_connect_error", message, retryable=True) + logger.error("[%s] %s", self._log_tag, message, exc_info=True) + await self._cleanup() + self._release_platform_lock() + return False + + async def disconnect(self) -> None: + """Close all connections and stop listeners.""" + self._running = False + self._mark_disconnected() + + if self._listen_task: + self._listen_task.cancel() + try: + await self._listen_task + except asyncio.CancelledError: + pass + self._listen_task = None + + if self._heartbeat_task: + self._heartbeat_task.cancel() + try: + await self._heartbeat_task + except asyncio.CancelledError: + pass + self._heartbeat_task = None + + await self._cleanup() + self._release_platform_lock() + logger.info("[%s] Disconnected", self._log_tag) + + async def _cleanup(self) -> None: + """Close WebSocket, HTTP session, and client.""" + if self._ws and not self._ws.closed: + await self._ws.close() + self._ws = None + + if self._session and not self._session.closed: + await self._session.close() + self._session = None + + if self._http_client: + await self._http_client.aclose() + self._http_client = None + + # Fail pending + for fut in self._pending_responses.values(): + if not fut.done(): + fut.set_exception(RuntimeError("Disconnected")) + self._pending_responses.clear() + + # ------------------------------------------------------------------ + # Token management + # ------------------------------------------------------------------ + + async def _ensure_token(self) -> str: + """Return a valid access token, refreshing if needed (with singleflight).""" + if self._access_token and time.time() < self._token_expires_at - 60: + return self._access_token + + async with self._token_lock: + # Double-check after acquiring lock + if self._access_token and time.time() < self._token_expires_at - 60: + return self._access_token + + try: + resp = await self._http_client.post( + TOKEN_URL, + json={"appId": self._app_id, "clientSecret": self._client_secret}, + timeout=DEFAULT_API_TIMEOUT, + ) + resp.raise_for_status() + data = resp.json() + except Exception as exc: + raise RuntimeError(f"Failed to get QQ Bot access token: {exc}") from exc + + token = data.get("access_token") + if not token: + raise RuntimeError( + f"QQ Bot token response missing access_token: {data}" + ) + + expires_in = int(data.get("expires_in", 7200)) + self._access_token = token + self._token_expires_at = time.time() + expires_in + logger.info( + "[%s] Access token refreshed, expires in %ds", self._log_tag, expires_in + ) + return self._access_token + + async def _get_gateway_url(self) -> str: + """Fetch the WebSocket gateway URL from the REST API.""" + token = await self._ensure_token() + try: + resp = await self._http_client.get( + f"{API_BASE}{GATEWAY_URL_PATH}", + headers={ + "Authorization": f"QQBot {token}", + "User-Agent": build_user_agent(), + }, + timeout=DEFAULT_API_TIMEOUT, + ) + resp.raise_for_status() + data = resp.json() + except Exception as exc: + raise RuntimeError(f"Failed to get QQ Bot gateway URL: {exc}") from exc + + url = data.get("url") + if not url: + raise RuntimeError(f"QQ Bot gateway response missing url: {data}") + return url + + # ------------------------------------------------------------------ + # WebSocket lifecycle + # ------------------------------------------------------------------ + + async def _open_ws(self, gateway_url: str) -> None: + """Open a WebSocket connection to the QQ Bot gateway.""" + # Only clean up WebSocket resources — keep _http_client alive for REST API calls. + if self._ws and not self._ws.closed: + await self._ws.close() + self._ws = None + if self._session and not self._session.closed: + await self._session.close() + self._session = None + + self._session = aiohttp.ClientSession() + self._ws = await self._session.ws_connect( + gateway_url, + headers={ + "User-Agent": build_user_agent(), + }, + timeout=CONNECT_TIMEOUT_SECONDS, + ) + logger.info("[%s] WebSocket connected to %s", self._log_tag, gateway_url) + + async def _listen_loop(self) -> None: + """Read WebSocket events and reconnect on errors. + + Close code handling follows the OpenClaw qqbot reference implementation: + 4004 → invalid token, refresh and reconnect + 4006/4007/4009 → session invalid, clear session and re-identify + 4008 → rate limited, back off 60s + 4914 → bot offline/sandbox, stop reconnecting + 4915 → bot banned, stop reconnecting + """ + backoff_idx = 0 + connect_time = 0.0 + quick_disconnect_count = 0 + + while self._running: + try: + connect_time = time.monotonic() + await self._read_events() + backoff_idx = 0 + quick_disconnect_count = 0 + except asyncio.CancelledError: + return + except QQCloseError as exc: + if not self._running: + return + + code = exc.code + logger.warning( + "[%s] WebSocket closed: code=%s reason=%s", + self._log_tag, + code, + exc.reason, + ) + + # Quick disconnect detection (permission issues, misconfiguration) + duration = time.monotonic() - connect_time + if duration < QUICK_DISCONNECT_THRESHOLD and connect_time > 0: + quick_disconnect_count += 1 + logger.info( + "[%s] Quick disconnect (%.1fs), count: %d", + self._log_tag, + duration, + quick_disconnect_count, + ) + if quick_disconnect_count >= MAX_QUICK_DISCONNECT_COUNT: + logger.error( + "[%s] Too many quick disconnects. " + "Check: 1) AppID/Secret correct 2) Bot permissions on QQ Open Platform", + self._log_tag, + ) + self._set_fatal_error( + "qq_quick_disconnect", + "Too many quick disconnects — check bot permissions", + retryable=True, + ) + return + else: + quick_disconnect_count = 0 + + self._mark_disconnected() + self._fail_pending("Connection closed") + + # Stop reconnecting for fatal codes + if code in (4914, 4915): + desc = "offline/sandbox-only" if code == 4914 else "banned" + logger.error( + "[%s] Bot is %s. Check QQ Open Platform.", self._log_tag, desc + ) + self._set_fatal_error( + f"qq_{desc}", f"Bot is {desc}", retryable=False + ) + return + + # Rate limited + if code == 4008: + logger.info( + "[%s] Rate limited (4008), waiting %ds", + self._log_tag, + RATE_LIMIT_DELAY, + ) + if backoff_idx >= MAX_RECONNECT_ATTEMPTS: + return + await asyncio.sleep(RATE_LIMIT_DELAY) + if await self._reconnect(backoff_idx): + backoff_idx = 0 + quick_disconnect_count = 0 + else: + backoff_idx += 1 + continue + + # Token invalid → clear cached token so _ensure_token() refreshes + if code == 4004: + logger.info( + "[%s] Invalid token (4004), will refresh and reconnect", + self._log_tag, + ) + self._access_token = None + self._token_expires_at = 0.0 + + # Session invalid → clear session, will re-identify on next Hello + if code in ( + 4006, + 4007, + 4009, + 4900, + 4901, + 4902, + 4903, + 4904, + 4905, + 4906, + 4907, + 4908, + 4909, + 4910, + 4911, + 4912, + 4913, + ): + logger.info( + "[%s] Session error (%d), clearing session for re-identify", + self._log_tag, + code, + ) + self._session_id = None + self._last_seq = None + + if await self._reconnect(backoff_idx): + backoff_idx = 0 + quick_disconnect_count = 0 + else: + backoff_idx += 1 + if backoff_idx >= MAX_RECONNECT_ATTEMPTS: + logger.error("[%s] Max reconnect attempts reached (QQCloseError)", self._log_tag) + return + + except Exception as exc: + if not self._running: + return + logger.warning("[%s] WebSocket error: %s", self._log_tag, exc) + self._mark_disconnected() + self._fail_pending("Connection interrupted") + + if backoff_idx >= MAX_RECONNECT_ATTEMPTS: + logger.error("[%s] Max reconnect attempts reached", self._log_tag) + return + + if await self._reconnect(backoff_idx): + backoff_idx = 0 + quick_disconnect_count = 0 + else: + backoff_idx += 1 + + async def _reconnect(self, backoff_idx: int) -> bool: + """Attempt to reconnect the WebSocket. Returns True on success.""" + delay = RECONNECT_BACKOFF[min(backoff_idx, len(RECONNECT_BACKOFF) - 1)] + logger.info( + "[%s] Reconnecting in %ds (attempt %d)...", + self._log_tag, + delay, + backoff_idx + 1, + ) + await asyncio.sleep(delay) + + self._heartbeat_interval = 30.0 # reset until Hello + try: + await self._ensure_token() + gateway_url = await self._get_gateway_url() + await self._open_ws(gateway_url) + self._mark_connected() + logger.info("[%s] Reconnected", self._log_tag) + return True + except Exception as exc: + logger.warning("[%s] Reconnect failed: %s", self._log_tag, exc) + return False + + async def _read_events(self) -> None: + """Read WebSocket frames until connection closes.""" + if not self._ws: + raise RuntimeError("WebSocket not connected") + + while self._running and self._ws and not self._ws.closed: + msg = await self._ws.receive() + if msg.type == aiohttp.WSMsgType.TEXT: + payload = self._parse_json(msg.data) + if payload: + self._dispatch_payload(payload) + elif msg.type in (aiohttp.WSMsgType.PING,): + # aiohttp auto-replies with PONG + pass + elif msg.type == aiohttp.WSMsgType.CLOSE: + raise QQCloseError(msg.data, msg.extra) + elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR): + raise RuntimeError("WebSocket closed") + + async def _heartbeat_loop(self) -> None: + """Send periodic heartbeats (QQ Gateway expects op 1 heartbeat with latest seq). + + The interval is set from the Hello (op 10) event's heartbeat_interval. + QQ's default is ~41s; we send at 80% of the interval to stay safe. + """ + try: + while self._running: + await asyncio.sleep(self._heartbeat_interval) + if not self._ws or self._ws.closed: + continue + try: + # d should be the latest sequence number received, or null + await self._ws.send_json({"op": 1, "d": self._last_seq}) + except Exception as exc: + logger.debug("[%s] Heartbeat failed: %s", self._log_tag, exc) + except asyncio.CancelledError: + pass + + async def _send_identify(self) -> None: + """Send op 2 Identify to authenticate the WebSocket connection. + + After receiving op 10 Hello, the client must send op 2 Identify with + the bot token and intents. On success the server replies with a + READY dispatch event. + + Reference: https://bot.q.qq.com/wiki/develop/api-v2/dev-prepare/interface-framework/reference.html + """ + token = await self._ensure_token() + identify_payload = { + "op": 2, + "d": { + "token": f"QQBot {token}", + "intents": (1 << 25) + | (1 << 30) + | ( + 1 << 12 + ), # C2C_GROUP_AT_MESSAGES + PUBLIC_GUILD_MESSAGES + DIRECT_MESSAGE + "shard": [0, 1], + "properties": { + "$os": "macOS", + "$browser": "hermes-agent", + "$device": "hermes-agent", + }, + }, + } + try: + if self._ws and not self._ws.closed: + await self._ws.send_json(identify_payload) + logger.info("[%s] Identify sent", self._log_tag) + else: + logger.warning( + "[%s] Cannot send Identify: WebSocket not connected", self._log_tag + ) + except Exception as exc: + logger.error("[%s] Failed to send Identify: %s", self._log_tag, exc) + + async def _send_resume(self) -> None: + """Send op 6 Resume to re-authenticate after a reconnection. + + Reference: https://bot.q.qq.com/wiki/develop/api-v2/dev-prepare/interface-framework/reference.html + """ + token = await self._ensure_token() + resume_payload = { + "op": 6, + "d": { + "token": f"QQBot {token}", + "session_id": self._session_id, + "seq": self._last_seq, + }, + } + try: + if self._ws and not self._ws.closed: + await self._ws.send_json(resume_payload) + logger.info( + "[%s] Resume sent (session_id=%s, seq=%s)", + self._log_tag, + self._session_id, + self._last_seq, + ) + else: + logger.warning( + "[%s] Cannot send Resume: WebSocket not connected", self._log_tag + ) + except Exception as exc: + logger.error("[%s] Failed to send Resume: %s", self._log_tag, exc) + # If resume fails, clear session and fall back to identify on next Hello + self._session_id = None + self._last_seq = None + + @staticmethod + def _create_task(coro): + """Schedule a coroutine, silently skipping if no event loop is running. + + This avoids ``RuntimeError: no running event loop`` when tests call + ``_dispatch_payload`` synchronously outside of ``asyncio.run()``. + """ + try: + loop = asyncio.get_running_loop() + return loop.create_task(coro) + except RuntimeError: + return None + + def _dispatch_payload(self, payload: Dict[str, Any]) -> None: + """Route inbound WebSocket payloads (dispatch synchronously, spawn async handlers).""" + op = payload.get("op") + t = payload.get("t") + s = payload.get("s") + d = payload.get("d") + if isinstance(s, int) and (self._last_seq is None or s > self._last_seq): + self._last_seq = s + + # op 10 = Hello (heartbeat interval) — must reply with Identify/Resume + if op == 10: + d_data = d if isinstance(d, dict) else {} + interval_ms = d_data.get("heartbeat_interval", 30000) + # Send heartbeats at 80% of the server interval to stay safe + self._heartbeat_interval = interval_ms / 1000.0 * 0.8 + logger.debug( + "[%s] Hello received, heartbeat_interval=%dms (sending every %.1fs)", + self._log_tag, + interval_ms, + self._heartbeat_interval, + ) + # Authenticate: send Resume if we have a session, else Identify. + # Use _create_task which is safe when no event loop is running (tests). + if self._session_id and self._last_seq is not None: + self._create_task(self._send_resume()) + else: + self._create_task(self._send_identify()) + return + + # op 0 = Dispatch + if op == 0 and t: + if t == "READY": + self._handle_ready(d) + elif t == "RESUMED": + logger.info("[%s] Session resumed", self._log_tag) + elif t in ( + "C2C_MESSAGE_CREATE", + "GROUP_AT_MESSAGE_CREATE", + "DIRECT_MESSAGE_CREATE", + "GUILD_MESSAGE_CREATE", + "GUILD_AT_MESSAGE_CREATE", + ): + asyncio.create_task(self._on_message(t, d)) + else: + logger.debug("[%s] Unhandled dispatch: %s", self._log_tag, t) + return + + # op 11 = Heartbeat ACK + if op == 11: + return + + logger.debug("[%s] Unknown op: %s", self._log_tag, op) + + def _handle_ready(self, d: Any) -> None: + """Handle the READY event — store session_id for resume.""" + if isinstance(d, dict): + self._session_id = d.get("session_id") + logger.info("[%s] Ready, session_id=%s", self._log_tag, self._session_id) + + # ------------------------------------------------------------------ + # JSON helpers + # ------------------------------------------------------------------ + + @staticmethod + def _parse_json(raw: Any) -> Optional[Dict[str, Any]]: + try: + payload = json.loads(raw) + except Exception: + logger.warning("[QQBot] Failed to parse JSON: %r", raw) + return None + return payload if isinstance(payload, dict) else None + + @staticmethod + def _next_msg_seq(msg_id: str) -> int: + """Generate a message sequence number in 0..65535 range.""" + time_part = int(time.time()) % 100000000 + rand = int(uuid.uuid4().hex[:4], 16) + return (time_part ^ rand) % 65536 + + # ------------------------------------------------------------------ + # Inbound message handling + # ------------------------------------------------------------------ + + async def handle_message(self, event: MessageEvent) -> None: + """Cache the last message ID per chat, then delegate to base.""" + if event.message_id and event.source.chat_id: + self._last_msg_id[event.source.chat_id] = event.message_id + await super().handle_message(event) + + async def _on_message(self, event_type: str, d: Any) -> None: + """Process an inbound QQ Bot message event.""" + if not isinstance(d, dict): + return + + # Extract common fields + msg_id = str(d.get("id", "")) + if not msg_id or self._is_duplicate(msg_id): + logger.debug( + "[%s] Duplicate or missing message id: %s", self._log_tag, msg_id + ) + return + + timestamp = str(d.get("timestamp", "")) + content = str(d.get("content", "")).strip() + author = d.get("author") if isinstance(d.get("author"), dict) else {} + + # Route by event type + if event_type == "C2C_MESSAGE_CREATE": + await self._handle_c2c_message(d, msg_id, content, author, timestamp) + elif event_type in ("GROUP_AT_MESSAGE_CREATE",): + await self._handle_group_message(d, msg_id, content, author, timestamp) + elif event_type in ("GUILD_MESSAGE_CREATE", "GUILD_AT_MESSAGE_CREATE"): + await self._handle_guild_message(d, msg_id, content, author, timestamp) + elif event_type == "DIRECT_MESSAGE_CREATE": + await self._handle_dm_message(d, msg_id, content, author, timestamp) + + async def _handle_c2c_message( + self, + d: Dict[str, Any], + msg_id: str, + content: str, + author: Dict[str, Any], + timestamp: str, + ) -> None: + """Handle a C2C (private) message event.""" + user_openid = str(author.get("user_openid", "")) + if not user_openid: + return + if not self._is_dm_allowed(user_openid): + return + + text = content + attachments_raw = d.get("attachments") + logger.info( + "[%s] C2C message: id=%s content=%r attachments=%s", + self._log_tag, + msg_id, + content[:50] if content else "", + ( + f"{len(attachments_raw) if isinstance(attachments_raw, list) else 0} items" + if attachments_raw + else "None" + ), + ) + if attachments_raw and isinstance(attachments_raw, list): + for _i, _att in enumerate(attachments_raw): + if isinstance(_att, dict): + logger.info( + "[%s] attachment[%d]: content_type=%s url=%s filename=%s", + self._log_tag, + _i, + _att.get("content_type", ""), + str(_att.get("url", ""))[:80], + _att.get("filename", ""), + ) + + # Process all attachments uniformly (images, voice, files) + att_result = await self._process_attachments(attachments_raw) + image_urls = att_result["image_urls"] + image_media_types = att_result["image_media_types"] + voice_transcripts = att_result["voice_transcripts"] + attachment_info = att_result["attachment_info"] + + # Append voice transcripts to the text body + if voice_transcripts: + voice_block = "\n".join(voice_transcripts) + text = ( + (text + "\n\n" + voice_block).strip() if text.strip() else voice_block + ) + # Append non-media attachment info + if attachment_info: + text = ( + (text + "\n\n" + attachment_info).strip() + if text.strip() + else attachment_info + ) + + logger.info( + "[%s] After processing: images=%d, voice=%d", + self._log_tag, + len(image_urls), + len(voice_transcripts), + ) + + if not text.strip() and not image_urls: + return + + self._chat_type_map[user_openid] = "c2c" + event = MessageEvent( + source=self.build_source( + chat_id=user_openid, + user_id=user_openid, + chat_type="dm", + ), + text=text, + message_type=self._detect_message_type(image_urls, image_media_types), + raw_message=d, + message_id=msg_id, + media_urls=image_urls, + media_types=image_media_types, + timestamp=self._parse_qq_timestamp(timestamp), + ) + await self.handle_message(event) + + async def _handle_group_message( + self, + d: Dict[str, Any], + msg_id: str, + content: str, + author: Dict[str, Any], + timestamp: str, + ) -> None: + """Handle a group @-message event.""" + group_openid = str(d.get("group_openid", "")) + if not group_openid: + return + if not self._is_group_allowed( + group_openid, str(author.get("member_openid", "")) + ): + return + + # Strip the @bot mention prefix from content + text = self._strip_at_mention(content) + att_result = await self._process_attachments(d.get("attachments")) + image_urls = att_result["image_urls"] + image_media_types = att_result["image_media_types"] + voice_transcripts = att_result["voice_transcripts"] + attachment_info = att_result["attachment_info"] + + # Append voice transcripts + if voice_transcripts: + voice_block = "\n".join(voice_transcripts) + text = ( + (text + "\n\n" + voice_block).strip() if text.strip() else voice_block + ) + if attachment_info: + text = ( + (text + "\n\n" + attachment_info).strip() + if text.strip() + else attachment_info + ) + + if not text.strip() and not image_urls: + return + + self._chat_type_map[group_openid] = "group" + event = MessageEvent( + source=self.build_source( + chat_id=group_openid, + user_id=str(author.get("member_openid", "")), + chat_type="group", + ), + text=text, + message_type=self._detect_message_type(image_urls, image_media_types), + raw_message=d, + message_id=msg_id, + media_urls=image_urls, + media_types=image_media_types, + timestamp=self._parse_qq_timestamp(timestamp), + ) + await self.handle_message(event) + + async def _handle_guild_message( + self, + d: Dict[str, Any], + msg_id: str, + content: str, + author: Dict[str, Any], + timestamp: str, + ) -> None: + """Handle a guild/channel message event.""" + channel_id = str(d.get("channel_id", "")) + if not channel_id: + return + + member = d.get("member") if isinstance(d.get("member"), dict) else {} + nick = str(member.get("nick", "")) or str(author.get("username", "")) + + text = content + att_result = await self._process_attachments(d.get("attachments")) + image_urls = att_result["image_urls"] + image_media_types = att_result["image_media_types"] + voice_transcripts = att_result["voice_transcripts"] + attachment_info = att_result["attachment_info"] + + if voice_transcripts: + voice_block = "\n".join(voice_transcripts) + text = ( + (text + "\n\n" + voice_block).strip() if text.strip() else voice_block + ) + if attachment_info: + text = ( + (text + "\n\n" + attachment_info).strip() + if text.strip() + else attachment_info + ) + + if not text.strip() and not image_urls: + return + + self._chat_type_map[channel_id] = "guild" + event = MessageEvent( + source=self.build_source( + chat_id=channel_id, + user_id=str(author.get("id", "")), + user_name=nick or None, + chat_type="group", + ), + text=text, + message_type=self._detect_message_type(image_urls, image_media_types), + raw_message=d, + message_id=msg_id, + media_urls=image_urls, + media_types=image_media_types, + timestamp=self._parse_qq_timestamp(timestamp), + ) + await self.handle_message(event) + + async def _handle_dm_message( + self, + d: Dict[str, Any], + msg_id: str, + content: str, + author: Dict[str, Any], + timestamp: str, + ) -> None: + """Handle a guild DM message event.""" + guild_id = str(d.get("guild_id", "")) + if not guild_id: + return + + text = content + att_result = await self._process_attachments(d.get("attachments")) + image_urls = att_result["image_urls"] + image_media_types = att_result["image_media_types"] + voice_transcripts = att_result["voice_transcripts"] + attachment_info = att_result["attachment_info"] + + if voice_transcripts: + voice_block = "\n".join(voice_transcripts) + text = ( + (text + "\n\n" + voice_block).strip() if text.strip() else voice_block + ) + if attachment_info: + text = ( + (text + "\n\n" + attachment_info).strip() + if text.strip() + else attachment_info + ) + + if not text.strip() and not image_urls: + return + + self._chat_type_map[guild_id] = "dm" + event = MessageEvent( + source=self.build_source( + chat_id=guild_id, + user_id=str(author.get("id", "")), + chat_type="dm", + ), + text=text, + message_type=self._detect_message_type(image_urls, image_media_types), + raw_message=d, + message_id=msg_id, + media_urls=image_urls, + media_types=image_media_types, + timestamp=self._parse_qq_timestamp(timestamp), + ) + await self.handle_message(event) + + # ------------------------------------------------------------------ + # Attachment processing + # ------------------------------------------------------------------ + + @staticmethod + def _detect_message_type(media_urls: list, media_types: list): + """Determine MessageType from attachment content types.""" + if not media_urls: + return MessageType.TEXT + if not media_types: + return MessageType.PHOTO + first_type = media_types[0].lower() if media_types else "" + if "audio" in first_type or "voice" in first_type or "silk" in first_type: + return MessageType.VOICE + if "video" in first_type: + return MessageType.VIDEO + if "image" in first_type or "photo" in first_type: + return MessageType.PHOTO + logger.debug( + "Unknown media content_type '%s', defaulting to TEXT", + first_type, + ) + return MessageType.TEXT + + async def _process_attachments( + self, + attachments: Any, + ) -> Dict[str, Any]: + """Process inbound attachments (all message types). + + Mirrors OpenClaw's ``processAttachments`` — handles images, voice, and + other files uniformly. + + Returns a dict with: + - image_urls: list[str] — cached local image paths + - image_media_types: list[str] — MIME types of cached images + - voice_transcripts: list[str] — STT transcripts for voice messages + - attachment_info: str — text description of non-image, non-voice attachments + """ + if not isinstance(attachments, list): + return { + "image_urls": [], + "image_media_types": [], + "voice_transcripts": [], + "attachment_info": "", + } + + image_urls: List[str] = [] + image_media_types: List[str] = [] + voice_transcripts: List[str] = [] + other_attachments: List[str] = [] + + for att in attachments: + if not isinstance(att, dict): + continue + + ct = str(att.get("content_type", "")).strip().lower() + url_raw = str(att.get("url", "")).strip() + filename = str(att.get("filename", "")) + if url_raw.startswith("//"): + url = f"https:{url_raw}" + elif url_raw: + url = url_raw + else: + url = "" + continue + + logger.debug( + "[%s] Processing attachment: content_type=%s, url=%s, filename=%s", + self._log_tag, + ct, + url[:80], + filename, + ) + + if self._is_voice_content_type(ct, filename): + # Voice: use QQ's asr_refer_text first, then voice_wav_url, then STT. + asr_refer = ( + str(att.get("asr_refer_text", "")).strip() + if isinstance(att.get("asr_refer_text"), str) + else "" + ) + voice_wav_url = ( + str(att.get("voice_wav_url", "")).strip() + if isinstance(att.get("voice_wav_url"), str) + else "" + ) + + transcript = await self._stt_voice_attachment( + url, + ct, + filename, + asr_refer_text=asr_refer or None, + voice_wav_url=voice_wav_url or None, + ) + if transcript: + voice_transcripts.append(f"[Voice] {transcript}") + logger.debug("[%s] Voice transcript: %s", self._log_tag, transcript) + else: + logger.warning("[%s] Voice STT failed for %s", self._log_tag, url[:60]) + voice_transcripts.append("[Voice] [语音识别失败]") + elif ct.startswith("image/"): + # Image: download and cache locally. + try: + cached_path = await self._download_and_cache(url, ct) + if cached_path and os.path.isfile(cached_path): + image_urls.append(cached_path) + image_media_types.append(ct or "image/jpeg") + elif cached_path: + logger.warning( + "[%s] Cached image path does not exist: %s", + self._log_tag, + cached_path, + ) + except Exception as exc: + logger.debug("[%s] Failed to cache image: %s", self._log_tag, exc) + else: + # Other attachments (video, file, etc.): record as text. + try: + cached_path = await self._download_and_cache(url, ct) + if cached_path: + other_attachments.append(f"[Attachment: {filename or ct}]") + except Exception as exc: + logger.debug("[%s] Failed to cache attachment: %s", self._log_tag, exc) + + attachment_info = "\n".join(other_attachments) if other_attachments else "" + return { + "image_urls": image_urls, + "image_media_types": image_media_types, + "voice_transcripts": voice_transcripts, + "attachment_info": attachment_info, + } + + async def _download_and_cache(self, url: str, content_type: str) -> Optional[str]: + """Download a URL and cache it locally.""" + from tools.url_safety import is_safe_url + + if not is_safe_url(url): + raise ValueError(f"Blocked unsafe URL: {url[:80]}") + + if not self._http_client: + return None + + try: + resp = await self._http_client.get( + url, + timeout=30.0, + headers=self._qq_media_headers(), + ) + resp.raise_for_status() + data = resp.content + except Exception as exc: + logger.debug( + "[%s] Download failed for %s: %s", self._log_tag, url[:80], exc + ) + return None + + if content_type.startswith("image/"): + ext = mimetypes.guess_extension(content_type) or ".jpg" + return cache_image_from_bytes(data, ext) + elif content_type == "voice" or content_type.startswith("audio/"): + # QQ voice messages are typically .amr or .silk format. + # Convert to .wav using ffmpeg so STT engines can process it. + return await self._convert_audio_to_wav(data, url) + else: + filename = Path(urlparse(url).path).name or "qq_attachment" + return cache_document_from_bytes(data, filename) + + @staticmethod + def _is_voice_content_type(content_type: str, filename: str) -> bool: + """Check if an attachment is a voice/audio message.""" + ct = content_type.strip().lower() + fn = filename.strip().lower() + if ct == "voice" or ct.startswith("audio/"): + return True + _VOICE_EXTENSIONS = ( + ".silk", + ".amr", + ".mp3", + ".wav", + ".ogg", + ".m4a", + ".aac", + ".speex", + ".flac", + ) + if any(fn.endswith(ext) for ext in _VOICE_EXTENSIONS): + return True + return False + + def _qq_media_headers(self) -> Dict[str, str]: + """Return Authorization headers for QQ multimedia CDN downloads. + + QQ's multimedia URLs (multimedia.nt.qq.com.cn) require the bot's + access token in an Authorization header, otherwise the download + returns a non-200 status. + """ + if self._access_token: + return {"Authorization": f"QQBot {self._access_token}"} + return {} + + async def _stt_voice_attachment( + self, + url: str, + content_type: str, + filename: str, + *, + asr_refer_text: Optional[str] = None, + voice_wav_url: Optional[str] = None, + ) -> Optional[str]: + """Download a voice attachment, convert to wav, and transcribe. + + Priority: + 1. QQ's built-in ``asr_refer_text`` (Tencent's own ASR — free, no API call). + 2. Self-hosted STT on ``voice_wav_url`` (pre-converted WAV from QQ, avoids SILK decoding). + 3. Self-hosted STT on the original attachment URL (requires SILK→WAV conversion). + + Returns the transcript text, or None on failure. + """ + # 1. Use QQ's built-in ASR text if available + if asr_refer_text: + logger.debug( + "[%s] STT: using QQ asr_refer_text: %r", self._log_tag, asr_refer_text[:100] + ) + return asr_refer_text + + # Determine which URL to download (prefer voice_wav_url — already WAV) + download_url = url + is_pre_wav = False + if voice_wav_url: + if voice_wav_url.startswith("//"): + voice_wav_url = f"https:{voice_wav_url}" + download_url = voice_wav_url + is_pre_wav = True + logger.debug("[%s] STT: using voice_wav_url (pre-converted WAV)", self._log_tag) + + from tools.url_safety import is_safe_url + if not is_safe_url(download_url): + logger.warning("[QQ] STT blocked unsafe URL: %s", download_url[:80]) + return None + + try: + # 2. Download audio (QQ CDN requires Authorization header) + if not self._http_client: + logger.warning("[%s] STT: no HTTP client", self._log_tag) + return None + + download_headers = self._qq_media_headers() + logger.debug( + "[%s] STT: downloading voice from %s (pre_wav=%s, headers=%s)", + self._log_tag, + download_url[:80], + is_pre_wav, + bool(download_headers), + ) + resp = await self._http_client.get( + download_url, + timeout=30.0, + headers=download_headers, + follow_redirects=True, + ) + resp.raise_for_status() + audio_data = resp.content + logger.debug( + "[%s] STT: downloaded %d bytes, content_type=%s", + self._log_tag, + len(audio_data), + resp.headers.get("content-type", "unknown"), + ) + + if len(audio_data) < 10: + logger.warning( + "[%s] STT: downloaded data too small (%d bytes), skipping", + self._log_tag, + len(audio_data), + ) + return None + + # 3. Convert to wav (skip if we already have a pre-converted WAV) + if is_pre_wav: + import tempfile + + with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp: + tmp.write(audio_data) + wav_path = tmp.name + logger.debug( + "[%s] STT: using pre-converted WAV directly (%d bytes)", + self._log_tag, + len(audio_data), + ) + else: + logger.debug( + "[%s] STT: converting to wav, filename=%r", self._log_tag, filename + ) + wav_path = await self._convert_audio_to_wav_file(audio_data, filename) + if not wav_path or not Path(wav_path).exists(): + logger.warning( + "[%s] STT: ffmpeg conversion produced no output", self._log_tag + ) + return None + + # 4. Call STT API + logger.debug("[%s] STT: calling ASR on %s", self._log_tag, wav_path) + transcript = await self._call_stt(wav_path) + + # 5. Cleanup temp file + try: + os.unlink(wav_path) + except OSError: + pass + + if transcript: + logger.debug("[%s] STT success: %r", self._log_tag, transcript[:100]) + else: + logger.warning("[%s] STT: ASR returned empty transcript", self._log_tag) + return transcript + except (httpx.HTTPStatusError, httpx.TransportError, IOError) as exc: + logger.warning( + "[%s] STT failed for voice attachment: %s: %s", + self._log_tag, + type(exc).__name__, + exc, + ) + return None + + async def _convert_audio_to_wav_file( + self, audio_data: bytes, filename: str + ) -> Optional[str]: + """Convert audio bytes to a temp .wav file using pilk (SILK) or ffmpeg. + + QQ voice messages are typically SILK format which ffmpeg cannot decode. + Strategy: always try pilk first, fall back to ffmpeg if pilk fails. + + Returns the wav file path, or None on failure. + """ + import tempfile + + ext = ( + Path(filename).suffix.lower() + if Path(filename).suffix + else self._guess_ext_from_data(audio_data) + ) + logger.info( + "[%s] STT: audio_data size=%d, ext=%r, first_20_bytes=%r", + self._log_tag, + len(audio_data), + ext, + audio_data[:20], + ) + + with tempfile.NamedTemporaryFile(suffix=ext, delete=False) as tmp_src: + tmp_src.write(audio_data) + src_path = tmp_src.name + + wav_path = src_path.rsplit(".", 1)[0] + ".wav" + + # Try pilk first (handles SILK and many other formats) + result = await self._convert_silk_to_wav(src_path, wav_path) + + # If pilk failed, try ffmpeg + if not result: + result = await self._convert_ffmpeg_to_wav(src_path, wav_path) + + # If ffmpeg also failed, try writing raw PCM as WAV (last resort) + if not result: + result = await self._convert_raw_to_wav(audio_data, wav_path) + + # Cleanup source file + try: + os.unlink(src_path) + except OSError: + pass + + return result + + @staticmethod + def _guess_ext_from_data(data: bytes) -> str: + """Guess file extension from magic bytes.""" + if data[:9] == b"#!SILK_V3" or data[:5] == b"#!SILK": + return ".silk" + if data[:2] == b"\x02!": + return ".silk" + if data[:4] == b"RIFF": + return ".wav" + if data[:4] == b"fLaC": + return ".flac" + if data[:2] in (b"\xff\xfb", b"\xff\xf3", b"\xff\xf2"): + return ".mp3" + if data[:4] == b"\x30\x26\xb2\x75" or data[:4] == b"\x4f\x67\x67\x53": + return ".ogg" + if data[:4] == b"\x00\x00\x00\x20" or data[:4] == b"\x00\x00\x00\x1c": + return ".amr" + # Default to .amr for unknown (QQ's most common voice format) + return ".amr" + + @staticmethod + def _looks_like_silk(data: bytes) -> bool: + """Check if bytes look like a SILK audio file.""" + return data[:4] == b"#!SILK" or data[:2] == b"\x02!" or data[:9] == b"#!SILK_V3" + + async def _convert_silk_to_wav(self, src_path: str, wav_path: str) -> Optional[str]: + """Convert audio file to WAV using the pilk library. + + Tries the file as-is first, then as .silk if the extension differs. + pilk can handle SILK files with various headers (or no header). + """ + try: + import pilk + except ImportError: + logger.warning( + "[%s] pilk not installed — cannot decode SILK audio. Run: pip install pilk", + self._log_tag, + ) + return None + + # Try converting the file as-is + try: + pilk.silk_to_wav(src_path, wav_path, rate=16000) + if Path(wav_path).exists() and Path(wav_path).stat().st_size > 44: + logger.debug( + "[%s] pilk converted %s to wav (%d bytes)", + self._log_tag, + Path(src_path).name, + Path(wav_path).stat().st_size, + ) + return wav_path + except Exception as exc: + logger.debug("[%s] pilk direct conversion failed: %s", self._log_tag, exc) + + # Try renaming to .silk and converting (pilk checks the extension) + silk_path = src_path.rsplit(".", 1)[0] + ".silk" + try: + import shutil + + shutil.copy2(src_path, silk_path) + pilk.silk_to_wav(silk_path, wav_path, rate=16000) + if Path(wav_path).exists() and Path(wav_path).stat().st_size > 44: + logger.debug( + "[%s] pilk converted %s (as .silk) to wav (%d bytes)", + self._log_tag, + Path(src_path).name, + Path(wav_path).stat().st_size, + ) + return wav_path + except Exception as exc: + logger.debug("[%s] pilk .silk conversion failed: %s", self._log_tag, exc) + finally: + try: + os.unlink(silk_path) + except OSError: + pass + + return None + + async def _convert_raw_to_wav(self, audio_data: bytes, wav_path: str) -> Optional[str]: + """Last resort: try writing audio data as raw PCM 16-bit mono 16kHz WAV. + + This will produce garbage if the data isn't raw PCM, but at least + the ASR engine won't crash — it'll just return empty. + """ + try: + import wave + + with wave.open(wav_path, "w") as wf: + wf.setnchannels(1) + wf.setsampwidth(2) + wf.setframerate(16000) + wf.writeframes(audio_data) + return wav_path + except Exception as exc: + logger.debug("[%s] raw PCM fallback failed: %s", self._log_tag, exc) + return None + + async def _convert_ffmpeg_to_wav(self, src_path: str, wav_path: str) -> Optional[str]: + """Convert audio file to WAV using ffmpeg.""" + try: + proc = await asyncio.create_subprocess_exec( + "ffmpeg", + "-y", + "-i", + src_path, + "-ar", + "16000", + "-ac", + "1", + wav_path, + stdout=asyncio.subprocess.DEVNULL, + stderr=asyncio.subprocess.PIPE, + ) + await asyncio.wait_for(proc.wait(), timeout=30) + if proc.returncode != 0: + stderr = await proc.stderr.read() if proc.stderr else b"" + logger.warning( + "[%s] ffmpeg failed for %s: %s", + self._log_tag, + Path(src_path).name, + stderr[:200].decode(errors="replace"), + ) + return None + except (asyncio.TimeoutError, FileNotFoundError) as exc: + logger.warning("[%s] ffmpeg conversion error: %s", self._log_tag, exc) + return None + + if not Path(wav_path).exists() or Path(wav_path).stat().st_size <= 44: + logger.warning( + "[%s] ffmpeg produced no/small output for %s", + self._log_tag, + Path(src_path).name, + ) + return None + logger.debug( + "[%s] ffmpeg converted %s to wav (%d bytes)", + self._log_tag, + Path(src_path).name, + Path(wav_path).stat().st_size, + ) + return wav_path + + def _resolve_stt_config(self) -> Optional[Dict[str, str]]: + """Resolve STT backend configuration from config/environment. + + Priority: + 1. Plugin-specific: ``channels.qqbot.stt`` in config.yaml → ``self.config.extra["stt"]`` + 2. QQ-specific env vars: ``QQ_STT_API_KEY`` / ``QQ_STT_BASE_URL`` / ``QQ_STT_MODEL`` + 3. Return None if nothing is configured (STT will be skipped, QQ built-in ASR still works). + """ + extra = self.config.extra or {} + + # 1. Plugin-specific STT config (matches OpenClaw's channels.qqbot.stt) + stt_cfg = extra.get("stt") + if isinstance(stt_cfg, dict) and stt_cfg.get("enabled") is not False: + base_url = stt_cfg.get("baseUrl") or stt_cfg.get("base_url", "") + api_key = stt_cfg.get("apiKey") or stt_cfg.get("api_key", "") + model = stt_cfg.get("model", "") + if base_url and api_key: + return { + "base_url": base_url.rstrip("/"), + "api_key": api_key, + "model": model or "whisper-1", + } + # Provider-only config: just model name, use default provider + if api_key: + provider = stt_cfg.get("provider", "zai") + # Map provider to base URL + _PROVIDER_BASE_URLS = { + "zai": "https://open.bigmodel.cn/api/coding/paas/v4", + "openai": "https://api.openai.com/v1", + "glm": "https://open.bigmodel.cn/api/coding/paas/v4", + } + base_url = _PROVIDER_BASE_URLS.get(provider, "") + if base_url: + return { + "base_url": base_url, + "api_key": api_key, + "model": model + or ("glm-asr" if provider in ("zai", "glm") else "whisper-1"), + } + + # 2. QQ-specific env vars (set by `hermes setup gateway` / `hermes gateway`) + qq_stt_key = os.getenv("QQ_STT_API_KEY", "") + if qq_stt_key: + base_url = os.getenv( + "QQ_STT_BASE_URL", + "https://open.bigmodel.cn/api/coding/paas/v4", + ) + model = os.getenv("QQ_STT_MODEL", "glm-asr") + return { + "base_url": base_url.rstrip("/"), + "api_key": qq_stt_key, + "model": model, + } + + return None + + async def _call_stt(self, wav_path: str) -> Optional[str]: + """Call an OpenAI-compatible STT API to transcribe a wav file. + + Uses the provider configured in ``channels.qqbot.stt`` config, + falling back to QQ's built-in ``asr_refer_text`` if not configured. + Returns None if STT is not configured or the call fails. + """ + stt_cfg = self._resolve_stt_config() + if not stt_cfg: + logger.warning( + "[%s] STT not configured (no stt config or QQ_STT_API_KEY)", + self._log_tag, + ) + return None + + base_url = stt_cfg["base_url"] + api_key = stt_cfg["api_key"] + model = stt_cfg["model"] + + try: + with open(wav_path, "rb") as f: + resp = await self._http_client.post( + f"{base_url}/audio/transcriptions", + headers={"Authorization": f"Bearer {api_key}"}, + files={"file": (Path(wav_path).name, f, "audio/wav")}, + data={"model": model}, + timeout=30.0, + ) + resp.raise_for_status() + result = resp.json() + # Zhipu/GLM format: {"choices": [{"message": {"content": "transcript text"}}]} + choices = result.get("choices", []) + if choices: + content = choices[0].get("message", {}).get("content", "") + if content.strip(): + return content.strip() + # OpenAI/Whisper format: {"text": "transcript text"} + text = result.get("text", "") + if text.strip(): + return text.strip() + return None + except (httpx.HTTPStatusError, IOError) as exc: + logger.warning( + "[%s] STT API call failed (model=%s, base=%s): %s", + self._log_tag, + model, + base_url[:50], + exc, + ) + return None + + async def _convert_audio_to_wav( + self, audio_data: bytes, source_url: str + ) -> Optional[str]: + """Convert audio bytes to .wav using pilk (SILK) or ffmpeg, caching the result.""" + import tempfile + + # Determine source format from magic bytes or URL + ext = ( + Path(urlparse(source_url).path).suffix.lower() + if urlparse(source_url).path + else "" + ) + if not ext or ext not in ( + ".silk", + ".amr", + ".mp3", + ".wav", + ".ogg", + ".m4a", + ".aac", + ".flac", + ): + ext = self._guess_ext_from_data(audio_data) + + with tempfile.NamedTemporaryFile(suffix=ext, delete=False) as tmp_src: + tmp_src.write(audio_data) + src_path = tmp_src.name + + wav_path = src_path.rsplit(".", 1)[0] + ".wav" + try: + is_silk = ext == ".silk" or self._looks_like_silk(audio_data) + if is_silk: + result = await self._convert_silk_to_wav(src_path, wav_path) + else: + result = await self._convert_ffmpeg_to_wav(src_path, wav_path) + + if not result: + logger.warning( + "[%s] audio conversion failed for %s (format=%s)", + self._log_tag, + source_url[:60], + ext, + ) + return cache_document_from_bytes(audio_data, f"qq_voice{ext}") + except Exception: + return cache_document_from_bytes(audio_data, f"qq_voice{ext}") + finally: + try: + os.unlink(src_path) + except OSError: + pass + + # Verify output and cache + try: + wav_data = Path(wav_path).read_bytes() + os.unlink(wav_path) + return cache_document_from_bytes(wav_data, "qq_voice.wav") + except Exception as exc: + logger.debug("[%s] Failed to read converted wav: %s", self._log_tag, exc) + return None + + # ------------------------------------------------------------------ + # Outbound messaging — REST API + # ------------------------------------------------------------------ + + async def _api_request( + self, + method: str, + path: str, + body: Optional[Dict[str, Any]] = None, + timeout: float = DEFAULT_API_TIMEOUT, + ) -> Dict[str, Any]: + """Make an authenticated REST API request to QQ Bot API.""" + if not self._http_client: + raise RuntimeError("HTTP client not initialized — not connected?") + + token = await self._ensure_token() + headers = { + "Authorization": f"QQBot {token}", + "Content-Type": "application/json", + "User-Agent": build_user_agent(), + } + + try: + resp = await self._http_client.request( + method, + f"{API_BASE}{path}", + headers=headers, + json=body, + timeout=timeout, + ) + data = resp.json() + if resp.status_code >= 400: + raise RuntimeError( + f"QQ Bot API error [{resp.status_code}] {path}: " + f"{data.get('message', data)}" + ) + return data + except httpx.TimeoutException as exc: + raise RuntimeError(f"QQ Bot API timeout [{path}]: {exc}") from exc + + async def _upload_media( + self, + target_type: str, + target_id: str, + file_type: int, + url: Optional[str] = None, + file_data: Optional[str] = None, + srv_send_msg: bool = False, + file_name: Optional[str] = None, + ) -> Dict[str, Any]: + """Upload media and return file_info.""" + path = ( + f"/v2/users/{target_id}/files" + if target_type == "c2c" + else f"/v2/groups/{target_id}/files" + ) + + body: Dict[str, Any] = { + "file_type": file_type, + "srv_send_msg": srv_send_msg, + } + if url: + body["url"] = url + elif file_data: + body["file_data"] = file_data + if file_type == MEDIA_TYPE_FILE and file_name: + body["file_name"] = file_name + + # Retry transient upload failures + for attempt in range(3): + try: + return await self._api_request( + "POST", path, body, timeout=FILE_UPLOAD_TIMEOUT + ) + except RuntimeError as exc: + err_msg = str(exc) + if any( + kw in err_msg + for kw in ("400", "401", "Invalid", "timeout", "Timeout") + ): + raise + if attempt < 2: + await asyncio.sleep(1.5 * (attempt + 1)) + else: + raise + + # Maximum time (seconds) to wait for reconnection before giving up on send. + _RECONNECT_WAIT_SECONDS = 15.0 + # How often (seconds) to poll is_connected while waiting. + _RECONNECT_POLL_INTERVAL = 0.5 + + async def _wait_for_reconnection(self) -> bool: + """Wait for the WebSocket listener to reconnect. + + The listener loop (_listen_loop) auto-reconnects on disconnect, but + there is a race window where send() is called right after a disconnect + and before the reconnect completes. This method polls is_connected + for up to _RECONNECT_WAIT_SECONDS. + + Returns True if reconnected, False if still disconnected. + """ + logger.info("[%s] Not connected — waiting for reconnection (up to %.0fs)", + self._log_tag, self._RECONNECT_WAIT_SECONDS) + waited = 0.0 + while waited < self._RECONNECT_WAIT_SECONDS: + await asyncio.sleep(self._RECONNECT_POLL_INTERVAL) + waited += self._RECONNECT_POLL_INTERVAL + if self.is_connected: + logger.info("[%s] Reconnected after %.1fs", self._log_tag, waited) + return True + logger.warning("[%s] Still not connected after %.0fs", self._log_tag, self._RECONNECT_WAIT_SECONDS) + return False + + async def send( + self, + chat_id: str, + content: str, + reply_to: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + ) -> SendResult: + """Send a text or markdown message to a QQ user or group. + + Applies format_message(), splits long messages via truncate_message(), + and retries transient failures with exponential backoff. + """ + del metadata + + if not self.is_connected: + if not await self._wait_for_reconnection(): + return SendResult(success=False, error="Not connected", retryable=True) + + if not content or not content.strip(): + return SendResult(success=True) + + formatted = self.format_message(content) + chunks = self.truncate_message(formatted, self.MAX_MESSAGE_LENGTH) + + last_result = SendResult(success=False, error="No chunks") + for chunk in chunks: + last_result = await self._send_chunk(chat_id, chunk, reply_to) + if not last_result.success: + return last_result + # Only reply_to the first chunk + reply_to = None + return last_result + + async def _send_chunk( + self, + chat_id: str, + content: str, + reply_to: Optional[str] = None, + ) -> SendResult: + """Send a single chunk with retry + exponential backoff.""" + last_exc: Optional[Exception] = None + chat_type = self._guess_chat_type(chat_id) + + for attempt in range(3): + try: + if chat_type == "c2c": + return await self._send_c2c_text(chat_id, content, reply_to) + elif chat_type == "group": + return await self._send_group_text(chat_id, content, reply_to) + elif chat_type == "guild": + return await self._send_guild_text(chat_id, content, reply_to) + else: + return SendResult( + success=False, error=f"Unknown chat type for {chat_id}" + ) + except Exception as exc: + last_exc = exc + err = str(exc).lower() + # Permanent errors — don't retry + if any( + k in err + for k in ("invalid", "forbidden", "not found", "bad request") + ): + break + # Transient — back off and retry + if attempt < 2: + delay = 1.0 * (2 ** attempt) + logger.warning( + "[%s] send retry %d/3 after %.1fs: %s", + self._log_tag, + attempt + 1, + delay, + exc, + ) + await asyncio.sleep(delay) + + error_msg = str(last_exc) if last_exc else "Unknown error" + logger.error("[%s] Send failed: %s", self._log_tag, error_msg) + retryable = not any( + k in error_msg.lower() for k in ("invalid", "forbidden", "not found") + ) + return SendResult(success=False, error=error_msg, retryable=retryable) + + async def _send_c2c_text( + self, openid: str, content: str, reply_to: Optional[str] = None + ) -> SendResult: + """Send text to a C2C user via REST API.""" + msg_seq = self._next_msg_seq(reply_to or openid) + body = self._build_text_body(content, reply_to) + if reply_to: + body["msg_id"] = reply_to + + data = await self._api_request("POST", f"/v2/users/{openid}/messages", body) + msg_id = str(data.get("id", uuid.uuid4().hex[:12])) + return SendResult(success=True, message_id=msg_id, raw_response=data) + + async def _send_group_text( + self, group_openid: str, content: str, reply_to: Optional[str] = None + ) -> SendResult: + """Send text to a group via REST API.""" + msg_seq = self._next_msg_seq(reply_to or group_openid) + body = self._build_text_body(content, reply_to) + if reply_to: + body["msg_id"] = reply_to + + data = await self._api_request( + "POST", f"/v2/groups/{group_openid}/messages", body + ) + msg_id = str(data.get("id", uuid.uuid4().hex[:12])) + return SendResult(success=True, message_id=msg_id, raw_response=data) + + async def _send_guild_text( + self, channel_id: str, content: str, reply_to: Optional[str] = None + ) -> SendResult: + """Send text to a guild channel via REST API.""" + body: Dict[str, Any] = {"content": content[: self.MAX_MESSAGE_LENGTH]} + if reply_to: + body["msg_id"] = reply_to + + data = await self._api_request("POST", f"/channels/{channel_id}/messages", body) + msg_id = str(data.get("id", uuid.uuid4().hex[:12])) + return SendResult(success=True, message_id=msg_id, raw_response=data) + + def _build_text_body( + self, content: str, reply_to: Optional[str] = None + ) -> Dict[str, Any]: + """Build the message body for C2C/group text sending.""" + msg_seq = self._next_msg_seq(reply_to or "default") + + if self._markdown_support: + body: Dict[str, Any] = { + "markdown": {"content": content[: self.MAX_MESSAGE_LENGTH]}, + "msg_type": MSG_TYPE_MARKDOWN, + "msg_seq": msg_seq, + } + else: + body = { + "content": content[: self.MAX_MESSAGE_LENGTH], + "msg_type": MSG_TYPE_TEXT, + "msg_seq": msg_seq, + } + + if reply_to: + # For non-markdown mode, add message_reference + if not self._markdown_support: + body["message_reference"] = {"message_id": reply_to} + + return body + + # ------------------------------------------------------------------ + # Native media sending + # ------------------------------------------------------------------ + + async def send_image( + self, + chat_id: str, + image_url: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + ) -> SendResult: + """Send an image natively via QQ Bot API upload.""" + del metadata + + result = await self._send_media( + chat_id, image_url, MEDIA_TYPE_IMAGE, "image", caption, reply_to + ) + if result.success or not self._is_url(image_url): + return result + + # Fallback to text URL + logger.warning( + "[%s] Image send failed, falling back to text: %s", + self._log_tag, + result.error, + ) + fallback = f"{caption}\n{image_url}" if caption else image_url + return await self.send(chat_id=chat_id, content=fallback, reply_to=reply_to) + + async def send_image_file( + self, + chat_id: str, + image_path: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + **kwargs, + ) -> SendResult: + """Send a local image file natively.""" + del kwargs + return await self._send_media( + chat_id, image_path, MEDIA_TYPE_IMAGE, "image", caption, reply_to + ) + + async def send_voice( + self, + chat_id: str, + audio_path: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + **kwargs, + ) -> SendResult: + """Send a voice message natively.""" + del kwargs + return await self._send_media( + chat_id, audio_path, MEDIA_TYPE_VOICE, "voice", caption, reply_to + ) + + async def send_video( + self, + chat_id: str, + video_path: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + **kwargs, + ) -> SendResult: + """Send a video natively.""" + del kwargs + return await self._send_media( + chat_id, video_path, MEDIA_TYPE_VIDEO, "video", caption, reply_to + ) + + async def send_document( + self, + chat_id: str, + file_path: str, + caption: Optional[str] = None, + file_name: Optional[str] = None, + reply_to: Optional[str] = None, + **kwargs, + ) -> SendResult: + """Send a file/document natively.""" + del kwargs + return await self._send_media( + chat_id, + file_path, + MEDIA_TYPE_FILE, + "file", + caption, + reply_to, + file_name=file_name, + ) + + async def _send_media( + self, + chat_id: str, + media_source: str, + file_type: int, + kind: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + file_name: Optional[str] = None, + ) -> SendResult: + """Upload media and send as a native message.""" + if not self.is_connected: + if not await self._wait_for_reconnection(): + return SendResult(success=False, error="Not connected", retryable=True) + + try: + # Resolve media source + data, content_type, resolved_name = await self._load_media( + media_source, file_name + ) + + # Route + chat_type = self._guess_chat_type(chat_id) + target_path = ( + f"/v2/users/{chat_id}/files" + if chat_type == "c2c" + else f"/v2/groups/{chat_id}/files" + ) + + if chat_type == "guild": + # Guild channels don't support native media upload in the same way + # Send as URL fallback + return SendResult( + success=False, error="Guild media send not supported via this path" + ) + + # Upload + upload = await self._upload_media( + chat_type, + chat_id, + file_type, + file_data=data if not self._is_url(media_source) else None, + url=media_source if self._is_url(media_source) else None, + srv_send_msg=False, + file_name=resolved_name if file_type == MEDIA_TYPE_FILE else None, + ) + + file_info = upload.get("file_info") + if not file_info: + return SendResult( + success=False, error=f"Upload returned no file_info: {upload}" + ) + + # Send media message + msg_seq = self._next_msg_seq(chat_id) + body: Dict[str, Any] = { + "msg_type": MSG_TYPE_MEDIA, + "media": {"file_info": file_info}, + "msg_seq": msg_seq, + } + if caption: + body["content"] = caption[: self.MAX_MESSAGE_LENGTH] + if reply_to: + body["msg_id"] = reply_to + + send_data = await self._api_request( + "POST", + ( + f"/v2/users/{chat_id}/messages" + if chat_type == "c2c" + else f"/v2/groups/{chat_id}/messages" + ), + body, + ) + return SendResult( + success=True, + message_id=str(send_data.get("id", uuid.uuid4().hex[:12])), + raw_response=send_data, + ) + except Exception as exc: + logger.error("[%s] Media send failed: %s", self._log_tag, exc) + return SendResult(success=False, error=str(exc)) + + async def _load_media( + self, source: str, file_name: Optional[str] = None + ) -> Tuple[str, str, str]: + """Load media from URL or local path. Returns (base64_or_url, content_type, filename).""" + source = str(source).strip() + if not source: + raise ValueError("Media source is required") + + parsed = urlparse(source) + if parsed.scheme in ("http", "https"): + # For URLs, pass through directly to the upload API + content_type = mimetypes.guess_type(source)[0] or "application/octet-stream" + resolved_name = file_name or Path(parsed.path).name or "media" + return source, content_type, resolved_name + + # Local file — encode as raw base64 for QQ Bot API file_data field. + # The QQ API expects plain base64, NOT a data URI. + local_path = Path(source).expanduser() + if not local_path.is_absolute(): + local_path = (Path.cwd() / local_path).resolve() + + if not local_path.exists() or not local_path.is_file(): + # Guard against placeholder paths like "" that the LLM + # sometimes emits instead of real file paths. + if source.startswith("<") or len(source) < 3: + raise ValueError( + f"Invalid media source (looks like a placeholder): {source!r}" + ) + raise FileNotFoundError(f"Media file not found: {local_path}") + + raw = local_path.read_bytes() + resolved_name = file_name or local_path.name + content_type = ( + mimetypes.guess_type(str(local_path))[0] or "application/octet-stream" + ) + b64 = base64.b64encode(raw).decode("ascii") + return b64, content_type, resolved_name + + # ------------------------------------------------------------------ + # Typing indicator + # ------------------------------------------------------------------ + + async def send_typing(self, chat_id: str, metadata=None) -> None: + """Send an input notify to a C2C user (only supported for C2C). + + Debounced to one request per ~50s (the API sets a 60s indicator). + The QQ API requires the originating message ID — retrieved from + ``_last_msg_id`` which is populated by ``_on_message``. + """ + if not self.is_connected: + return + + chat_type = self._guess_chat_type(chat_id) + if chat_type != "c2c": + return + + msg_id = self._last_msg_id.get(chat_id) + if not msg_id: + return + + # Debounce — skip if we sent recently + now = time.time() + last_sent = self._typing_sent_at.get(chat_id, 0.0) + if now - last_sent < self._TYPING_DEBOUNCE_SECONDS: + return + + try: + msg_seq = self._next_msg_seq(chat_id) + body = { + "msg_type": MSG_TYPE_INPUT_NOTIFY, + "msg_id": msg_id, + "input_notify": { + "input_type": 1, + "input_second": self._TYPING_INPUT_SECONDS, + }, + "msg_seq": msg_seq, + } + await self._api_request("POST", f"/v2/users/{chat_id}/messages", body) + self._typing_sent_at[chat_id] = now + except Exception as exc: + logger.debug("[%s] send_typing failed: %s", self._log_tag, exc) + + # ------------------------------------------------------------------ + # Format + # ------------------------------------------------------------------ + + def format_message(self, content: str) -> str: + """Format message for QQ. + + When markdown_support is enabled, content is sent as-is (QQ renders it). + When disabled, strip markdown via shared helper (same as BlueBubbles/SMS). + """ + if self._markdown_support: + return content + return strip_markdown(content) + + # ------------------------------------------------------------------ + # Chat info + # ------------------------------------------------------------------ + + async def get_chat_info(self, chat_id: str) -> Dict[str, Any]: + """Return chat info based on chat type heuristics.""" + chat_type = self._guess_chat_type(chat_id) + return { + "name": chat_id, + "type": "group" if chat_type in ("group", "guild") else "dm", + } + + # ------------------------------------------------------------------ + # Helpers + # ------------------------------------------------------------------ + + @staticmethod + def _is_url(source: str) -> bool: + return urlparse(str(source)).scheme in ("http", "https") + + def _guess_chat_type(self, chat_id: str) -> str: + """Determine chat type from stored inbound metadata, fallback to 'c2c'.""" + if chat_id in self._chat_type_map: + return self._chat_type_map[chat_id] + return "c2c" + + @staticmethod + def _strip_at_mention(content: str) -> str: + """Strip the @bot mention prefix from group message content.""" + # QQ group @-messages may have the bot's QQ/ID as prefix + import re + + stripped = re.sub(r"^@\S+\s*", "", content.strip()) + return stripped + + def _is_dm_allowed(self, user_id: str) -> bool: + if self._dm_policy == "disabled": + return False + if self._dm_policy == "allowlist": + return self._entry_matches(self._allow_from, user_id) + return True + + def _is_group_allowed(self, group_id: str, user_id: str) -> bool: + if self._group_policy == "disabled": + return False + if self._group_policy == "allowlist": + return self._entry_matches(self._group_allow_from, group_id) + return True + + @staticmethod + def _entry_matches(entries: List[str], target: str) -> bool: + normalized_target = str(target).strip().lower() + for entry in entries: + normalized = str(entry).strip().lower() + if normalized == "*" or normalized == normalized_target: + return True + return False + + def _parse_qq_timestamp(self, raw: str) -> datetime: + """Parse QQ API timestamp (ISO 8601 string or integer ms). + + The QQ API changed from integer milliseconds to ISO 8601 strings. + This handles both formats gracefully. + """ + if not raw: + return datetime.now(tz=timezone.utc) + try: + return datetime.fromisoformat(raw) + except (ValueError, TypeError): + pass + try: + return datetime.fromtimestamp(int(raw) / 1000, tz=timezone.utc) + except (ValueError, TypeError): + pass + return datetime.now(tz=timezone.utc) + + def _is_duplicate(self, msg_id: str) -> bool: + now = time.time() + if len(self._seen_messages) > DEDUP_MAX_SIZE: + cutoff = now - DEDUP_WINDOW_SECONDS + self._seen_messages = { + key: ts for key, ts in self._seen_messages.items() if ts > cutoff + } + if msg_id in self._seen_messages: + return True + self._seen_messages[msg_id] = now + return False diff --git a/gateway/platforms/qqbot/constants.py b/gateway/platforms/qqbot/constants.py new file mode 100644 index 000000000000..ddae3c133edc --- /dev/null +++ b/gateway/platforms/qqbot/constants.py @@ -0,0 +1,74 @@ +"""QQBot package-level constants shared across adapter, onboard, and other modules.""" + +from __future__ import annotations + +import os + +# --------------------------------------------------------------------------- +# QQBot adapter version — bump on functional changes to the adapter package. +# --------------------------------------------------------------------------- + +QQBOT_VERSION = "1.1.0" + +# --------------------------------------------------------------------------- +# API endpoints +# --------------------------------------------------------------------------- + +# The portal domain is configurable via QQ_API_HOST for corporate proxies +# or test environments. Default: q.qq.com (production). +PORTAL_HOST = os.getenv("QQ_PORTAL_HOST", "q.qq.com") + +API_BASE = "https://api.sgroup.qq.com" +TOKEN_URL = "https://bots.qq.com/app/getAppAccessToken" +GATEWAY_URL_PATH = "/gateway" + +# QR-code onboard endpoints (on the portal host) +ONBOARD_CREATE_PATH = "/lite/create_bind_task" +ONBOARD_POLL_PATH = "/lite/poll_bind_result" +QR_URL_TEMPLATE = ( + "https://q.qq.com/qqbot/openclaw/connect.html" + "?task_id={task_id}&_wv=2&source=hermes" +) + +# --------------------------------------------------------------------------- +# Timeouts & retry +# --------------------------------------------------------------------------- + +DEFAULT_API_TIMEOUT = 30.0 +FILE_UPLOAD_TIMEOUT = 120.0 +CONNECT_TIMEOUT_SECONDS = 20.0 + +RECONNECT_BACKOFF = [2, 5, 10, 30, 60] +MAX_RECONNECT_ATTEMPTS = 100 +RATE_LIMIT_DELAY = 60 # seconds +QUICK_DISCONNECT_THRESHOLD = 5.0 # seconds +MAX_QUICK_DISCONNECT_COUNT = 3 + +ONBOARD_POLL_INTERVAL = 2.0 # seconds between poll_bind_result calls +ONBOARD_API_TIMEOUT = 10.0 + +# --------------------------------------------------------------------------- +# Message limits +# --------------------------------------------------------------------------- + +MAX_MESSAGE_LENGTH = 4000 +DEDUP_WINDOW_SECONDS = 300 +DEDUP_MAX_SIZE = 1000 + +# --------------------------------------------------------------------------- +# QQ Bot message types +# --------------------------------------------------------------------------- + +MSG_TYPE_TEXT = 0 +MSG_TYPE_MARKDOWN = 2 +MSG_TYPE_MEDIA = 7 +MSG_TYPE_INPUT_NOTIFY = 6 + +# --------------------------------------------------------------------------- +# QQ Bot file media types +# --------------------------------------------------------------------------- + +MEDIA_TYPE_IMAGE = 1 +MEDIA_TYPE_VIDEO = 2 +MEDIA_TYPE_VOICE = 3 +MEDIA_TYPE_FILE = 4 diff --git a/gateway/platforms/qqbot/crypto.py b/gateway/platforms/qqbot/crypto.py new file mode 100644 index 000000000000..426bd29de5e1 --- /dev/null +++ b/gateway/platforms/qqbot/crypto.py @@ -0,0 +1,45 @@ +"""AES-256-GCM utilities for QQBot scan-to-configure credential decryption.""" + +from __future__ import annotations + +import base64 +import os + + +def generate_bind_key() -> str: + """Generate a 256-bit random AES key and return it as base64. + + The key is passed to ``create_bind_task`` so the server can encrypt + the bot's *client_secret* before returning it. Only this CLI holds + the key, ensuring the secret never travels in plaintext. + """ + return base64.b64encode(os.urandom(32)).decode() + + +def decrypt_secret(encrypted_base64: str, key_base64: str) -> str: + """Decrypt a base64-encoded AES-256-GCM ciphertext. + + Ciphertext layout (after base64-decoding):: + + IV (12 bytes) ‖ ciphertext (N bytes) ‖ AuthTag (16 bytes) + + Args: + encrypted_base64: The ``bot_encrypt_secret`` value from + ``poll_bind_result``. + key_base64: The base64 AES key generated by + :func:`generate_bind_key`. + + Returns: + The decrypted *client_secret* as a UTF-8 string. + """ + from cryptography.hazmat.primitives.ciphers.aead import AESGCM + + key = base64.b64decode(key_base64) + raw = base64.b64decode(encrypted_base64) + + iv = raw[:12] + ciphertext_with_tag = raw[12:] # AESGCM expects ciphertext + tag concatenated + + aesgcm = AESGCM(key) + plaintext = aesgcm.decrypt(iv, ciphertext_with_tag, None) + return plaintext.decode("utf-8") diff --git a/gateway/platforms/qqbot/onboard.py b/gateway/platforms/qqbot/onboard.py new file mode 100644 index 000000000000..b48c39a4f87a --- /dev/null +++ b/gateway/platforms/qqbot/onboard.py @@ -0,0 +1,220 @@ +""" +QQBot scan-to-configure (QR code onboard) module. + +Mirrors the Feishu onboarding pattern: synchronous HTTP + a single public +entry-point ``qr_register()`` that handles the full flow (create task → +display QR code → poll → decrypt credentials). + +Calls the ``q.qq.com`` ``create_bind_task`` / ``poll_bind_result`` APIs to +generate a QR-code URL and poll for scan completion. On success the caller +receives the bot's *app_id*, *client_secret* (decrypted locally), and the +scanner's *user_openid* — enough to fully configure the QQBot gateway. + +Reference: https://bot.q.qq.com/wiki/develop/api-v2/ +""" + +from __future__ import annotations + +import logging +import time +from enum import IntEnum +from typing import Optional, Tuple +from urllib.parse import quote + +from .constants import ( + ONBOARD_API_TIMEOUT, + ONBOARD_CREATE_PATH, + ONBOARD_POLL_INTERVAL, + ONBOARD_POLL_PATH, + PORTAL_HOST, + QR_URL_TEMPLATE, +) +from .crypto import decrypt_secret, generate_bind_key +from .utils import get_api_headers + +logger = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# Bind status +# --------------------------------------------------------------------------- + + +class BindStatus(IntEnum): + """Status codes returned by ``_poll_bind_result``.""" + + NONE = 0 + PENDING = 1 + COMPLETED = 2 + EXPIRED = 3 + + +# --------------------------------------------------------------------------- +# QR rendering +# --------------------------------------------------------------------------- + +try: + import qrcode as _qrcode_mod +except (ImportError, TypeError): + _qrcode_mod = None # type: ignore[assignment] + + +def _render_qr(url: str) -> bool: + """Try to render a QR code in the terminal. Returns True if successful.""" + if _qrcode_mod is None: + return False + try: + qr = _qrcode_mod.QRCode( + error_correction=_qrcode_mod.constants.ERROR_CORRECT_M, + border=2, + ) + qr.add_data(url) + qr.make(fit=True) + qr.print_ascii(invert=True) + return True + except Exception: + return False + + +# --------------------------------------------------------------------------- +# Synchronous HTTP helpers (mirrors Feishu _post_registration pattern) +# --------------------------------------------------------------------------- + + +def _create_bind_task(timeout: float = ONBOARD_API_TIMEOUT) -> Tuple[str, str]: + """Create a bind task and return *(task_id, aes_key_base64)*. + + Raises: + RuntimeError: If the API returns a non-zero ``retcode``. + """ + import httpx + + url = f"https://{PORTAL_HOST}{ONBOARD_CREATE_PATH}" + key = generate_bind_key() + + with httpx.Client(timeout=timeout, follow_redirects=True) as client: + resp = client.post(url, json={"key": key}, headers=get_api_headers()) + resp.raise_for_status() + data = resp.json() + + if data.get("retcode") != 0: + raise RuntimeError(data.get("msg", "create_bind_task failed")) + + task_id = data.get("data", {}).get("task_id") + if not task_id: + raise RuntimeError("create_bind_task: missing task_id in response") + + logger.debug("create_bind_task ok: task_id=%s", task_id) + return task_id, key + + +def _poll_bind_result( + task_id: str, + timeout: float = ONBOARD_API_TIMEOUT, +) -> Tuple[BindStatus, str, str, str]: + """Poll the bind result for *task_id*. + + Returns: + A 4-tuple of ``(status, bot_appid, bot_encrypt_secret, user_openid)``. + + Raises: + RuntimeError: If the API returns a non-zero ``retcode``. + """ + import httpx + + url = f"https://{PORTAL_HOST}{ONBOARD_POLL_PATH}" + + with httpx.Client(timeout=timeout, follow_redirects=True) as client: + resp = client.post(url, json={"task_id": task_id}, headers=get_api_headers()) + resp.raise_for_status() + data = resp.json() + + if data.get("retcode") != 0: + raise RuntimeError(data.get("msg", "poll_bind_result failed")) + + d = data.get("data", {}) + return ( + BindStatus(d.get("status", 0)), + str(d.get("bot_appid", "")), + d.get("bot_encrypt_secret", ""), + d.get("user_openid", ""), + ) + + +def build_connect_url(task_id: str) -> str: + """Build the QR-code target URL for a given *task_id*.""" + return QR_URL_TEMPLATE.format(task_id=quote(task_id)) + + +# --------------------------------------------------------------------------- +# Public entry-point +# --------------------------------------------------------------------------- + +_MAX_REFRESHES = 3 + + +def qr_register(timeout_seconds: int = 600) -> Optional[dict]: + """Run the QQBot scan-to-configure QR registration flow. + + Mirrors ``feishu.qr_register()``: handles create → display → poll → + decrypt in one call. Unexpected errors propagate to the caller. + + :returns: + ``{"app_id": ..., "client_secret": ..., "user_openid": ...}`` on + success, or ``None`` on failure / expiry / cancellation. + """ + deadline = time.monotonic() + timeout_seconds + + for refresh_count in range(_MAX_REFRESHES + 1): + # ── Create bind task ── + try: + task_id, aes_key = _create_bind_task() + except Exception as exc: + logger.warning("[QQBot onboard] Failed to create bind task: %s", exc) + return None + + url = build_connect_url(task_id) + + # ── Display QR code + URL ── + print() + if _render_qr(url): + print(f" Scan the QR code above, or open this URL directly:\n {url}") + else: + print(f" Open this URL in QQ on your phone:\n {url}") + print(" Tip: pip install qrcode to display a scannable QR code here") + print() + + # ── Poll loop ── + while time.monotonic() < deadline: + try: + status, app_id, encrypted_secret, user_openid = _poll_bind_result(task_id) + except Exception: + time.sleep(ONBOARD_POLL_INTERVAL) + continue + + if status == BindStatus.COMPLETED: + client_secret = decrypt_secret(encrypted_secret, aes_key) + print() + print(f" QR scan complete! (App ID: {app_id})") + if user_openid: + print(f" Scanner's OpenID: {user_openid}") + return { + "app_id": app_id, + "client_secret": client_secret, + "user_openid": user_openid, + } + + if status == BindStatus.EXPIRED: + if refresh_count >= _MAX_REFRESHES: + logger.warning("[QQBot onboard] QR code expired %d times — giving up", _MAX_REFRESHES) + return None + print(f"\n QR code expired, refreshing... ({refresh_count + 1}/{_MAX_REFRESHES})") + break # next for-loop iteration creates a new task + + time.sleep(ONBOARD_POLL_INTERVAL) + else: + # deadline reached without completing + logger.warning("[QQBot onboard] Poll timed out after %ds", timeout_seconds) + return None + + return None diff --git a/gateway/platforms/qqbot/utils.py b/gateway/platforms/qqbot/utils.py new file mode 100644 index 000000000000..873e58d2a5d8 --- /dev/null +++ b/gateway/platforms/qqbot/utils.py @@ -0,0 +1,71 @@ +"""QQBot shared utilities — User-Agent, HTTP helpers, config coercion.""" + +from __future__ import annotations + +import platform +import sys +from typing import Any, Dict, List + +from .constants import QQBOT_VERSION + + +# --------------------------------------------------------------------------- +# User-Agent +# --------------------------------------------------------------------------- + +def _get_hermes_version() -> str: + """Return the hermes-agent package version, or 'dev' if unavailable.""" + try: + from importlib.metadata import version + return version("hermes-agent") + except Exception: + return "dev" + + +def build_user_agent() -> str: + """Build a descriptive User-Agent string. + + Format:: + + QQBotAdapter/ (Python/; ; Hermes/) + + Example:: + + QQBotAdapter/1.0.0 (Python/3.11.15; darwin; Hermes/0.9.0) + """ + py_version = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}" + os_name = platform.system().lower() + hermes_version = _get_hermes_version() + return f"QQBotAdapter/{QQBOT_VERSION} (Python/{py_version}; {os_name}; Hermes/{hermes_version})" + + +def get_api_headers() -> Dict[str, str]: + """Return standard HTTP headers for QQBot API requests. + + Includes ``Content-Type``, ``Accept``, and a dynamic ``User-Agent``. + ``q.qq.com`` requires ``Accept: application/json`` — without it, + the server returns a JavaScript anti-bot challenge page. + """ + return { + "Content-Type": "application/json", + "Accept": "application/json", + "User-Agent": build_user_agent(), + } + + +# --------------------------------------------------------------------------- +# Config helpers +# --------------------------------------------------------------------------- + +def coerce_list(value: Any) -> List[str]: + """Coerce config values into a trimmed string list. + + Accepts comma-separated strings, lists, tuples, sets, or single values. + """ + if value is None: + return [] + if isinstance(value, str): + return [item.strip() for item in value.split(",") if item.strip()] + if isinstance(value, (list, tuple, set)): + return [str(item).strip() for item in value if str(item).strip()] + return [str(value).strip()] if str(value).strip() else [] diff --git a/gateway/platforms/signal.py b/gateway/platforms/signal.py index 8ef7bd0d6054..9a0a6256a4b8 100644 --- a/gateway/platforms/signal.py +++ b/gateway/platforms/signal.py @@ -17,8 +17,8 @@ import logging import os import random -import re import time +import uuid from datetime import datetime, timezone from pathlib import Path from typing import Dict, List, Optional, Any @@ -128,6 +128,27 @@ def _render_mentions(text: str, mentions: list) -> str: return text +def _is_signal_service_id(value: str) -> bool: + """Return True if *value* already looks like a Signal service identifier.""" + if not value: + return False + if value.startswith("PNI:") or value.startswith("u:"): + return True + try: + uuid.UUID(value) + return True + except (ValueError, AttributeError, TypeError): + return False + + +def _looks_like_e164_number(value: str) -> bool: + """Return True for a plausible E.164 phone number.""" + if not value or not value.startswith("+"): + return False + digits = value[1:] + return digits.isdigit() and 7 <= len(digits) <= 15 + + def check_signal_requirements() -> bool: """Check if Signal is configured (has URL and account).""" return bool(os.getenv("SIGNAL_HTTP_URL") and os.getenv("SIGNAL_ACCOUNT")) @@ -161,6 +182,14 @@ def __init__(self, config: PlatformConfig): self._sse_task: Optional[asyncio.Task] = None self._health_monitor_task: Optional[asyncio.Task] = None self._typing_tasks: Dict[str, asyncio.Task] = {} + # Per-chat typing-indicator backoff. When signal-cli reports + # NETWORK_FAILURE (recipient offline / unroutable), base.py's + # _keep_typing refresh loop would otherwise hammer sendTyping every + # ~2s indefinitely, producing WARNING-level log spam and pointless + # RPC traffic. We track consecutive failures per chat and skip the + # RPC during a cooldown window instead. + self._typing_failures: Dict[str, int] = {} + self._typing_skip_until: Dict[str, float] = {} self._running = False self._last_sse_activity = 0.0 self._sse_response: Optional[httpx.Response] = None @@ -172,6 +201,12 @@ def __init__(self, config: PlatformConfig): # in Note to Self / self-chat mode (mirrors WhatsApp recentlySentIds) self._recent_sent_timestamps: set = set() self._max_recent_timestamps = 50 + # Signal increasingly exposes ACI/PNI UUIDs as stable recipient IDs. + # Keep a best-effort mapping so outbound sends can upgrade from a + # phone number to the corresponding UUID when signal-cli prefers it. + self._recipient_uuid_by_number: Dict[str, str] = {} + self._recipient_number_by_uuid: Dict[str, str] = {} + self._recipient_cache_lock = asyncio.Lock() logger.info("Signal adapter initialized: url=%s account=%s groups=%s", self.http_url, redact_phone(self.account), @@ -188,31 +223,40 @@ async def connect(self) -> bool: return False # Acquire scoped lock to prevent duplicate Signal listeners for the same phone + lock_acquired = False try: if not self._acquire_platform_lock('signal-phone', self.account, 'Signal account'): return False + lock_acquired = True except Exception as e: logger.warning("Signal: Could not acquire phone lock (non-fatal): %s", e) self.client = httpx.AsyncClient(timeout=30.0) - - # Health check — verify signal-cli daemon is reachable try: - resp = await self.client.get(f"{self.http_url}/api/v1/check", timeout=10.0) - if resp.status_code != 200: - logger.error("Signal: health check failed (status %d)", resp.status_code) + # Health check — verify signal-cli daemon is reachable + try: + resp = await self.client.get(f"{self.http_url}/api/v1/check", timeout=10.0) + if resp.status_code != 200: + logger.error("Signal: health check failed (status %d)", resp.status_code) + return False + except Exception as e: + logger.error("Signal: cannot reach signal-cli at %s: %s", self.http_url, e) return False - except Exception as e: - logger.error("Signal: cannot reach signal-cli at %s: %s", self.http_url, e) - return False - self._running = True - self._last_sse_activity = time.time() - self._sse_task = asyncio.create_task(self._sse_listener()) - self._health_monitor_task = asyncio.create_task(self._health_monitor()) + self._running = True + self._last_sse_activity = time.time() + self._sse_task = asyncio.create_task(self._sse_listener()) + self._health_monitor_task = asyncio.create_task(self._health_monitor()) - logger.info("Signal: connected to %s", self.http_url) - return True + logger.info("Signal: connected to %s", self.http_url) + return True + finally: + if not self._running: + if self.client: + await self.client.aclose() + self.client = None + if lock_acquired: + self._release_platform_lock() async def disconnect(self) -> None: """Stop SSE listener and clean up.""" @@ -393,6 +437,7 @@ async def _handle_envelope(self, envelope: dict) -> None: ) sender_name = envelope_data.get("sourceName", "") sender_uuid = envelope_data.get("sourceUuid", "") + self._remember_recipient_identifiers(sender, sender_uuid) if not sender: logger.debug("Signal: ignoring envelope with no sender") @@ -511,6 +556,64 @@ async def _handle_envelope(self, envelope: dict) -> None: await self.handle_message(event) + def _remember_recipient_identifiers(self, number: Optional[str], service_id: Optional[str]) -> None: + """Cache any number↔UUID mapping observed from Signal envelopes.""" + if not number or not service_id or not _is_signal_service_id(service_id): + return + self._recipient_uuid_by_number[number] = service_id + self._recipient_number_by_uuid[service_id] = number + + def _extract_contact_uuid(self, contact: Any, phone_number: str) -> Optional[str]: + """Best-effort extraction of a Signal service ID from listContacts output.""" + if not isinstance(contact, dict): + return None + + number = contact.get("number") + recipient = contact.get("recipient") + service_id = contact.get("uuid") or contact.get("serviceId") + if not service_id: + profile = contact.get("profile") + if isinstance(profile, dict): + service_id = profile.get("serviceId") or profile.get("uuid") + + if service_id and _is_signal_service_id(service_id): + matches_number = number == phone_number or recipient == phone_number + if matches_number: + return service_id + return None + + async def _resolve_recipient(self, chat_id: str) -> str: + """Return the preferred Signal recipient identifier for a direct chat.""" + if ( + not chat_id + or chat_id.startswith("group:") + or _is_signal_service_id(chat_id) + or not _looks_like_e164_number(chat_id) + ): + return chat_id + + cached = self._recipient_uuid_by_number.get(chat_id) + if cached: + return cached + + async with self._recipient_cache_lock: + cached = self._recipient_uuid_by_number.get(chat_id) + if cached: + return cached + + contacts = await self._rpc("listContacts", { + "account": self.account, + "allRecipients": True, + }) + if isinstance(contacts, list): + for contact in contacts: + number = contact.get("number") if isinstance(contact, dict) else None + service_id = self._extract_contact_uuid(contact, chat_id) + if number and service_id: + self._remember_recipient_identifiers(number, service_id) + + return self._recipient_uuid_by_number.get(chat_id, chat_id) + # ------------------------------------------------------------------ # Attachment Handling # ------------------------------------------------------------------ @@ -549,8 +652,22 @@ async def _fetch_attachment(self, attachment_id: str) -> tuple: # JSON-RPC Communication # ------------------------------------------------------------------ - async def _rpc(self, method: str, params: dict, rpc_id: str = None) -> Any: - """Send a JSON-RPC 2.0 request to signal-cli daemon.""" + async def _rpc( + self, + method: str, + params: dict, + rpc_id: str = None, + *, + log_failures: bool = True, + ) -> Any: + """Send a JSON-RPC 2.0 request to signal-cli daemon. + + When ``log_failures=False``, error and exception paths log at DEBUG + instead of WARNING — used by the typing-indicator path to silence + repeated NETWORK_FAILURE spam for unreachable recipients while + still preserving visibility for the first occurrence and for + unrelated RPCs. + """ if not self.client: logger.warning("Signal: RPC called but client not connected") return None @@ -575,13 +692,19 @@ async def _rpc(self, method: str, params: dict, rpc_id: str = None) -> Any: data = resp.json() if "error" in data: - logger.warning("Signal RPC error (%s): %s", method, data["error"]) + if log_failures: + logger.warning("Signal RPC error (%s): %s", method, data["error"]) + else: + logger.debug("Signal RPC error (%s): %s", method, data["error"]) return None return data.get("result") except Exception as e: - logger.warning("Signal RPC %s failed: %s", method, e) + if log_failures: + logger.warning("Signal RPC %s failed: %s", method, e) + else: + logger.debug("Signal RPC %s failed: %s", method, e) return None # ------------------------------------------------------------------ @@ -606,7 +729,7 @@ async def send( if chat_id.startswith("group:"): params["groupId"] = chat_id[6:] else: - params["recipient"] = [chat_id] + params["recipient"] = [await self._resolve_recipient(chat_id)] result = await self._rpc("send", params) @@ -628,7 +751,28 @@ def _track_sent_timestamp(self, rpc_result) -> None: self._recent_sent_timestamps.pop() async def send_typing(self, chat_id: str, metadata=None) -> None: - """Send a typing indicator.""" + """Send a typing indicator. + + base.py's ``_keep_typing`` refresh loop calls this every ~2s while + the agent is processing. If signal-cli returns NETWORK_FAILURE for + this recipient (offline, unroutable, group membership lost, etc.) + the unmitigated behaviour is: a WARNING log every 2 seconds for as + long as the agent keeps running. Instead we: + + - silence the WARNING after the first consecutive failure (subsequent + attempts log at DEBUG) so transport issues are still visible once + but don't flood the log, + - skip the RPC entirely during an exponential cooldown window once + three consecutive failures have happened, so we stop hammering + signal-cli with requests it can't deliver. + + A successful sendTyping clears the counters. + """ + now = time.monotonic() + skip_until = self._typing_skip_until.get(chat_id, 0.0) + if now < skip_until: + return + params: Dict[str, Any] = { "account": self.account, } @@ -636,9 +780,28 @@ async def send_typing(self, chat_id: str, metadata=None) -> None: if chat_id.startswith("group:"): params["groupId"] = chat_id[6:] else: - params["recipient"] = [chat_id] + params["recipient"] = [await self._resolve_recipient(chat_id)] + + fails = self._typing_failures.get(chat_id, 0) + result = await self._rpc( + "sendTyping", + params, + rpc_id="typing", + log_failures=(fails == 0), + ) - await self._rpc("sendTyping", params, rpc_id="typing") + if result is None: + fails += 1 + self._typing_failures[chat_id] = fails + # After 3 consecutive failures, back off exponentially (16s, + # 32s, 60s cap) to stop spamming signal-cli for a recipient + # that clearly isn't reachable right now. + if fails >= 3: + backoff = min(60.0, 16.0 * (2 ** (fails - 3))) + self._typing_skip_until[chat_id] = now + backoff + else: + self._typing_failures.pop(chat_id, None) + self._typing_skip_until.pop(chat_id, None) async def send_image( self, @@ -678,7 +841,7 @@ async def send_image( if chat_id.startswith("group:"): params["groupId"] = chat_id[6:] else: - params["recipient"] = [chat_id] + params["recipient"] = [await self._resolve_recipient(chat_id)] result = await self._rpc("send", params) if result is not None: @@ -717,7 +880,7 @@ async def _send_attachment( if chat_id.startswith("group:"): params["groupId"] = chat_id[6:] else: - params["recipient"] = [chat_id] + params["recipient"] = [await self._resolve_recipient(chat_id)] result = await self._rpc("send", params) if result is not None: @@ -781,21 +944,6 @@ async def send_video( # Typing Indicators # ------------------------------------------------------------------ - async def _start_typing_indicator(self, chat_id: str) -> None: - """Start a typing indicator loop for a chat.""" - if chat_id in self._typing_tasks: - return # Already running - - async def _typing_loop(): - try: - while True: - await self.send_typing(chat_id) - await asyncio.sleep(TYPING_INTERVAL) - except asyncio.CancelledError: - pass - - self._typing_tasks[chat_id] = asyncio.create_task(_typing_loop()) - async def _stop_typing_indicator(self, chat_id: str) -> None: """Stop a typing indicator loop for a chat.""" task = self._typing_tasks.pop(chat_id, None) @@ -805,6 +953,10 @@ async def _stop_typing_indicator(self, chat_id: str) -> None: await task except asyncio.CancelledError: pass + # Reset per-chat typing backoff state so the next agent turn starts + # fresh rather than inheriting a cooldown from a prior conversation. + self._typing_failures.pop(chat_id, None) + self._typing_skip_until.pop(chat_id, None) async def stop_typing(self, chat_id: str) -> None: """Public interface for stopping typing — called by base adapter's diff --git a/gateway/platforms/slack.py b/gateway/platforms/slack.py index 8f9934cf7a2d..191689a5aed1 100644 --- a/gateway/platforms/slack.py +++ b/gateway/platforms/slack.py @@ -38,6 +38,7 @@ BasePlatformAdapter, MessageEvent, MessageType, + ProcessingOutcome, SendResult, SUPPORTED_DOCUMENT_TYPES, safe_url_for_log, @@ -113,6 +114,11 @@ def __init__(self, config: PlatformConfig): # Cache for _fetch_thread_context results: cache_key → _ThreadContextCache self._thread_context_cache: Dict[str, _ThreadContextCache] = {} self._THREAD_CACHE_TTL = 60.0 + # Track message IDs that should get reaction lifecycle (DMs / @mentions). + self._reacting_message_ids: set = set() + # Track active assistant thread status indicators so stop_typing can + # clear them (chat_id → thread_ts). + self._active_status_threads: Dict[str, str] = {} async def connect(self) -> bool: """Connect to Slack via Socket Mode.""" @@ -150,9 +156,11 @@ async def connect(self) -> bool: except Exception as e: logger.warning("[Slack] Failed to read %s: %s", tokens_file, e) + lock_acquired = False try: if not self._acquire_platform_lock('slack-app-token', app_token, 'Slack app token'): return False + lock_acquired = True # First token is the primary — used for AsyncApp / Socket Mode primary_token = bot_tokens[0] @@ -228,6 +236,9 @@ async def handle_hermes_command(ack, command): except Exception as e: # pragma: no cover - defensive logging logger.error("[Slack] Connection failed: %s", e, exc_info=True) return False + finally: + if lock_acquired and not self._running: + self._release_platform_lock() async def disconnect(self) -> None: """Disconnect from Slack.""" @@ -316,6 +327,8 @@ async def edit_message( chat_id: str, message_id: str, content: str, + *, + finalize: bool = False, ) -> SendResult: """Edit a previously sent Slack message.""" if not self._app: @@ -355,6 +368,7 @@ async def send_typing(self, chat_id: str, metadata=None) -> None: if not thread_ts: return # Can only set status in a thread context + self._active_status_threads[chat_id] = thread_ts try: await self._get_client(chat_id).assistant_threads_setStatus( channel_id=chat_id, @@ -366,6 +380,36 @@ async def send_typing(self, chat_id: str, metadata=None) -> None: # in an assistant-enabled context. Falls back to reactions. logger.debug("[Slack] assistant.threads.setStatus failed: %s", e) + async def stop_typing(self, chat_id: str) -> None: + """Clear the assistant thread status indicator.""" + if not self._app: + return + thread_ts = self._active_status_threads.pop(chat_id, None) + if not thread_ts: + return + try: + await self._get_client(chat_id).assistant_threads_setStatus( + channel_id=chat_id, + thread_ts=thread_ts, + status="", + ) + except Exception as e: + logger.debug("[Slack] assistant.threads.setStatus clear failed: %s", e) + + def _dm_top_level_threads_as_sessions(self) -> bool: + """Whether top-level Slack DMs get per-message session threads. + + Defaults to ``True`` so each visible DM reply thread is isolated as its + own Hermes session — matching the per-thread behavior channels already + have. Set ``platforms.slack.extra.dm_top_level_threads_as_sessions`` + to ``false`` in config.yaml to revert to the legacy behavior where all + top-level DMs share one continuous session. + """ + raw = self.config.extra.get("dm_top_level_threads_as_sessions") + if raw is None: + return True # default: each DM thread is its own session + return str(raw).strip().lower() in ("1", "true", "yes", "on") + def _resolve_thread_ts( self, reply_to: Optional[str] = None, @@ -563,6 +607,38 @@ async def _remove_reaction( logger.debug("[Slack] reactions.remove failed (%s): %s", emoji, e) return False + def _reactions_enabled(self) -> bool: + """Check if message reactions are enabled via config/env.""" + return os.getenv("SLACK_REACTIONS", "true").lower() not in ("false", "0", "no") + + async def on_processing_start(self, event: MessageEvent) -> None: + """Add an in-progress reaction when message processing begins.""" + if not self._reactions_enabled(): + return + ts = getattr(event, "message_id", None) + if not ts or ts not in self._reacting_message_ids: + return + channel_id = getattr(event.source, "chat_id", None) + if channel_id: + await self._add_reaction(channel_id, ts, "eyes") + + async def on_processing_complete(self, event: MessageEvent, outcome: ProcessingOutcome) -> None: + """Swap the in-progress reaction for a final success/failure reaction.""" + if not self._reactions_enabled(): + return + ts = getattr(event, "message_id", None) + if not ts or ts not in self._reacting_message_ids: + return + self._reacting_message_ids.discard(ts) + channel_id = getattr(event.source, "chat_id", None) + if not channel_id: + return + await self._remove_reaction(channel_id, ts, "eyes") + if outcome == ProcessingOutcome.SUCCESS: + await self._add_reaction(channel_id, ts, "white_check_mark") + elif outcome == ProcessingOutcome.FAILURE: + await self._add_reaction(channel_id, ts, "x") + # ----- User identity resolution ----- async def _resolve_user_name(self, user_id: str, chat_id: str = "") -> str: @@ -996,10 +1072,14 @@ async def _handle_slack_message(self, event: dict) -> None: # Build thread_ts for session keying. # In channels: fall back to ts so each top-level @mention starts a # new thread/session (the bot always replies in a thread). - # In DMs: only use the real thread_ts — top-level DMs should share - # one continuous session, threaded DMs get their own session. + # In DMs: fall back to ts so each top-level DM reply thread gets + # its own session key (matching channel behavior). Set + # dm_top_level_threads_as_sessions: false in config to revert to + # legacy single-session-per-DM-channel behavior. if is_dm: - thread_ts = event.get("thread_ts") or assistant_meta.get("thread_ts") # None for top-level DMs + thread_ts = event.get("thread_ts") or assistant_meta.get("thread_ts") + if not thread_ts and self._dm_top_level_threads_as_sessions(): + thread_ts = ts else: thread_ts = event.get("thread_ts") or ts # ts fallback for channels @@ -1167,6 +1247,12 @@ async def _handle_slack_message(self, event: dict) -> None: thread_id=thread_ts, ) + # Per-channel ephemeral prompt + from gateway.platforms.base import resolve_channel_prompt + _channel_prompt = resolve_channel_prompt( + self.config.extra, channel_id, None, + ) + msg_event = MessageEvent( text=text, message_type=msg_type, @@ -1176,22 +1262,18 @@ async def _handle_slack_message(self, event: dict) -> None: media_urls=media_urls, media_types=media_types, reply_to_message_id=thread_ts if thread_ts != ts else None, + channel_prompt=_channel_prompt, ) # Only react when bot is directly addressed (DM or @mention). # In listen-all channels (require_mention=false), reacting to every # casual message would be noisy. - _should_react = is_dm or is_mentioned - + _should_react = (is_dm or is_mentioned) and self._reactions_enabled() if _should_react: - await self._add_reaction(channel_id, ts, "eyes") + self._reacting_message_ids.add(ts) await self.handle_message(msg_event) - if _should_react: - await self._remove_reaction(channel_id, ts, "eyes") - await self._add_reaction(channel_id, ts, "white_check_mark") - # ----- Approval button support (Block Kit) ----- async def send_exec_approval( @@ -1568,11 +1650,9 @@ def _has_active_session_for_thread( async def _download_slack_file(self, url: str, ext: str, audio: bool = False, team_id: str = "") -> str: """Download a Slack file using the bot token for auth, with retry.""" - import asyncio import httpx bot_token = self._team_clients[team_id].token if team_id and team_id in self._team_clients else self.config.token - last_exc = None async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client: for attempt in range(3): @@ -1602,7 +1682,6 @@ async def _download_slack_file(self, url: str, ext: str, audio: bool = False, te from gateway.platforms.base import cache_image_from_bytes return cache_image_from_bytes(response.content, ext) except (httpx.TimeoutException, httpx.HTTPStatusError) as exc: - last_exc = exc if isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code < 429: raise if attempt < 2: @@ -1611,15 +1690,12 @@ async def _download_slack_file(self, url: str, ext: str, audio: bool = False, te await asyncio.sleep(1.5 * (attempt + 1)) continue raise - raise last_exc async def _download_slack_file_bytes(self, url: str, team_id: str = "") -> bytes: """Download a Slack file and return raw bytes, with retry.""" - import asyncio import httpx bot_token = self._team_clients[team_id].token if team_id and team_id in self._team_clients else self.config.token - last_exc = None async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client: for attempt in range(3): @@ -1631,7 +1707,6 @@ async def _download_slack_file_bytes(self, url: str, team_id: str = "") -> bytes response.raise_for_status() return response.content except (httpx.TimeoutException, httpx.HTTPStatusError) as exc: - last_exc = exc if isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code < 429: raise if attempt < 2: @@ -1640,7 +1715,6 @@ async def _download_slack_file_bytes(self, url: str, team_id: str = "") -> bytes await asyncio.sleep(1.5 * (attempt + 1)) continue raise - raise last_exc # ── Channel mention gating ───────────────────────────────────────────── diff --git a/gateway/platforms/telegram.py b/gateway/platforms/telegram.py index 439367b7d763..bec0d690a3be 100644 --- a/gateway/platforms/telegram.py +++ b/gateway/platforms/telegram.py @@ -11,6 +11,8 @@ import json import logging import os +import tempfile +import html as _html import re from typing import Dict, List, Optional, Any @@ -18,6 +20,10 @@ try: from telegram import Update, Bot, Message, InlineKeyboardButton, InlineKeyboardMarkup + try: + from telegram import LinkPreviewOptions + except ImportError: + LinkPreviewOptions = None from telegram.ext import ( Application, CommandHandler, @@ -36,6 +42,7 @@ Message = Any InlineKeyboardButton = Any InlineKeyboardMarkup = Any + LinkPreviewOptions = None Application = Any CommandHandler = Any CallbackQueryHandler = Any @@ -64,8 +71,10 @@ class _MockContextTypes: SendResult, cache_image_from_bytes, cache_audio_from_bytes, + cache_video_from_bytes, cache_document_from_bytes, resolve_proxy_url, + SUPPORTED_VIDEO_TYPES, SUPPORTED_DOCUMENT_TYPES, utf16_len, _prefix_within_utf16_limit, @@ -112,6 +121,84 @@ def _strip_mdv2(text: str) -> str: return cleaned +# --------------------------------------------------------------------------- +# Markdown table → code block conversion +# --------------------------------------------------------------------------- +# Telegram's MarkdownV2 has no table syntax — '|' is just an escaped literal, +# so pipe tables render as noisy backslash-pipe text with no alignment. +# Wrapping the table in a fenced code block makes Telegram render it as +# monospace preformatted text with columns intact. + +# Matches a GFM table delimiter row: optional outer pipes, cells containing +# only dashes (with optional leading/trailing colons for alignment) separated +# by '|'. Requires at least one internal '|' so lone '---' horizontal rules +# are NOT matched. +_TABLE_SEPARATOR_RE = re.compile( + r'^\s*\|?\s*:?-+:?\s*(?:\|\s*:?-+:?\s*){1,}\|?\s*$' +) + + +def _is_table_row(line: str) -> bool: + """Return True if *line* could plausibly be a table data row.""" + stripped = line.strip() + return bool(stripped) and '|' in stripped + + +def _wrap_markdown_tables(text: str) -> str: + """Wrap GFM-style pipe tables in ``` fences so Telegram renders them. + + Detected by a row containing '|' immediately followed by a delimiter + row matching :data:`_TABLE_SEPARATOR_RE`. Subsequent pipe-containing + non-blank lines are consumed as the table body and included in the + wrapped block. Tables inside existing fenced code blocks are left + alone. + """ + if '|' not in text or '-' not in text: + return text + + lines = text.split('\n') + out: list[str] = [] + in_fence = False + i = 0 + while i < len(lines): + line = lines[i] + stripped = line.lstrip() + + # Track existing fenced code blocks — never touch content inside. + if stripped.startswith('```'): + in_fence = not in_fence + out.append(line) + i += 1 + continue + if in_fence: + out.append(line) + i += 1 + continue + + # Look for a header row (contains '|') immediately followed by a + # delimiter row. + if ( + '|' in line + and i + 1 < len(lines) + and _TABLE_SEPARATOR_RE.match(lines[i + 1]) + ): + table_block = [line, lines[i + 1]] + j = i + 2 + while j < len(lines) and _is_table_row(lines[j]): + table_block.append(lines[j]) + j += 1 + out.append('```') + out.extend(table_block) + out.append('```') + i = j + continue + + out.append(line) + i += 1 + + return '\n'.join(out) + + class TelegramAdapter(BasePlatformAdapter): """ Telegram bot adapter. @@ -129,6 +216,7 @@ class TelegramAdapter(BasePlatformAdapter): # When a chunk is near this limit, a continuation is almost certain. _SPLIT_THRESHOLD = 4000 MEDIA_GROUP_WAIT_SECONDS = 0.8 + _GENERAL_TOPIC_THREAD_ID = "1" def __init__(self, config: PlatformConfig): super().__init__(config, Platform.TELEGRAM) @@ -137,6 +225,7 @@ def __init__(self, config: PlatformConfig): self._webhook_mode: bool = False self._mention_patterns = self._compile_mention_patterns() self._reply_to_mode: str = getattr(config, 'reply_to_mode', 'first') or 'first' + self._disable_link_previews: bool = self._coerce_bool_extra("disable_link_previews", False) # Buffer rapid/album photo updates so Telegram image bursts are handled # as a single MessageEvent instead of self-interrupting multiple turns. self._media_batch_delay_seconds = float(os.getenv("HERMES_TELEGRAM_MEDIA_BATCH_DELAY_SECONDS", "0.8")) @@ -163,6 +252,38 @@ def __init__(self, config: PlatformConfig): # Approval button state: message_id → session_key self._approval_state: Dict[int, str] = {} + @staticmethod + def _is_callback_user_authorized(user_id: str) -> bool: + """Return whether a Telegram inline-button caller may perform gated actions.""" + allowed_csv = os.getenv("TELEGRAM_ALLOWED_USERS", "").strip() + if not allowed_csv: + return True + allowed_ids = {uid.strip() for uid in allowed_csv.split(",") if uid.strip()} + return "*" in allowed_ids or user_id in allowed_ids + + @classmethod + def _metadata_thread_id(cls, metadata: Optional[Dict[str, Any]]) -> Optional[str]: + if not metadata: + return None + thread_id = metadata.get("thread_id") or metadata.get("message_thread_id") + return str(thread_id) if thread_id is not None else None + + @classmethod + def _message_thread_id_for_send(cls, thread_id: Optional[str]) -> Optional[int]: + if not thread_id or str(thread_id) == cls._GENERAL_TOPIC_THREAD_ID: + return None + return int(thread_id) + + @classmethod + def _message_thread_id_for_typing(cls, thread_id: Optional[str]) -> Optional[int]: + if not thread_id: + return None + return int(thread_id) + + @staticmethod + def _is_thread_not_found_error(error: Exception) -> bool: + return "thread not found" in str(error).lower() + def _fallback_ips(self) -> list[str]: """Return validated fallback IPs from config (populated by _apply_env_overrides).""" configured = self.config.extra.get("fallback_ips", []) if getattr(self.config, "extra", None) else [] @@ -193,6 +314,26 @@ def _looks_like_network_error(error: Exception) -> bool: pass return isinstance(error, OSError) + def _coerce_bool_extra(self, key: str, default: bool = False) -> bool: + value = self.config.extra.get(key) if getattr(self.config, "extra", None) else None + if value is None: + return default + if isinstance(value, str): + lowered = value.strip().lower() + if lowered in ("true", "1", "yes", "on"): + return True + if lowered in ("false", "0", "no", "off"): + return False + return default + return bool(value) + + def _link_preview_kwargs(self) -> Dict[str, Any]: + if not getattr(self, "_disable_link_previews", False): + return {} + if LinkPreviewOptions is not None: + return {"link_preview_options": LinkPreviewOptions(is_disabled=True)} + return {"disable_web_page_preview": True} + async def _handle_polling_network_error(self, error: Exception) -> None: """Reconnect polling after a transient network interruption. @@ -355,6 +496,13 @@ async def _create_dm_topic( "[%s] DM topic '%s' already exists in chat %s (will be mapped from incoming messages)", self.name, name, chat_id, ) + elif "not a forum" in error_text or "forums_disabled" in error_text: + logger.warning( + "[%s] Cannot create DM topic '%s' in chat %s: Topics mode is not enabled. " + "The user must open the DM with this bot in Telegram, tap the bot name " + "at the top, and enable 'Topics' in chat settings before topics can be created.", + self.name, name, chat_id, + ) else: logger.warning( "[%s] Failed to create DM topic '%s' in chat %s: %s", @@ -396,8 +544,23 @@ def _persist_dm_topic_thread_id(self, chat_id: int, topic_name: str, thread_id: break if changed: - with open(config_path, "w") as f: - _yaml.dump(config, f, default_flow_style=False, sort_keys=False) + fd, tmp_path = tempfile.mkstemp( + dir=str(config_path.parent), + suffix=".tmp", + prefix=".config_", + ) + try: + with os.fdopen(fd, "w", encoding="utf-8") as f: + _yaml.dump(config, f, default_flow_style=False, sort_keys=False) + f.flush() + os.fsync(f.fileno()) + os.replace(tmp_path, config_path) + except BaseException: + try: + os.unlink(tmp_path) + except OSError: + pass + raise logger.info( "[%s] Persisted thread_id=%s for topic '%s' in config.yaml", self.name, thread_id, topic_name, @@ -540,7 +703,7 @@ def _env_float(name: str, default: float) -> float: "write_timeout": _env_float("HERMES_TELEGRAM_HTTP_WRITE_TIMEOUT", 20.0), } - proxy_url = resolve_proxy_url() + proxy_url = resolve_proxy_url("TELEGRAM_PROXY") disable_fallback = (os.getenv("HERMES_TELEGRAM_DISABLE_FALLBACK_IPS", "").strip().lower() in ("1", "true", "yes", "on")) fallback_ips = self._fallback_ips() if not fallback_ips: @@ -606,14 +769,14 @@ def _env_float(name: str, default: float) -> float: from telegram.error import NetworkError, TimedOut except ImportError: NetworkError = TimedOut = OSError # type: ignore[misc,assignment] - _max_connect = 3 + _max_connect = 8 for _attempt in range(_max_connect): try: await self._app.initialize() break except (NetworkError, TimedOut, OSError) as init_err: if _attempt < _max_connect - 1: - wait = 2 ** _attempt + wait = min(2 ** _attempt, 15) logger.warning( "[%s] Connect attempt %d/%d failed: %s — retrying in %ds", self.name, _attempt + 1, _max_connect, init_err, wait, @@ -631,8 +794,28 @@ def _env_float(name: str, default: float) -> float: # Telegram pushes updates to our HTTP endpoint. This # enables cloud platforms (Fly.io, Railway) to auto-wake # suspended machines on inbound HTTP traffic. + # + # SECURITY: TELEGRAM_WEBHOOK_SECRET is REQUIRED. Without it, + # python-telegram-bot passes secret_token=None and the + # webhook endpoint accepts any HTTP POST — attackers can + # inject forged updates as if from Telegram. Refuse to + # start rather than silently run in fail-open mode. + # See GHSA-3vpc-7q5r-276h. webhook_port = int(os.getenv("TELEGRAM_WEBHOOK_PORT", "8443")) - webhook_secret = os.getenv("TELEGRAM_WEBHOOK_SECRET", "").strip() or None + webhook_secret = os.getenv("TELEGRAM_WEBHOOK_SECRET", "").strip() + if not webhook_secret: + raise RuntimeError( + "TELEGRAM_WEBHOOK_SECRET is required when " + "TELEGRAM_WEBHOOK_URL is set. Without it, the " + "webhook endpoint accepts forged updates from " + "anyone who can reach it — see " + "https://github.com/NousResearch/hermes-agent/" + "security/advisories/GHSA-3vpc-7q5r-276h.\n\n" + "Generate a secret and set it in your .env:\n" + " export TELEGRAM_WEBHOOK_SECRET=\"$(openssl rand -hex 32)\"\n\n" + "Then register it with Telegram when setting the " + "webhook via setWebhook's secret_token parameter." + ) from urllib.parse import urlparse webhook_path = urlparse(webhook_url).path or "/telegram" @@ -814,7 +997,7 @@ async def send( ] message_ids = [] - thread_id = metadata.get("thread_id") if metadata else None + thread_id = self._metadata_thread_id(metadata) try: from telegram.error import NetworkError as _NetErr @@ -834,7 +1017,7 @@ async def send( for i, chunk in enumerate(chunks): should_thread = self._should_thread_reply(reply_to, i) reply_to_id = int(reply_to) if should_thread else None - effective_thread_id = int(thread_id) if thread_id else None + effective_thread_id = self._message_thread_id_for_send(thread_id) msg = None for _send_attempt in range(3): @@ -847,6 +1030,7 @@ async def send( parse_mode=ParseMode.MARKDOWN_V2, reply_to_message_id=reply_to_id, message_thread_id=effective_thread_id, + **self._link_preview_kwargs(), ) except Exception as md_error: # Markdown parsing failed, try plain text @@ -859,6 +1043,7 @@ async def send( parse_mode=None, reply_to_message_id=reply_to_id, message_thread_id=effective_thread_id, + **self._link_preview_kwargs(), ) else: raise @@ -869,8 +1054,7 @@ async def send( # (not transient network issues). Detect and handle # specific cases instead of blindly retrying. if _BadReq and isinstance(send_err, _BadReq): - err_lower = str(send_err).lower() - if "thread not found" in err_lower and effective_thread_id is not None: + if self._is_thread_not_found_error(send_err) and effective_thread_id is not None: # Thread doesn't exist — retry without # message_thread_id so the message still # reaches the chat. @@ -880,6 +1064,7 @@ async def send( ) effective_thread_id = None continue + err_lower = str(send_err).lower() if "message to be replied not found" in err_lower and reply_to_id is not None: # Original message was deleted before we # could reply — clear reply target and retry @@ -941,6 +1126,8 @@ async def edit_message( chat_id: str, message_id: str, content: str, + *, + finalize: bool = False, ) -> SendResult: """Edit a previously sent Telegram message.""" if not self._bot: @@ -1046,6 +1233,7 @@ async def send_update_prompt( text=text, parse_mode=ParseMode.MARKDOWN, reply_markup=keyboard, + **self._link_preview_kwargs(), ) return SendResult(success=True, message_id=str(msg.message_id)) except Exception as e: @@ -1068,15 +1256,13 @@ async def send_exec_approval( try: cmd_preview = command[:3800] + "..." if len(command) > 3800 else command text = ( - f"⚠️ *Command Approval Required*\n\n" - f"`{cmd_preview}`\n\n" - f"Reason: {description}" + f"⚠️ Command Approval Required\n\n" + f"
{_html.escape(cmd_preview)}
\n\n" + f"Reason: {_html.escape(description)}" ) # Resolve thread context for thread replies - thread_id = None - if metadata: - thread_id = metadata.get("thread_id") or metadata.get("message_thread_id") + thread_id = self._metadata_thread_id(metadata) # We'll use the message_id as part of callback_data to look up session_key # Send a placeholder first, then update — or use a counter. @@ -1100,11 +1286,13 @@ async def send_exec_approval( kwargs: Dict[str, Any] = { "chat_id": int(chat_id), "text": text, - "parse_mode": ParseMode.MARKDOWN, + "parse_mode": ParseMode.HTML, "reply_markup": keyboard, + **self._link_preview_kwargs(), } - if thread_id: - kwargs["message_thread_id"] = int(thread_id) + message_thread_id = self._message_thread_id_for_send(thread_id) + if message_thread_id is not None: + kwargs["message_thread_id"] = message_thread_id msg = await self._bot.send_message(**kwargs) @@ -1172,6 +1360,7 @@ def get_label(slug): parse_mode=ParseMode.MARKDOWN, reply_markup=keyboard, message_thread_id=int(thread_id) if thread_id else None, + **self._link_preview_kwargs(), ) # Store picker state keyed by chat_id @@ -1440,12 +1629,9 @@ async def _handle_callback_query( # Only authorized users may click approval buttons. caller_id = str(getattr(query.from_user, "id", "")) - allowed_csv = os.getenv("TELEGRAM_ALLOWED_USERS", "").strip() - if allowed_csv: - allowed_ids = {uid.strip() for uid in allowed_csv.split(",") if uid.strip()} - if "*" not in allowed_ids and caller_id not in allowed_ids: - await query.answer(text="⛔ You are not authorized to approve commands.") - return + if not self._is_callback_user_authorized(caller_id): + await query.answer(text="⛔ You are not authorized to approve commands.") + return session_key = self._approval_state.pop(approval_id, None) if not session_key: @@ -1490,6 +1676,10 @@ async def _handle_callback_query( if not data.startswith("update_prompt:"): return answer = data.split(":", 1)[1] # "y" or "n" + caller_id = str(getattr(query.from_user, "id", "")) + if not self._is_callback_user_authorized(caller_id): + await query.answer(text="⛔ You are not authorized to answer update prompts.") + return await query.answer(text=f"Sent '{answer}' to the update process.") # Edit the message to show the choice and remove buttons label = "Yes" if answer == "y" else "No" @@ -1514,6 +1704,21 @@ async def _handle_callback_query( except Exception as exc: logger.error("Failed to write update response from callback: %s", exc) + def _missing_media_path_error(self, label: str, path: str) -> str: + """Build an actionable file-not-found error for gateway MEDIA delivery. + + Paths like /workspace/... or /output/... often only exist inside the + Docker sandbox, while the gateway process runs on the host. + """ + error = f"{label} file not found: {path}" + if path.startswith(("/workspace/", "/output/", "/outputs/")): + error += ( + " (path may only exist inside the Docker sandbox. " + "Bind-mount a host directory and emit the host-visible " + "path in MEDIA: for gateway file delivery.)" + ) + return error + async def send_voice( self, chat_id: str, @@ -1528,30 +1733,29 @@ async def send_voice( return SendResult(success=False, error="Not connected") try: - import os if not os.path.exists(audio_path): - return SendResult(success=False, error=f"Audio file not found: {audio_path}") + return SendResult(success=False, error=self._missing_media_path_error("Audio", audio_path)) with open(audio_path, "rb") as audio_file: # .ogg files -> send as voice (round playable bubble) if audio_path.endswith((".ogg", ".opus")): - _voice_thread = metadata.get("thread_id") if metadata else None + _voice_thread = self._metadata_thread_id(metadata) msg = await self._bot.send_voice( chat_id=int(chat_id), voice=audio_file, caption=caption[:1024] if caption else None, reply_to_message_id=int(reply_to) if reply_to else None, - message_thread_id=int(_voice_thread) if _voice_thread else None, + message_thread_id=self._message_thread_id_for_send(_voice_thread), ) else: # .mp3 and others -> send as audio file - _audio_thread = metadata.get("thread_id") if metadata else None + _audio_thread = self._metadata_thread_id(metadata) msg = await self._bot.send_audio( chat_id=int(chat_id), audio=audio_file, caption=caption[:1024] if caption else None, reply_to_message_id=int(reply_to) if reply_to else None, - message_thread_id=int(_audio_thread) if _audio_thread else None, + message_thread_id=self._message_thread_id_for_send(_audio_thread), ) return SendResult(success=True, message_id=str(msg.message_id)) except Exception as e: @@ -1577,18 +1781,17 @@ async def send_image_file( return SendResult(success=False, error="Not connected") try: - import os if not os.path.exists(image_path): - return SendResult(success=False, error=f"Image file not found: {image_path}") + return SendResult(success=False, error=self._missing_media_path_error("Image", image_path)) - _thread = metadata.get("thread_id") if metadata else None + _thread = self._metadata_thread_id(metadata) with open(image_path, "rb") as image_file: msg = await self._bot.send_photo( chat_id=int(chat_id), photo=image_file, caption=caption[:1024] if caption else None, reply_to_message_id=int(reply_to) if reply_to else None, - message_thread_id=int(_thread) if _thread else None, + message_thread_id=self._message_thread_id_for_send(_thread), ) return SendResult(success=True, message_id=str(msg.message_id)) except Exception as e: @@ -1616,10 +1819,10 @@ async def send_document( try: if not os.path.exists(file_path): - return SendResult(success=False, error=f"File not found: {file_path}") + return SendResult(success=False, error=self._missing_media_path_error("File", file_path)) display_name = file_name or os.path.basename(file_path) - _thread = metadata.get("thread_id") if metadata else None + _thread = self._metadata_thread_id(metadata) with open(file_path, "rb") as f: msg = await self._bot.send_document( @@ -1628,7 +1831,7 @@ async def send_document( filename=display_name, caption=caption[:1024] if caption else None, reply_to_message_id=int(reply_to) if reply_to else None, - message_thread_id=int(_thread) if _thread else None, + message_thread_id=self._message_thread_id_for_send(_thread), ) return SendResult(success=True, message_id=str(msg.message_id)) except Exception as e: @@ -1650,16 +1853,16 @@ async def send_video( try: if not os.path.exists(video_path): - return SendResult(success=False, error=f"Video file not found: {video_path}") + return SendResult(success=False, error=self._missing_media_path_error("Video", video_path)) - _thread = metadata.get("thread_id") if metadata else None + _thread = self._metadata_thread_id(metadata) with open(video_path, "rb") as f: msg = await self._bot.send_video( chat_id=int(chat_id), video=f, caption=caption[:1024] if caption else None, reply_to_message_id=int(reply_to) if reply_to else None, - message_thread_id=int(_thread) if _thread else None, + message_thread_id=self._message_thread_id_for_send(_thread), ) return SendResult(success=True, message_id=str(msg.message_id)) except Exception as e: @@ -1689,13 +1892,13 @@ async def send_image( try: # Telegram can send photos directly from URLs (up to ~5MB) - _photo_thread = metadata.get("thread_id") if metadata else None + _photo_thread = self._metadata_thread_id(metadata) msg = await self._bot.send_photo( chat_id=int(chat_id), photo=image_url, caption=caption[:1024] if caption else None, # Telegram caption limit reply_to_message_id=int(reply_to) if reply_to else None, - message_thread_id=int(_photo_thread) if _photo_thread else None, + message_thread_id=self._message_thread_id_for_send(_photo_thread), ) return SendResult(success=True, message_id=str(msg.message_id)) except Exception as e: @@ -1718,6 +1921,7 @@ async def send_image( photo=image_data, caption=caption[:1024] if caption else None, reply_to_message_id=int(reply_to) if reply_to else None, + message_thread_id=self._message_thread_id_for_send(_photo_thread), ) return SendResult(success=True, message_id=str(msg.message_id)) except Exception as e2: @@ -1743,13 +1947,13 @@ async def send_animation( return SendResult(success=False, error="Not connected") try: - _anim_thread = metadata.get("thread_id") if metadata else None + _anim_thread = self._metadata_thread_id(metadata) msg = await self._bot.send_animation( chat_id=int(chat_id), animation=animation_url, caption=caption[:1024] if caption else None, reply_to_message_id=int(reply_to) if reply_to else None, - message_thread_id=int(_anim_thread) if _anim_thread else None, + message_thread_id=self._message_thread_id_for_send(_anim_thread), ) return SendResult(success=True, message_id=str(msg.message_id)) except Exception as e: @@ -1766,12 +1970,23 @@ async def send_typing(self, chat_id: str, metadata: Optional[Dict[str, Any]] = N """Send typing indicator.""" if self._bot: try: - _typing_thread = metadata.get("thread_id") if metadata else None - await self._bot.send_chat_action( - chat_id=int(chat_id), - action="typing", - message_thread_id=int(_typing_thread) if _typing_thread else None, - ) + _typing_thread = self._metadata_thread_id(metadata) + message_thread_id = self._message_thread_id_for_typing(_typing_thread) + try: + await self._bot.send_chat_action( + chat_id=int(chat_id), + action="typing", + message_thread_id=message_thread_id, + ) + except Exception as e: + if message_thread_id is not None and self._is_thread_not_found_error(e): + await self._bot.send_chat_action( + chat_id=int(chat_id), + action="typing", + message_thread_id=None, + ) + else: + raise except Exception as e: # Typing failures are non-fatal; log at debug level only. logger.debug( @@ -1839,6 +2054,12 @@ def _ph(value: str) -> str: text = content + # 0) Pre-wrap GFM-style pipe tables in ``` fences. Telegram can't + # render tables natively, but fenced code blocks render as + # monospace preformatted text with columns intact. The wrapped + # tables then flow through step (1) below as protected regions. + text = _wrap_markdown_tables(text) + # 1) Protect fenced code blocks (``` ... ```) # Per MarkdownV2 spec, \ and ` inside pre/code must be escaped. def _protect_fenced(m): @@ -1872,7 +2093,7 @@ def _convert_link(m): url = m.group(2).replace('\\', '\\\\').replace(')', '\\)') return _ph(f'[{display}]({url})') - text = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', _convert_link, text) + text = re.sub(r'\[([^\]]+)\]\(([^()]*(?:\([^()]*\)[^()]*)*)\)', _convert_link, text) # 4) Convert markdown headers (## Title) → bold *Title* def _convert_header(m): @@ -1916,9 +2137,20 @@ def _convert_header(m): ) # 9) Convert blockquotes: > at line start → protect > from escaping + # Handle both regular blockquotes (> text) and expandable blockquotes + # (Telegram MarkdownV2: **> for expandable start, || to end the quote) + def _convert_blockquote(m): + prefix = m.group(1) # >, >>, >>>, **>, or **>> etc. + content = m.group(2) + # Check if content ends with || (expandable blockquote end marker) + # In this case, preserve the trailing || unescaped for Telegram + if prefix.startswith('**') and content.endswith('||'): + return _ph(f'{prefix} {_escape_mdv2(content[:-2])}||') + return _ph(f'{prefix} {_escape_mdv2(content)}') + text = re.sub( - r'^(>{1,3}) (.+)$', - lambda m: _ph(m.group(1) + ' ' + _escape_mdv2(m.group(2))), + r'^((?:\*\*)?>{1,3}) (.+)$', + _convert_blockquote, text, flags=re.MULTILINE, ) @@ -1991,6 +2223,27 @@ def _telegram_free_response_chats(self) -> set[str]: return {str(part).strip() for part in raw if str(part).strip()} return {part.strip() for part in str(raw).split(",") if part.strip()} + def _telegram_ignored_threads(self) -> set[int]: + raw = self.config.extra.get("ignored_threads") + if raw is None: + raw = os.getenv("TELEGRAM_IGNORED_THREADS", "") + + if isinstance(raw, list): + values = raw + else: + values = str(raw).split(",") + + ignored: set[int] = set() + for value in values: + text = str(value).strip() + if not text: + continue + try: + ignored.add(int(text)) + except (TypeError, ValueError): + logger.warning("[%s] Ignoring invalid Telegram thread id: %r", self.name, value) + return ignored + def _compile_mention_patterns(self) -> List[re.Pattern]: """Compile optional regex wake-word patterns for group triggers.""" patterns = self.config.extra.get("mention_patterns") @@ -2048,22 +2301,27 @@ def _message_mentions_bot(self, message: Message) -> bool: bot_username = (getattr(self._bot, "username", None) or "").lstrip("@").lower() bot_id = getattr(self._bot, "id", None) + expected = f"@{bot_username}" if bot_username else None def _iter_sources(): yield getattr(message, "text", None) or "", getattr(message, "entities", None) or [] yield getattr(message, "caption", None) or "", getattr(message, "caption_entities", None) or [] + # Telegram parses mentions server-side and emits MessageEntity objects + # (type=mention for @username, type=text_mention for @FirstName targeting + # a user without a public username). Only those entities are authoritative — + # raw substring matches like "foo@hermes_bot.example" are not mentions + # (bug #12545). Entities also correctly handle @handles inside URLs, code + # blocks, and quoted text, where a regex scan would over-match. for source_text, entities in _iter_sources(): - if bot_username and f"@{bot_username}" in source_text.lower(): - return True for entity in entities: entity_type = str(getattr(entity, "type", "")).split(".")[-1].lower() - if entity_type == "mention" and bot_username: + if entity_type == "mention" and expected: offset = int(getattr(entity, "offset", -1)) length = int(getattr(entity, "length", 0)) if offset < 0 or length <= 0: continue - if source_text[offset:offset + length].strip().lower() == f"@{bot_username}": + if source_text[offset:offset + length].strip().lower() == expected: return True elif entity_type == "text_mention": user = getattr(entity, "user", None) @@ -2095,19 +2353,30 @@ def _should_process_message(self, message: Message, *, is_command: bool = False) DMs remain unrestricted. Group/supergroup messages are accepted when: - the chat is explicitly allowlisted in ``free_response_chats`` - ``require_mention`` is disabled - - the message is a command - the message replies to the bot - the bot is @mentioned - the text/caption matches a configured regex wake-word pattern + + When ``require_mention`` is enabled, slash commands are not given + special treatment — they must pass the same mention/reply checks + as any other group message. Users can still trigger commands via + the Telegram bot menu (``/command@botname``) or by explicitly + mentioning the bot (``@botname /command``), both of which are + recognised as mentions by :meth:`_message_mentions_bot`. """ if not self._is_group_chat(message): return True + thread_id = getattr(message, "message_thread_id", None) + if thread_id is not None: + try: + if int(thread_id) in self._telegram_ignored_threads(): + return False + except (TypeError, ValueError): + logger.warning("[%s] Ignoring non-numeric Telegram message_thread_id: %r", self.name, thread_id) if str(getattr(getattr(message, "chat", None), "id", "")) in self._telegram_free_response_chats(): return True if not self._telegram_require_mention(): return True - if is_command: - return True if self._is_reply_to_bot(message): return True if self._message_mentions_bot(message): @@ -2126,7 +2395,7 @@ async def _handle_text_message(self, update: Update, context: ContextTypes.DEFAU if not self._should_process_message(update.message): return - event = self._build_message_event(update.message, MessageType.TEXT) + event = self._build_message_event(update.message, MessageType.TEXT, update_id=update.update_id) event.text = self._clean_bot_trigger_text(event.text) self._enqueue_text_event(event) @@ -2137,7 +2406,7 @@ async def _handle_command(self, update: Update, context: ContextTypes.DEFAULT_TY if not self._should_process_message(update.message, is_command=True): return - event = self._build_message_event(update.message, MessageType.COMMAND) + event = self._build_message_event(update.message, MessageType.COMMAND, update_id=update.update_id) await self.handle_message(event) async def _handle_location_message(self, update: Update, context: ContextTypes.DEFAULT_TYPE) -> None: @@ -2173,7 +2442,7 @@ async def _handle_location_message(self, update: Update, context: ContextTypes.D parts.append(f"Map: https://www.google.com/maps/search/?api=1&query={lat},{lon}") parts.append("Ask what they'd like to find nearby (restaurants, cafes, etc.) and any preferences.") - event = self._build_message_event(msg, MessageType.LOCATION) + event = self._build_message_event(msg, MessageType.LOCATION, update_id=update.update_id) event.text = "\n".join(parts) await self.handle_message(event) @@ -2324,7 +2593,7 @@ async def _handle_media_message(self, update: Update, context: ContextTypes.DEFA else: msg_type = MessageType.DOCUMENT - event = self._build_message_event(msg, msg_type) + event = self._build_message_event(msg, msg_type, update_id=update.update_id) # Add caption as text if msg.caption: @@ -2390,6 +2659,23 @@ async def _handle_media_message(self, update: Update, context: ContextTypes.DEFA except Exception as e: logger.warning("[Telegram] Failed to cache audio: %s", e, exc_info=True) + elif msg.video: + try: + file_obj = await msg.video.get_file() + video_bytes = await file_obj.download_as_bytearray() + ext = ".mp4" + if getattr(file_obj, "file_path", None): + for candidate in SUPPORTED_VIDEO_TYPES: + if file_obj.file_path.lower().endswith(candidate): + ext = candidate + break + cached_path = cache_video_from_bytes(bytes(video_bytes), ext=ext) + event.media_urls = [cached_path] + event.media_types = [SUPPORTED_VIDEO_TYPES.get(ext, "video/mp4")] + logger.info("[Telegram] Cached user video at %s", cached_path) + except Exception as e: + logger.warning("[Telegram] Failed to cache video: %s", e, exc_info=True) + # Download document files to cache for agent processing elif msg.document: doc = msg.document @@ -2406,6 +2692,21 @@ async def _handle_media_message(self, update: Update, context: ContextTypes.DEFA mime_to_ext = {v: k for k, v in SUPPORTED_DOCUMENT_TYPES.items()} ext = mime_to_ext.get(doc.mime_type, "") + if not ext and doc.mime_type: + video_mime_to_ext = {v: k for k, v in SUPPORTED_VIDEO_TYPES.items()} + ext = video_mime_to_ext.get(doc.mime_type, "") + + if ext in SUPPORTED_VIDEO_TYPES: + file_obj = await doc.get_file() + video_bytes = await file_obj.download_as_bytearray() + cached_path = cache_video_from_bytes(bytes(video_bytes), ext=ext) + event.media_urls = [cached_path] + event.media_types = [SUPPORTED_VIDEO_TYPES[ext]] + event.message_type = MessageType.VIDEO + logger.info("[Telegram] Cached user video document at %s", cached_path) + await self.handle_message(event) + return + # Check if supported if ext not in SUPPORTED_DOCUMENT_TYPES: supported_list = ", ".join(sorted(SUPPORTED_DOCUMENT_TYPES.keys())) @@ -2544,13 +2845,11 @@ async def _handle_sticker(self, msg: Message, event: "MessageEvent") -> None: logger.info("[Telegram] Analyzing sticker at %s", cached_path) from tools.vision_tools import vision_analyze_tool - import json as _json - result_json = await vision_analyze_tool( image_url=cached_path, user_prompt=STICKER_VISION_PROMPT, ) - result = _json.loads(result_json) + result = json.loads(result_json) if result.get("success"): description = result.get("analysis", "a sticker") @@ -2663,8 +2962,19 @@ def _cache_dm_topic_from_message(self, chat_id: str, thread_id: str, topic_name: self.name, cache_key, thread_id, ) - def _build_message_event(self, message: Message, msg_type: MessageType) -> MessageEvent: - """Build a MessageEvent from a Telegram message.""" + def _build_message_event( + self, + message: Message, + msg_type: MessageType, + update_id: Optional[int] = None, + ) -> MessageEvent: + """Build a MessageEvent from a Telegram message. + + ``update_id`` is the ``Update.update_id`` from PTB; passing it through + lets ``/restart`` record the triggering offset so the new gateway + process can advance past it (prevents ``/restart`` being re-delivered + when PTB's graceful-shutdown ACK fails). + """ chat = message.chat user = message.from_user @@ -2677,7 +2987,9 @@ def _build_message_event(self, message: Message, msg_type: MessageType) -> Messa # Resolve DM topic name and skill binding thread_id_raw = message.message_thread_id - thread_id_str = str(thread_id_raw) if thread_id_raw else None + thread_id_str = str(thread_id_raw) if thread_id_raw is not None else None + if chat_type == "group" and thread_id_str is None and getattr(chat, "is_forum", False): + thread_id_str = self._GENERAL_TOPIC_THREAD_ID chat_topic = None topic_skill = None @@ -2713,8 +3025,8 @@ def _build_message_event(self, message: Message, msg_type: MessageType) -> Messa chat_id=str(chat.id), chat_name=chat.title or (chat.full_name if hasattr(chat, "full_name") else None), chat_type=chat_type, - user_id=str(user.id) if user else None, - user_name=user.full_name if user else None, + user_id=str(user.id) if user else (str(chat.id) if chat_type == "dm" else None), + user_name=user.full_name if user else (chat.full_name if hasattr(chat, "full_name") and chat_type == "dm" else None), thread_id=thread_id_str, chat_topic=chat_topic, ) @@ -2726,15 +3038,26 @@ def _build_message_event(self, message: Message, msg_type: MessageType) -> Messa reply_to_id = str(message.reply_to_message.message_id) reply_to_text = message.reply_to_message.text or message.reply_to_message.caption or None + # Per-channel/topic ephemeral prompt + from gateway.platforms.base import resolve_channel_prompt + _chat_id_str = str(chat.id) + _channel_prompt = resolve_channel_prompt( + self.config.extra, + thread_id_str or _chat_id_str, + _chat_id_str if thread_id_str else None, + ) + return MessageEvent( text=message.text or "", message_type=msg_type, source=source, raw_message=message, message_id=str(message.message_id), + platform_update_id=update_id, reply_to_message_id=reply_to_id, reply_to_text=reply_to_text, auto_skill=topic_skill, + channel_prompt=_channel_prompt, timestamp=message.date, ) diff --git a/gateway/platforms/telegram_network.py b/gateway/platforms/telegram_network.py index d9832a269623..ed2d60d79729 100644 --- a/gateway/platforms/telegram_network.py +++ b/gateway/platforms/telegram_network.py @@ -12,7 +12,6 @@ import asyncio import ipaddress import logging -import os import socket from typing import Iterable, Optional @@ -47,7 +46,7 @@ def _resolve_proxy_url() -> str | None: # Delegate to shared implementation (env vars + macOS system proxy detection) from gateway.platforms.base import resolve_proxy_url - return resolve_proxy_url() + return resolve_proxy_url("TELEGRAM_PROXY") class TelegramFallbackTransport(httpx.AsyncBaseTransport): diff --git a/gateway/platforms/webhook.py b/gateway/platforms/webhook.py index dfe7a70f3f22..e3a736a451dc 100644 --- a/gateway/platforms/webhook.py +++ b/gateway/platforms/webhook.py @@ -13,6 +13,10 @@ - skills: optional list of skills to load for the agent - deliver: where to send the response (github_comment, telegram, etc.) - deliver_extra: additional delivery config (repo, pr_number, chat_id) + - deliver_only: if true, skip the agent — the rendered prompt IS the + message that gets delivered. Use for external push notifications + (Supabase, monitoring alerts, inter-agent pings) where zero LLM cost + and sub-second delivery matter more than agent reasoning. Security: - HMAC secret is required per route (validated at startup) @@ -27,7 +31,6 @@ import hmac import json import logging -import os import re import subprocess import time @@ -123,6 +126,19 @@ async def connect(self) -> bool: f"For testing without auth, set secret to '{_INSECURE_NO_AUTH}'." ) + # deliver_only routes bypass the agent — the POST body becomes a + # direct push notification via the configured delivery target. + # Validate up-front so misconfiguration surfaces at startup rather + # than on the first webhook POST. + if route.get("deliver_only"): + deliver = route.get("deliver", "log") + if not deliver or deliver == "log": + raise ValueError( + f"[webhook] Route '{name}' has deliver_only=true but " + f"deliver is '{deliver}'. Direct delivery requires a " + f"real target (telegram, discord, slack, github_comment, etc.)." + ) + app = web.Application() app.router.add_get("/health", self._handle_health) app.router.add_post("/webhooks/{route_name}", self._handle_webhook) @@ -204,6 +220,7 @@ async def send( "wecom_callback", "weixin", "bluebubbles", + "qqbot", ): return await self._deliver_cross_platform( deliver_type, content, delivery @@ -296,24 +313,14 @@ async def _handle_webhook(self, request: "web.Request") -> "web.Response": {"error": "Payload too large"}, status=413 ) - # ── Rate limiting ──────────────────────────────────────── - now = time.time() - window = self._rate_counts.setdefault(route_name, []) - window[:] = [t for t in window if now - t < 60] - if len(window) >= self._rate_limit: - return web.json_response( - {"error": "Rate limit exceeded"}, status=429 - ) - window.append(now) - - # Read body + # Read body (must be done before any validation) try: raw_body = await request.read() except Exception as e: logger.error("[webhook] Failed to read body: %s", e) return web.json_response({"error": "Bad request"}, status=400) - # Validate HMAC signature (skip for INSECURE_NO_AUTH testing mode) + # Validate HMAC signature FIRST (skip for INSECURE_NO_AUTH testing mode) secret = route_config.get("secret", self._global_secret) if secret and secret != _INSECURE_NO_AUTH: if not self._validate_signature(request, raw_body, secret): @@ -324,6 +331,16 @@ async def _handle_webhook(self, request: "web.Request") -> "web.Response": {"error": "Invalid signature"}, status=401 ) + # ── Rate limiting (after auth) ─────────────────────────── + now = time.time() + window = self._rate_counts.setdefault(route_name, []) + window[:] = [t for t in window if now - t < 60] + if len(window) >= self._rate_limit: + return web.json_response( + {"error": "Rate limit exceeded"}, status=429 + ) + window.append(now) + # Parse payload try: payload = json.loads(raw_body) @@ -419,6 +436,64 @@ async def _handle_webhook(self, request: "web.Request") -> "web.Response": ) self._seen_deliveries[delivery_id] = now + # ── Direct delivery mode (deliver_only) ───────────────── + # Skip the agent entirely — the rendered prompt IS the message we + # deliver. Use case: external services (Supabase, monitoring, + # cron jobs, other agents) that need to push a plain notification + # to a user's chat with zero LLM cost. Reuses the same HMAC auth, + # rate limiting, idempotency, and template rendering as agent mode. + if route_config.get("deliver_only"): + delivery = { + "deliver": route_config.get("deliver", "log"), + "deliver_extra": self._render_delivery_extra( + route_config.get("deliver_extra", {}), payload + ), + "payload": payload, + } + logger.info( + "[webhook] direct-deliver event=%s route=%s target=%s msg_len=%d delivery=%s", + event_type, + route_name, + delivery["deliver"], + len(prompt), + delivery_id, + ) + try: + result = await self._direct_deliver(prompt, delivery) + except Exception: + logger.exception( + "[webhook] direct-deliver failed route=%s delivery=%s", + route_name, + delivery_id, + ) + return web.json_response( + {"status": "error", "error": "Delivery failed", "delivery_id": delivery_id}, + status=502, + ) + + if result.success: + return web.json_response( + { + "status": "delivered", + "route": route_name, + "target": delivery["deliver"], + "delivery_id": delivery_id, + }, + status=200, + ) + # Delivery attempted but target rejected it — surface as 502 + # with a generic error (don't leak adapter-level detail). + logger.warning( + "[webhook] direct-deliver target rejected route=%s target=%s error=%s", + route_name, + delivery["deliver"], + result.error, + ) + return web.json_response( + {"status": "error", "error": "Delivery failed", "delivery_id": delivery_id}, + status=502, + ) + # Use delivery_id in session key so concurrent webhooks on the # same route get independent agent runs (not queued/interrupted). session_chat_id = f"webhook:{route_name}:{delivery_id}" @@ -572,6 +647,34 @@ def _render_delivery_extra( # Response delivery # ------------------------------------------------------------------ + async def _direct_deliver( + self, content: str, delivery: dict + ) -> SendResult: + """Deliver *content* directly without invoking the agent. + + Used by ``deliver_only`` routes: the rendered template becomes the + literal message body, and we dispatch to the same delivery helpers + that the agent-mode ``send()`` flow uses. All target types that + work in agent mode work here — Telegram, Discord, Slack, GitHub + PR comments, etc. + """ + deliver_type = delivery.get("deliver", "log") + + if deliver_type == "log": + # Shouldn't reach here — startup validation rejects deliver_only + # with deliver=log — but guard defensively. + logger.info("[webhook] direct-deliver log-only: %s", content[:200]) + return SendResult(success=True) + + if deliver_type == "github_comment": + return await self._deliver_github_comment(content, delivery) + + # Fall through to the cross-platform dispatcher, which validates the + # target name and routes via the gateway runner. + return await self._deliver_cross_platform( + deliver_type, content, delivery + ) + async def _deliver_github_comment( self, content: str, delivery: dict ) -> SendResult: diff --git a/gateway/platforms/wecom.py b/gateway/platforms/wecom.py index 0249ae6751e5..7ba0fa21b901 100644 --- a/gateway/platforms/wecom.py +++ b/gateway/platforms/wecom.py @@ -37,7 +37,6 @@ import mimetypes import os import re -import time import uuid from datetime import datetime, timezone from pathlib import Path @@ -181,6 +180,8 @@ def __init__(self, config: PlatformConfig): self._text_batch_split_delay_seconds = float(os.getenv("HERMES_WECOM_TEXT_BATCH_SPLIT_DELAY_SECONDS", "2.0")) self._pending_text_batches: Dict[str, MessageEvent] = {} self._pending_text_batch_tasks: Dict[str, asyncio.Task] = {} + self._device_id = uuid.uuid4().hex + self._last_chat_req_ids: Dict[str, str] = {} # ------------------------------------------------------------------ # Connection lifecycle @@ -278,7 +279,11 @@ async def _open_connection(self) -> None: { "cmd": APP_CMD_SUBSCRIBE, "headers": {"req_id": req_id}, - "body": {"bot_id": self._bot_id, "secret": self._secret}, + "body": { + "bot_id": self._bot_id, + "secret": self._secret, + "device_id": self._device_id, + }, } ) @@ -497,7 +502,17 @@ async def _on_message(self, payload: Dict[str, Any]) -> None: logger.debug("[%s] DM sender %s blocked by policy", self.name, sender_id) return + # Cache the inbound req_id after policy checks so proactive sends to + # this chat can fall back to APP_CMD_RESPONSE (required for groups — + # WeCom AI Bots cannot initiate APP_CMD_SEND in group chats). + self._remember_chat_req_id(chat_id, self._payload_req_id(payload)) + text, reply_text = self._extract_text(body) + # Strip leading @mention in group chats so slash commands like + # "@BotName /approve" are correctly recognized as "/approve". + # Mirrors what the Telegram adapter does (re.sub @botname). + if is_group and text: + text = re.sub(r"^@\S+\s*", "", text).strip() media_urls, media_types = await self._extract_media(body) message_type = self._derive_message_type(body, text, media_types) has_reply_context = bool(reply_text and (text or media_urls)) @@ -614,13 +629,16 @@ def _extract_text(body: Dict[str, Any]) -> Tuple[str, Optional[str]]: msgtype = str(body.get("msgtype") or "").lower() if msgtype == "mixed": - mixed = body.get("mixed") if isinstance(body.get("mixed"), dict) else {} - items = mixed.get("msg_item") if isinstance(mixed.get("msg_item"), list) else [] + _raw_mixed = body.get("mixed") + mixed = _raw_mixed if isinstance(_raw_mixed, dict) else {} + _raw_items = mixed.get("msg_item") + items = _raw_items if isinstance(_raw_items, list) else [] for item in items: if not isinstance(item, dict): continue if str(item.get("msgtype") or "").lower() == "text": - text_block = item.get("text") if isinstance(item.get("text"), dict) else {} + _raw_text = item.get("text") + text_block = _raw_text if isinstance(_raw_text, dict) else {} content = str(text_block.get("content") or "").strip() if content: text_parts.append(content) @@ -662,8 +680,10 @@ async def _extract_media(self, body: Dict[str, Any]) -> Tuple[List[str], List[st msgtype = str(body.get("msgtype") or "").lower() if msgtype == "mixed": - mixed = body.get("mixed") if isinstance(body.get("mixed"), dict) else {} - items = mixed.get("msg_item") if isinstance(mixed.get("msg_item"), list) else [] + _raw_mixed = body.get("mixed") + mixed = _raw_mixed if isinstance(_raw_mixed, dict) else {} + _raw_items = mixed.get("msg_item") + items = _raw_items if isinstance(_raw_items, list) else [] for item in items: if not isinstance(item, dict): continue @@ -848,6 +868,23 @@ def _remember_reply_req_id(self, message_id: str, req_id: str) -> None: while len(self._reply_req_ids) > DEDUP_MAX_SIZE: self._reply_req_ids.pop(next(iter(self._reply_req_ids))) + def _remember_chat_req_id(self, chat_id: str, req_id: str) -> None: + """Cache the most recent inbound req_id per chat. + + Used as a fallback reply target when we need to send into a group + without an explicit ``reply_to`` — WeCom AI Bots are blocked from + APP_CMD_SEND in groups and must use APP_CMD_RESPONSE bound to some + prior req_id. Bounded like _reply_req_ids so long-running gateways + don't leak memory across many chats. + """ + normalized_chat_id = str(chat_id or "").strip() + normalized_req_id = str(req_id or "").strip() + if not normalized_chat_id or not normalized_req_id: + return + self._last_chat_req_ids[normalized_chat_id] = normalized_req_id + while len(self._last_chat_req_ids) > DEDUP_MAX_SIZE: + self._last_chat_req_ids.pop(next(iter(self._last_chat_req_ids))) + def _reply_req_id_for_message(self, reply_to: Optional[str]) -> Optional[str]: normalized = str(reply_to or "").strip() if not normalized or normalized.startswith("quote:"): @@ -1164,19 +1201,15 @@ async def _send_media_message(self, chat_id: str, media_type: str, media_id: str self._raise_for_wecom_error(response, "send media message") return response - async def _send_reply_stream(self, reply_req_id: str, content: str) -> Dict[str, Any]: + async def _send_reply_markdown(self, reply_req_id: str, content: str) -> Dict[str, Any]: response = await self._send_reply_request( reply_req_id, { - "msgtype": "stream", - "stream": { - "id": self._new_req_id("stream"), - "finish": True, - "content": content[:self.MAX_MESSAGE_LENGTH], - }, + "msgtype": "markdown", + "markdown": {"content": content[:self.MAX_MESSAGE_LENGTH]}, }, ) - self._raise_for_wecom_error(response, "send reply stream") + self._raise_for_wecom_error(response, "send reply markdown") return response async def _send_reply_media_message( @@ -1236,6 +1269,9 @@ async def _send_media_source( return SendResult(success=False, error=prepared["reject_reason"]) reply_req_id = self._reply_req_id_for_message(reply_to) + if not reply_req_id and chat_id in self._last_chat_req_ids: + reply_req_id = self._last_chat_req_ids[chat_id] + try: upload_result = await self._upload_media_bytes( prepared["data"], @@ -1303,8 +1339,12 @@ async def send( try: reply_req_id = self._reply_req_id_for_message(reply_to) + + if not reply_req_id and chat_id in self._last_chat_req_ids: + reply_req_id = self._last_chat_req_ids[chat_id] + if reply_req_id: - response = await self._send_reply_stream(reply_req_id, content) + response = await self._send_reply_markdown(reply_req_id, content) else: response = await self._send_request( APP_CMD_SEND, @@ -1429,3 +1469,134 @@ async def get_chat_info(self, chat_id: str) -> Dict[str, Any]: "name": chat_id, "type": "group" if chat_id and chat_id.lower().startswith("group") else "dm", } + + +# ------------------------------------------------------------------ +# QR code scan flow for obtaining bot credentials +# ------------------------------------------------------------------ + +_QR_GENERATE_URL = "https://work.weixin.qq.com/ai/qc/generate" +_QR_QUERY_URL = "https://work.weixin.qq.com/ai/qc/query_result" +_QR_CODE_PAGE = "https://work.weixin.qq.com/ai/qc/gen?source=hermes&scode=" +_QR_POLL_INTERVAL = 3 # seconds +_QR_POLL_TIMEOUT = 300 # 5 minutes + + +def qr_scan_for_bot_info( + *, + timeout_seconds: int = _QR_POLL_TIMEOUT, +) -> Optional[Dict[str, str]]: + """Run the WeCom QR scan flow to obtain bot_id and secret. + + Fetches a QR code from WeCom, renders it in the terminal, and polls + until the user scans it or the timeout expires. + + Returns ``{"bot_id": ..., "secret": ...}`` on success, ``None`` on + failure or timeout. + + Note: the ``work.weixin.qq.com/ai/qc/{generate,query_result}`` endpoints + used here are not part of WeCom's public developer API — they back the + admin-console web UI's bot-creation flow and may change without notice. + The same pattern is used by the feishu/dingtalk QR setup wizards. + """ + try: + import urllib.request + import urllib.parse + except ImportError: # pragma: no cover + logger.error("urllib is required for WeCom QR scan") + return None + + generate_url = f"{_QR_GENERATE_URL}?source=hermes" + + # ── Step 1: Fetch QR code ── + print(" Connecting to WeCom...", end="", flush=True) + try: + req = urllib.request.Request(generate_url, headers={"User-Agent": "HermesAgent/1.0"}) + with urllib.request.urlopen(req, timeout=15) as resp: + raw = json.loads(resp.read().decode("utf-8")) + except Exception as exc: + logger.error("WeCom QR: failed to fetch QR code: %s", exc) + print(f" failed: {exc}") + return None + + data = raw.get("data") or {} + scode = str(data.get("scode") or "").strip() + auth_url = str(data.get("auth_url") or "").strip() + + if not scode or not auth_url: + logger.error("WeCom QR: unexpected response format: %s", raw) + print(" failed: unexpected response format") + return None + + print(" done.") + + # ── Step 2: Render QR code in terminal ── + print() + qr_rendered = False + try: + import qrcode as _qrcode + qr = _qrcode.QRCode() + qr.add_data(auth_url) + qr.make(fit=True) + qr.print_ascii(invert=True) + qr_rendered = True + except ImportError: + pass + except Exception: + pass + + page_url = f"{_QR_CODE_PAGE}{urllib.parse.quote(scode)}" + if qr_rendered: + print(f"\n Scan the QR code above, or open this URL directly:\n {page_url}") + else: + print(f" Open this URL in WeCom on your phone:\n\n {page_url}\n") + print(" Tip: pip install qrcode to display a scannable QR code here next time") + print() + print(" Fetching configuration results...", end="", flush=True) + + # ── Step 3: Poll for result ── + import time + deadline = time.time() + timeout_seconds + query_url = f"{_QR_QUERY_URL}?scode={urllib.parse.quote(scode)}" + poll_count = 0 + + while time.time() < deadline: + try: + req = urllib.request.Request(query_url, headers={"User-Agent": "HermesAgent/1.0"}) + with urllib.request.urlopen(req, timeout=10) as resp: + result = json.loads(resp.read().decode("utf-8")) + except Exception as exc: + logger.debug("WeCom QR poll error: %s", exc) + time.sleep(_QR_POLL_INTERVAL) + continue + + poll_count += 1 + # Print a dot on every poll so progress is visible within 3s. + print(".", end="", flush=True) + + result_data = result.get("data") or {} + status = str(result_data.get("status") or "").lower() + + if status == "success": + print() # newline after "Fetching configuration results..." dots + bot_info = result_data.get("bot_info") or {} + bot_id = str(bot_info.get("botid") or bot_info.get("bot_id") or "").strip() + secret = str(bot_info.get("secret") or "").strip() + if bot_id and secret: + return {"bot_id": bot_id, "secret": secret} + logger.warning( + "WeCom QR: scan reported success but bot_info missing or incomplete: %s", + result_data, + ) + print( + " QR scan reported success but no bot credentials were returned.\n" + " This usually means the bot was not actually created on the WeCom side.\n" + " Falling back to manual credential entry." + ) + return None + + time.sleep(_QR_POLL_INTERVAL) + + print() # newline after dots + print(f" QR scan timed out ({timeout_seconds // 60} minutes). Please try again.") + return None diff --git a/gateway/platforms/wecom_callback.py b/gateway/platforms/wecom_callback.py index 4bb67d5cfadc..5440792dea18 100644 --- a/gateway/platforms/wecom_callback.py +++ b/gateway/platforms/wecom_callback.py @@ -258,6 +258,20 @@ async def _handle_callback(self, request: web.Request) -> web.Response: ) event = self._build_event(app, decrypted) if event is not None: + # Deduplicate: WeCom retries callbacks on timeout, + # producing duplicate inbound messages (#10305). + if event.message_id: + now = time.time() + if event.message_id in self._seen_messages: + if now - self._seen_messages[event.message_id] < MESSAGE_DEDUP_TTL_SECONDS: + logger.debug("[WecomCallback] Duplicate MsgId %s, skipping", event.message_id) + return web.Response(text="success", content_type="text/plain") + del self._seen_messages[event.message_id] + self._seen_messages[event.message_id] = now + # Prune expired entries when cache grows large + if len(self._seen_messages) > 2000: + cutoff = now - MESSAGE_DEDUP_TTL_SECONDS + self._seen_messages = {k: v for k, v in self._seen_messages.items() if v > cutoff} # Record which app this user belongs to. if event.source and event.source.user_id: map_key = self._user_app_key( diff --git a/gateway/platforms/weixin.py b/gateway/platforms/weixin.py index e5859e41a4d8..958e71da176b 100644 --- a/gateway/platforms/weixin.py +++ b/gateway/platforms/weixin.py @@ -28,7 +28,7 @@ from datetime import datetime from pathlib import Path from typing import Any, Dict, List, Optional, Tuple -from urllib.parse import quote +from urllib.parse import quote, urlparse logger = logging.getLogger(__name__) @@ -96,6 +96,28 @@ MEDIA_FILE = 3 MEDIA_VOICE = 4 +_LIVE_ADAPTERS: Dict[str, Any] = {} + + +def _make_ssl_connector() -> Optional["aiohttp.TCPConnector"]: + """Return a TCPConnector with a certifi CA bundle, or None if certifi is unavailable. + + Tencent's iLink server (``ilinkai.weixin.qq.com``) is not verifiable against + some system CA stores (notably Homebrew's OpenSSL on macOS Apple Silicon). + When ``certifi`` is installed, use its Mozilla CA bundle to guarantee + verification. Otherwise fall back to aiohttp's default (which honors + ``SSL_CERT_FILE`` env var via ``trust_env=True``). + """ + try: + import ssl + import certifi + except ImportError: + return None + if not AIOHTTP_AVAILABLE: + return None + ssl_ctx = ssl.create_default_context(cafile=certifi.where()) + return aiohttp.TCPConnector(ssl=ssl_ctx) + ITEM_TEXT = 1 ITEM_IMAGE = 2 ITEM_VOICE = 3 @@ -398,7 +420,12 @@ async def _send_message( text: str, context_token: Optional[str], client_id: str, -) -> None: +) -> Dict[str, Any]: + """Send a text message via iLink sendmessage API. + + Returns the raw API response dict (may contain error codes like + ``errcode: -14`` for session expiry that the caller can inspect). + """ if not text or not text.strip(): raise ValueError("_send_message: text must not be empty") message: Dict[str, Any] = { @@ -411,7 +438,7 @@ async def _send_message( } if context_token: message["context_token"] = context_token - await _api_post( + return await _api_post( session, base_url=base_url, endpoint=EP_SEND_MESSAGE, @@ -533,6 +560,39 @@ async def _download_bytes( return await response.read() +_WEIXIN_CDN_ALLOWLIST: frozenset[str] = frozenset( + { + "novac2c.cdn.weixin.qq.com", + "ilinkai.weixin.qq.com", + "wx.qlogo.cn", + "thirdwx.qlogo.cn", + "res.wx.qq.com", + "mmbiz.qpic.cn", + "mmbiz.qlogo.cn", + } +) + + +def _assert_weixin_cdn_url(url: str) -> None: + """Raise ValueError if *url* does not point at a known WeChat CDN host.""" + try: + parsed = urlparse(url) + scheme = parsed.scheme.lower() + host = parsed.hostname or "" + except Exception as exc: # noqa: BLE001 + raise ValueError(f"Unparseable media URL: {url!r}") from exc + + if scheme not in ("http", "https"): + raise ValueError( + f"Media URL has disallowed scheme {scheme!r}; only http/https are permitted." + ) + if host not in _WEIXIN_CDN_ALLOWLIST: + raise ValueError( + f"Media URL host {host!r} is not in the WeChat CDN allowlist. " + "Refusing to fetch to prevent SSRF." + ) + + def _media_reference(item: Dict[str, Any], key: str) -> Dict[str, Any]: return (item.get(key) or {}).get("media") or {} @@ -553,6 +613,7 @@ async def _download_and_decrypt_media( timeout_seconds=timeout_seconds, ) elif full_url: + _assert_weixin_cdn_url(full_url) raw = await _download_bytes(session, url=full_url, timeout_seconds=timeout_seconds) else: raise RuntimeError("media item had neither encrypt_query_param nor full_url") @@ -623,42 +684,31 @@ def _rewrite_table_block_for_weixin(lines: List[str]) -> str: def _normalize_markdown_blocks(content: str) -> str: lines = content.splitlines() result: List[str] = [] - i = 0 in_code_block = False + blank_run = 0 - while i < len(lines): - line = lines[i].rstrip() - fence_match = _FENCE_RE.match(line.strip()) - if fence_match: + for raw_line in lines: + line = raw_line.rstrip() + if _FENCE_RE.match(line.strip()): in_code_block = not in_code_block result.append(line) - i += 1 + blank_run = 0 continue if in_code_block: result.append(line) - i += 1 continue - if ( - i + 1 < len(lines) - and "|" in lines[i] - and _TABLE_RULE_RE.match(lines[i + 1].rstrip()) - ): - table_lines = [lines[i].rstrip(), lines[i + 1].rstrip()] - i += 2 - while i < len(lines) and "|" in lines[i]: - table_lines.append(lines[i].rstrip()) - i += 1 - result.append(_rewrite_table_block_for_weixin(table_lines)) + if not line.strip(): + blank_run += 1 + if blank_run <= 1: + result.append("") continue - result.append(_MARKDOWN_LINK_RE.sub(r"\1 (\2)", _rewrite_headers_for_weixin(line))) - i += 1 + blank_run = 0 + result.append(line) - normalized = "\n".join(item.rstrip() for item in result) - normalized = re.sub(r"\n{3,}", "\n\n", normalized) - return normalized.strip() + return "\n".join(result).strip() def _split_markdown_blocks(content: str) -> List[str]: @@ -704,8 +754,8 @@ def _split_delivery_units_for_weixin(content: str) -> List[str]: Weixin can render Markdown, but chat readability is better when top-level line breaks become separate messages. Keep fenced code blocks intact and - attach indented continuation lines to the previous top-level line so - transformed tables/lists do not get torn apart. + attach indented continuation lines to the previous top-level line so nested + list items do not get torn apart. """ units: List[str] = [] @@ -747,7 +797,9 @@ def _looks_like_chatty_line_for_weixin(line: str) -> bool: return False if line.startswith((" ", "\t")): return False - if stripped.startswith((">", "-", "*", "【")): + if stripped.startswith((">", "-", "*", "【", "#", "|")): + return False + if _TABLE_RULE_RE.match(stripped): return False if re.match(r"^\*\*[^*]+\*\*$", stripped): return False @@ -757,10 +809,12 @@ def _looks_like_chatty_line_for_weixin(line: str) -> bool: def _looks_like_heading_line_for_weixin(line: str) -> bool: - """Return True when a short line behaves like a plain-text heading.""" + """Return True when a short line behaves like a heading.""" stripped = line.strip() if not stripped: return False + if _HEADER_RE.match(stripped): + return True return len(stripped) <= 24 and stripped.endswith((":", "：")) @@ -935,7 +989,7 @@ async def qr_login( if not AIOHTTP_AVAILABLE: raise RuntimeError("aiohttp is required for Weixin QR login") - async with aiohttp.ClientSession(trust_env=True) as session: + async with aiohttp.ClientSession(trust_env=True, connector=_make_ssl_connector()) as session: try: qr_resp = await _api_get( session, @@ -953,6 +1007,10 @@ async def qr_login( logger.error("weixin: QR response missing qrcode") return None + # qrcode_url is the full scannable liteapp URL; qrcode_value is just the hex token + # WeChat needs to scan the full URL, not the raw hex string + qr_scan_data = qrcode_url if qrcode_url else qrcode_value + print("\n请使用微信扫描以下二维码：") if qrcode_url: print(qrcode_url) @@ -960,11 +1018,11 @@ async def qr_login( import qrcode qr = qrcode.QRCode() - qr.add_data(qrcode_url or qrcode_value) + qr.add_data(qr_scan_data) qr.make(fit=True) qr.print_ascii(invert=True) - except Exception: - print("（终端二维码渲染失败，请直接打开上面的二维码链接）") + except Exception as _qr_exc: + print(f"（终端二维码渲染失败: {_qr_exc}，请直接打开上面的二维码链接）") deadline = time.time() + timeout_seconds current_base_url = ILINK_BASE_URL @@ -1010,8 +1068,17 @@ async def qr_login( ) qrcode_value = str(qr_resp.get("qrcode") or "") qrcode_url = str(qr_resp.get("qrcode_img_content") or "") + qr_scan_data = qrcode_url if qrcode_url else qrcode_value if qrcode_url: print(qrcode_url) + try: + import qrcode as _qrcode + qr = _qrcode.QRCode() + qr.add_data(qr_scan_data) + qr.make(fit=True) + qr.print_ascii(invert=True) + except Exception: + pass except Exception as exc: logger.error("weixin: QR refresh failed: %s", exc) return None @@ -1059,7 +1126,8 @@ def __init__(self, config: PlatformConfig): self._hermes_home = hermes_home self._token_store = ContextTokenStore(hermes_home) self._typing_cache = TypingTicketCache() - self._session: Optional[aiohttp.ClientSession] = None + self._poll_session: Optional[aiohttp.ClientSession] = None + self._send_session: Optional[aiohttp.ClientSession] = None self._poll_task: Optional[asyncio.Task] = None self._dedup = MessageDeduplicator(ttl_seconds=MESSAGE_DEDUP_TTL_SECONDS) @@ -1134,14 +1202,17 @@ async def connect(self) -> bool: except Exception as exc: logger.debug("[%s] Token lock unavailable (non-fatal): %s", self.name, exc) - self._session = aiohttp.ClientSession(trust_env=True) + self._poll_session = aiohttp.ClientSession(trust_env=True, connector=_make_ssl_connector()) + self._send_session = aiohttp.ClientSession(trust_env=True, connector=_make_ssl_connector()) self._token_store.restore(self._account_id) self._poll_task = asyncio.create_task(self._poll_loop(), name="weixin-poll") self._mark_connected() + _LIVE_ADAPTERS[self._token] = self logger.info("[%s] Connected account=%s base=%s", self.name, _safe_id(self._account_id), self._base_url) return True async def disconnect(self) -> None: + _LIVE_ADAPTERS.pop(self._token, None) self._running = False if self._poll_task and not self._poll_task.done(): self._poll_task.cancel() @@ -1150,15 +1221,18 @@ async def disconnect(self) -> None: except asyncio.CancelledError: pass self._poll_task = None - if self._session and not self._session.closed: - await self._session.close() - self._session = None + if self._poll_session and not self._poll_session.closed: + await self._poll_session.close() + self._poll_session = None + if self._send_session and not self._send_session.closed: + await self._send_session.close() + self._send_session = None self._release_platform_lock() self._mark_disconnected() logger.info("[%s] Disconnected", self.name) async def _poll_loop(self) -> None: - assert self._session is not None + assert self._poll_session is not None sync_buf = _load_sync_buf(self._hermes_home, self._account_id) timeout_ms = LONG_POLL_TIMEOUT_MS consecutive_failures = 0 @@ -1166,7 +1240,7 @@ async def _poll_loop(self) -> None: while self._running: try: response = await _get_updates( - self._session, + self._poll_session, base_url=self._base_url, token=self._token, sync_buf=sync_buf, @@ -1223,7 +1297,7 @@ async def _process_message_safe(self, message: Dict[str, Any]) -> None: logger.error("[%s] unhandled inbound error from=%s: %s", self.name, _safe_id(message.get("from_user_id")), exc, exc_info=True) async def _process_message(self, message: Dict[str, Any]) -> None: - assert self._session is not None + assert self._poll_session is not None sender_id = str(message.get("from_user_id") or "").strip() if not sender_id: return @@ -1316,7 +1390,7 @@ async def _download_image(self, item: Dict[str, Any]) -> Optional[str]: media = _media_reference(item, "image_item") try: data = await _download_and_decrypt_media( - self._session, + self._poll_session, cdn_base_url=self._cdn_base_url, encrypted_query_param=media.get("encrypt_query_param"), aes_key_b64=(item.get("image_item") or {}).get("aeskey") @@ -1334,7 +1408,7 @@ async def _download_video(self, item: Dict[str, Any]) -> Optional[str]: media = _media_reference(item, "video_item") try: data = await _download_and_decrypt_media( - self._session, + self._poll_session, cdn_base_url=self._cdn_base_url, encrypted_query_param=media.get("encrypt_query_param"), aes_key_b64=media.get("aes_key"), @@ -1353,7 +1427,7 @@ async def _download_file(self, item: Dict[str, Any]) -> Tuple[Optional[str], str mime = _mime_from_filename(filename) try: data = await _download_and_decrypt_media( - self._session, + self._poll_session, cdn_base_url=self._cdn_base_url, encrypted_query_param=media.get("encrypt_query_param"), aes_key_b64=media.get("aes_key"), @@ -1372,7 +1446,7 @@ async def _download_voice(self, item: Dict[str, Any]) -> Optional[str]: return None try: data = await _download_and_decrypt_media( - self._session, + self._poll_session, cdn_base_url=self._cdn_base_url, encrypted_query_param=media.get("encrypt_query_param"), aes_key_b64=media.get("aes_key"), @@ -1385,13 +1459,13 @@ async def _download_voice(self, item: Dict[str, Any]) -> Optional[str]: return None async def _maybe_fetch_typing_ticket(self, user_id: str, context_token: Optional[str]) -> None: - if not self._session or not self._token: + if not self._poll_session or not self._token: return if self._typing_cache.get(user_id): return try: response = await _get_config( - self._session, + self._poll_session, base_url=self._base_url, token=self._token, user_id=user_id, @@ -1416,12 +1490,19 @@ async def _send_text_chunk( context_token: Optional[str], client_id: str, ) -> None: - """Send a single text chunk with per-chunk retry and backoff.""" + """Send a single text chunk with per-chunk retry and backoff. + + On session-expired errors (errcode -14), automatically retries + *without* ``context_token`` — iLink accepts tokenless sends as a + degraded fallback, which keeps cron-initiated push messages working + even when no user message has refreshed the session recently. + """ last_error: Optional[Exception] = None + retried_without_token = False for attempt in range(self._send_chunk_retries + 1): try: - await _send_message( - self._session, + resp = await _send_message( + self._send_session, base_url=self._base_url, token=self._token, to=chat_id, @@ -1429,6 +1510,31 @@ async def _send_text_chunk( context_token=context_token, client_id=client_id, ) + # Check iLink response for session-expired error + if resp and isinstance(resp, dict): + ret = resp.get("ret") + errcode = resp.get("errcode") + if (ret is not None and ret not in (0,)) or (errcode is not None and errcode not in (0,)): + is_session_expired = ( + ret == SESSION_EXPIRED_ERRCODE + or errcode == SESSION_EXPIRED_ERRCODE + ) + # Session expired — strip token and retry once + if is_session_expired and not retried_without_token and context_token: + retried_without_token = True + context_token = None + self._token_store._cache.pop( + self._token_store._key(self._account_id, chat_id), None + ) + logger.warning( + "[%s] session expired for %s; retrying without context_token", + self.name, _safe_id(chat_id), + ) + continue + errmsg = resp.get("errmsg") or resp.get("msg") or "unknown error" + raise RuntimeError( + f"iLink sendmessage error: ret={ret} errcode={errcode} errmsg={errmsg}" + ) return except Exception as exc: last_error = exc @@ -1456,12 +1562,48 @@ async def send( reply_to: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, ) -> SendResult: - if not self._session or not self._token: + if not self._send_session or not self._token: return SendResult(success=False, error="Not connected") context_token = self._token_store.get(self._account_id, chat_id) last_message_id: Optional[str] = None + + # Extract MEDIA: tags and bare local file paths before text delivery. + media_files, cleaned_content = self.extract_media(content) + _, image_cleaned = self.extract_images(cleaned_content) + local_files, final_content = self.extract_local_files(image_cleaned) + + _AUDIO_EXTS = {".ogg", ".opus", ".mp3", ".wav", ".m4a"} + _VIDEO_EXTS = {".mp4", ".mov", ".avi", ".mkv", ".webm", ".3gp"} + _IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".gif"} + + async def _deliver_media(path: str, is_voice: bool = False) -> None: + ext = Path(path).suffix.lower() + if is_voice or ext in _AUDIO_EXTS: + await self.send_voice(chat_id=chat_id, audio_path=path, metadata=metadata) + elif ext in _VIDEO_EXTS: + await self.send_video(chat_id=chat_id, video_path=path, metadata=metadata) + elif ext in _IMAGE_EXTS: + await self.send_image_file(chat_id=chat_id, image_path=path, metadata=metadata) + else: + await self.send_document(chat_id=chat_id, file_path=path, metadata=metadata) + try: - chunks = [c for c in self._split_text(self.format_message(content)) if c and c.strip()] + # Deliver extracted MEDIA: attachments first. + for media_path, is_voice in media_files: + try: + await _deliver_media(media_path, is_voice) + except Exception as exc: + logger.warning("[%s] media delivery failed for %s: %s", self.name, media_path, exc) + + # Deliver bare local file paths. + for file_path in local_files: + try: + await _deliver_media(file_path, is_voice=False) + except Exception as exc: + logger.warning("[%s] local file delivery failed for %s: %s", self.name, file_path, exc) + + # Deliver text content. + chunks = [c for c in self._split_text(self.format_message(final_content)) if c and c.strip()] for idx, chunk in enumerate(chunks): client_id = f"hermes-weixin-{uuid.uuid4().hex}" await self._send_text_chunk( @@ -1479,14 +1621,14 @@ async def send( return SendResult(success=False, error=str(exc)) async def send_typing(self, chat_id: str, metadata: Optional[Dict[str, Any]] = None) -> None: - if not self._session or not self._token: + if not self._send_session or not self._token: return typing_ticket = self._typing_cache.get(chat_id) if not typing_ticket: return try: await _send_typing( - self._session, + self._send_session, base_url=self._base_url, token=self._token, to_user_id=chat_id, @@ -1497,14 +1639,14 @@ async def send_typing(self, chat_id: str, metadata: Optional[Dict[str, Any]] = N logger.debug("[%s] typing start failed for %s: %s", self.name, _safe_id(chat_id), exc) async def stop_typing(self, chat_id: str) -> None: - if not self._session or not self._token: + if not self._send_session or not self._token: return typing_ticket = self._typing_cache.get(chat_id) if not typing_ticket: return try: await _send_typing( - self._session, + self._send_session, base_url=self._base_url, token=self._token, to_user_id=chat_id, @@ -1542,24 +1684,35 @@ async def send_image( async def send_image_file( self, chat_id: str, - path: str, - caption: str = "", + image_path: str, + caption: Optional[str] = None, reply_to: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, + **kwargs, ) -> SendResult: - return await self.send_document(chat_id, file_path=path, caption=caption, metadata=metadata) + del reply_to, kwargs + return await self.send_document( + chat_id=chat_id, + file_path=image_path, + caption=caption, + metadata=metadata, + ) async def send_document( self, chat_id: str, file_path: str, - caption: str = "", + caption: Optional[str] = None, + file_name: Optional[str] = None, + reply_to: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, + **kwargs, ) -> SendResult: - if not self._session or not self._token: + del file_name, reply_to, metadata, kwargs + if not self._send_session or not self._token: return SendResult(success=False, error="Not connected") try: - message_id = await self._send_file(chat_id, file_path, caption) + message_id = await self._send_file(chat_id, file_path, caption or "") return SendResult(success=True, message_id=message_id) except Exception as exc: logger.error("[%s] send_document failed to=%s: %s", self.name, _safe_id(chat_id), exc) @@ -1573,7 +1726,7 @@ async def send_video( reply_to: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, ) -> SendResult: - if not self._session or not self._token: + if not self._send_session or not self._token: return SendResult(success=False, error="Not connected") try: message_id = await self._send_file(chat_id, video_path, caption or "") @@ -1590,7 +1743,24 @@ async def send_voice( reply_to: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, ) -> SendResult: - return await self.send_document(chat_id, audio_path, caption=caption or "", metadata=metadata) + if not self._send_session or not self._token: + return SendResult(success=False, error="Not connected") + + # Native outbound Weixin voice bubbles are not proven-working in the + # upstream reference implementation. Prefer a reliable file attachment + # fallback so users at least receive playable audio, even for .silk. + fallback_caption = caption or "[voice message as attachment]" + try: + message_id = await self._send_file( + chat_id, + audio_path, + fallback_caption, + force_file_attachment=True, + ) + return SendResult(success=True, message_id=message_id) + except Exception as exc: + logger.error("[%s] send_voice failed to=%s: %s", self.name, _safe_id(chat_id), exc) + return SendResult(success=False, error=str(exc)) async def _download_remote_media(self, url: str) -> str: from tools.url_safety import is_safe_url @@ -1598,8 +1768,8 @@ async def _download_remote_media(self, url: str) -> str: if not is_safe_url(url): raise ValueError(f"Blocked unsafe URL (SSRF protection): {url}") - assert self._session is not None - async with self._session.get(url, timeout=aiohttp.ClientTimeout(total=30)) as response: + assert self._send_session is not None + async with self._send_session.get(url, timeout=aiohttp.ClientTimeout(total=30)) as response: response.raise_for_status() data = await response.read() suffix = Path(url.split("?", 1)[0]).suffix or ".bin" @@ -1607,16 +1777,22 @@ async def _download_remote_media(self, url: str) -> str: handle.write(data) return handle.name - async def _send_file(self, chat_id: str, path: str, caption: str) -> str: - assert self._session is not None and self._token is not None + async def _send_file( + self, + chat_id: str, + path: str, + caption: str, + force_file_attachment: bool = False, + ) -> str: + assert self._send_session is not None and self._token is not None plaintext = Path(path).read_bytes() - media_type, item_builder = self._outbound_media_builder(path) + media_type, item_builder = self._outbound_media_builder(path, force_file_attachment=force_file_attachment) filekey = secrets.token_hex(16) aes_key = secrets.token_bytes(16) rawsize = len(plaintext) rawfilemd5 = hashlib.md5(plaintext).hexdigest() upload_response = await _get_upload_url( - self._session, + self._send_session, base_url=self._base_url, token=self._token, to_user_id=chat_id, @@ -1642,30 +1818,34 @@ async def _send_file(self, chat_id: str, path: str, caption: str) -> str: raise RuntimeError(f"getUploadUrl returned neither upload_param nor upload_full_url: {upload_response}") encrypted_query_param = await _upload_ciphertext( - self._session, + self._send_session, ciphertext=ciphertext, upload_url=upload_url, ) - context_token = self._token_store.get(self._account_id, chat_id) # The iLink API expects aes_key as base64(hex_string), not base64(raw_bytes). # Sending base64(raw_bytes) causes images to show as grey boxes on the # receiver side because the decryption key doesn't match. aes_key_for_api = base64.b64encode(aes_key.hex().encode("ascii")).decode("ascii") - media_item = item_builder( - encrypt_query_param=encrypted_query_param, - aes_key_for_api=aes_key_for_api, - ciphertext_size=len(ciphertext), - plaintext_size=rawsize, - filename=Path(path).name, - rawfilemd5=rawfilemd5, - ) + item_kwargs = { + "encrypt_query_param": encrypted_query_param, + "aes_key_for_api": aes_key_for_api, + "ciphertext_size": len(ciphertext), + "plaintext_size": rawsize, + "filename": Path(path).name, + "rawfilemd5": rawfilemd5, + } + if media_type == MEDIA_VOICE and path.endswith(".silk"): + item_kwargs["encode_type"] = 6 + item_kwargs["sample_rate"] = 24000 + item_kwargs["bits_per_sample"] = 16 + media_item = item_builder(**item_kwargs) last_message_id = None if caption: last_message_id = f"hermes-weixin-{uuid.uuid4().hex}" await _send_message( - self._session, + self._send_session, base_url=self._base_url, token=self._token, to=chat_id, @@ -1676,7 +1856,7 @@ async def _send_file(self, chat_id: str, path: str, caption: str) -> str: last_message_id = f"hermes-weixin-{uuid.uuid4().hex}" await _api_post( - self._session, + self._send_session, base_url=self._base_url, endpoint=EP_SEND_MESSAGE, payload={ @@ -1695,7 +1875,7 @@ async def _send_file(self, chat_id: str, path: str, caption: str) -> str: ) return last_message_id - def _outbound_media_builder(self, path: str): + def _outbound_media_builder(self, path: str, force_file_attachment: bool = False): mime = mimetypes.guess_type(path)[0] or "application/octet-stream" if mime.startswith("image/"): return MEDIA_IMAGE, lambda **kw: { @@ -1723,7 +1903,7 @@ def _outbound_media_builder(self, path: str): "video_md5": kw.get("rawfilemd5", ""), }, } - if mime.startswith("audio/") or path.endswith(".silk"): + if path.endswith(".silk") and not force_file_attachment: return MEDIA_VOICE, lambda **kw: { "type": ITEM_VOICE, "voice_item": { @@ -1732,9 +1912,25 @@ def _outbound_media_builder(self, path: str): "aes_key": kw["aes_key_for_api"], "encrypt_type": 1, }, + "encode_type": kw.get("encode_type"), + "bits_per_sample": kw.get("bits_per_sample"), + "sample_rate": kw.get("sample_rate"), "playtime": kw.get("playtime", 0), }, } + if mime.startswith("audio/"): + return MEDIA_FILE, lambda **kw: { + "type": ITEM_FILE, + "file_item": { + "media": { + "encrypt_query_param": kw["encrypt_query_param"], + "aes_key": kw["aes_key_for_api"], + "encrypt_type": 1, + }, + "file_name": kw["filename"], + "len": str(kw["plaintext_size"]), + }, + } return MEDIA_FILE, lambda **kw: { "type": ITEM_FILE, "file_item": { @@ -1784,7 +1980,34 @@ async def send_weixin_direct( token_store.restore(account_id) context_token = token_store.get(account_id, chat_id) - async with aiohttp.ClientSession(trust_env=True) as session: + live_adapter = _LIVE_ADAPTERS.get(resolved_token) + send_session = getattr(live_adapter, '_send_session', None) + if live_adapter is not None and send_session is not None and not send_session.closed: + last_result: Optional[SendResult] = None + cleaned = live_adapter.format_message(message) + if cleaned: + last_result = await live_adapter.send(chat_id, cleaned) + if not last_result.success: + return {"error": f"Weixin send failed: {last_result.error}"} + + for media_path, _is_voice in media_files or []: + ext = Path(media_path).suffix.lower() + if ext in {".jpg", ".jpeg", ".png", ".gif", ".webp", ".bmp"}: + last_result = await live_adapter.send_image_file(chat_id, media_path) + else: + last_result = await live_adapter.send_document(chat_id, media_path) + if not last_result.success: + return {"error": f"Weixin media send failed: {last_result.error}"} + + return { + "success": True, + "platform": "weixin", + "chat_id": chat_id, + "message_id": last_result.message_id if last_result else None, + "context_token_used": bool(context_token), + } + + async with aiohttp.ClientSession(trust_env=True, connector=_make_ssl_connector()) as session: adapter = WeixinAdapter( PlatformConfig( enabled=True, @@ -1797,6 +2020,7 @@ async def send_weixin_direct( }, ) ) + adapter._send_session = session adapter._session = session adapter._token = resolved_token adapter._account_id = account_id diff --git a/gateway/platforms/whatsapp.py b/gateway/platforms/whatsapp.py index d1de5b856870..a82417a6015c 100644 --- a/gateway/platforms/whatsapp.py +++ b/gateway/platforms/whatsapp.py @@ -66,6 +66,37 @@ def _kill_port_process(port: int) -> None: except Exception: pass + +def _terminate_bridge_process(proc, *, force: bool = False) -> None: + """Terminate the bridge process using process-tree semantics where possible.""" + if _IS_WINDOWS: + cmd = ["taskkill", "/PID", str(proc.pid), "/T"] + if force: + cmd.append("/F") + try: + result = subprocess.run( + cmd, + capture_output=True, + text=True, + timeout=10, + ) + except FileNotFoundError: + if force: + proc.kill() + else: + proc.terminate() + return + + if result.returncode != 0: + details = (result.stderr or result.stdout or "").strip() + raise OSError(details or f"taskkill failed for PID {proc.pid}") + return + + import signal + + sig = signal.SIGTERM if not force else signal.SIGKILL + os.killpg(os.getpgid(proc.pid), sig) + import sys sys.path.insert(0, str(Path(__file__).resolve().parents[2])) @@ -118,6 +149,10 @@ class WhatsAppAdapter(BasePlatformAdapter): - bridge_script: Path to the Node.js bridge script - bridge_port: Port for HTTP communication (default: 3000) - session_path: Path to store WhatsApp session data + - dm_policy: "open" | "allowlist" | "disabled" — how DMs are handled (default: "open") + - allow_from: List of sender IDs allowed in DMs (when dm_policy="allowlist") + - group_policy: "open" | "allowlist" | "disabled" — which groups are processed (default: "open") + - group_allow_from: List of group JIDs allowed (when group_policy="allowlist") """ # WhatsApp message limits — practical UX limit, not protocol max. @@ -140,6 +175,10 @@ def __init__(self, config: PlatformConfig): get_hermes_dir("platforms/whatsapp/session", "whatsapp/session") )) self._reply_prefix: Optional[str] = config.extra.get("reply_prefix") + self._dm_policy = str(config.extra.get("dm_policy") or os.getenv("WHATSAPP_DM_POLICY", "open")).strip().lower() + self._allow_from = self._coerce_allow_list(config.extra.get("allow_from") or config.extra.get("allowFrom")) + self._group_policy = str(config.extra.get("group_policy") or os.getenv("WHATSAPP_GROUP_POLICY", "open")).strip().lower() + self._group_allow_from = self._coerce_allow_list(config.extra.get("group_allow_from") or config.extra.get("groupAllowFrom")) self._mention_patterns = self._compile_mention_patterns() self._message_queue: asyncio.Queue = asyncio.Queue() self._bridge_log_fh = None @@ -163,6 +202,33 @@ def _whatsapp_free_response_chats(self) -> set[str]: return {str(part).strip() for part in raw if str(part).strip()} return {part.strip() for part in str(raw).split(",") if part.strip()} + @staticmethod + def _coerce_allow_list(raw) -> set[str]: + """Parse allow_from / group_allow_from from config or env var.""" + if raw is None: + return set() + if isinstance(raw, list): + return {str(part).strip() for part in raw if str(part).strip()} + return {part.strip() for part in str(raw).split(",") if part.strip()} + + def _is_dm_allowed(self, sender_id: str) -> bool: + """Check whether a DM from the given sender should be processed.""" + if self._dm_policy == "disabled": + return False + if self._dm_policy == "allowlist": + return sender_id in self._allow_from + # "open" — all DMs allowed + return True + + def _is_group_allowed(self, chat_id: str) -> bool: + """Check whether a group chat should be processed.""" + if self._group_policy == "disabled": + return False + if self._group_policy == "allowlist": + return chat_id in self._group_allow_from + # "open" — all groups allowed + return True + def _compile_mention_patterns(self): patterns = self.config.extra.get("mention_patterns") if patterns is None: @@ -255,8 +321,18 @@ def _clean_bot_mention_text(self, text: str, data: Dict[str, Any]) -> str: return cleaned.strip() or text def _should_process_message(self, data: Dict[str, Any]) -> bool: - if not data.get("isGroup"): + is_group = data.get("isGroup", False) + if is_group: + chat_id = str(data.get("chatId") or "") + if not self._is_group_allowed(chat_id): + return False + else: + sender_id = str(data.get("senderId") or data.get("from") or "") + if not self._is_dm_allowed(sender_id): + return False + # DMs that pass the policy gate are always processed return True + # Group messages: check mention / free-response settings chat_id = str(data.get("chatId") or "") if chat_id in self._whatsapp_free_response_chats(): return True @@ -289,39 +365,40 @@ async def connect(self) -> bool: logger.info("[%s] Bridge found at %s", self.name, bridge_path) # Acquire scoped lock to prevent duplicate sessions + lock_acquired = False try: if not self._acquire_platform_lock('whatsapp-session', str(self._session_path), 'WhatsApp session'): return False + lock_acquired = True except Exception as e: logger.warning("[%s] Could not acquire session lock (non-fatal): %s", self.name, e) - # Auto-install npm dependencies if node_modules doesn't exist - bridge_dir = bridge_path.parent - if not (bridge_dir / "node_modules").exists(): - print(f"[{self.name}] Installing WhatsApp bridge dependencies...") - try: - install_result = subprocess.run( - ["npm", "install", "--silent"], - cwd=str(bridge_dir), - capture_output=True, - text=True, - timeout=60, - ) - if install_result.returncode != 0: - print(f"[{self.name}] npm install failed: {install_result.stderr}") - return False - print(f"[{self.name}] Dependencies installed") - except Exception as e: - print(f"[{self.name}] Failed to install dependencies: {e}") - return False - try: + # Auto-install npm dependencies if node_modules doesn't exist + bridge_dir = bridge_path.parent + if not (bridge_dir / "node_modules").exists(): + print(f"[{self.name}] Installing WhatsApp bridge dependencies...") + try: + install_result = subprocess.run( + ["npm", "install", "--silent"], + cwd=str(bridge_dir), + capture_output=True, + text=True, + timeout=60, + ) + if install_result.returncode != 0: + print(f"[{self.name}] npm install failed: {install_result.stderr}") + return False + print(f"[{self.name}] Dependencies installed") + except Exception as e: + print(f"[{self.name}] Failed to install dependencies: {e}") + return False + # Ensure session directory exists self._session_path.mkdir(parents=True, exist_ok=True) # Check if bridge is already running and connected import aiohttp - import asyncio try: async with aiohttp.ClientSession() as session: async with session.get( @@ -452,10 +529,13 @@ async def connect(self) -> bool: return True except Exception as e: - self._release_platform_lock() logger.error("[%s] Failed to start bridge: %s", self.name, e, exc_info=True) - self._close_bridge_log() return False + finally: + if not self._running: + if lock_acquired: + self._release_platform_lock() + self._close_bridge_log() def _close_bridge_log(self) -> None: """Close the bridge log file handle if open.""" @@ -487,22 +567,14 @@ async def disconnect(self) -> None: """Stop the WhatsApp bridge and clean up any orphaned processes.""" if self._bridge_process: try: - # Kill the entire process group so child node processes die too - import signal try: - if _IS_WINDOWS: - self._bridge_process.terminate() - else: - os.killpg(os.getpgid(self._bridge_process.pid), signal.SIGTERM) + _terminate_bridge_process(self._bridge_process, force=False) except (ProcessLookupError, PermissionError): self._bridge_process.terminate() await asyncio.sleep(1) if self._bridge_process.poll() is None: try: - if _IS_WINDOWS: - self._bridge_process.kill() - else: - os.killpg(os.getpgid(self._bridge_process.pid), signal.SIGKILL) + _terminate_bridge_process(self._bridge_process, force=True) except (ProcessLookupError, PermissionError): self._bridge_process.kill() except Exception as e: @@ -655,6 +727,8 @@ async def edit_message( chat_id: str, message_id: str, content: str, + *, + finalize: bool = False, ) -> SendResult: """Edit a previously sent message via the WhatsApp bridge.""" if not self._running or not self._http_session: @@ -766,6 +840,17 @@ async def send_video( """Send a video natively via bridge — plays inline in WhatsApp.""" return await self._send_media_to_bridge(chat_id, video_path, "video", caption) + async def send_voice( + self, + chat_id: str, + audio_path: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + **kwargs, + ) -> SendResult: + """Send an audio file as a WhatsApp voice message via bridge.""" + return await self._send_media_to_bridge(chat_id, audio_path, "audio", caption) + async def send_document( self, chat_id: str, diff --git a/gateway/run.py b/gateway/run.py index afc5aa035edb..db3f8b00d5ed 100644 --- a/gateway/run.py +++ b/gateway/run.py @@ -24,10 +24,22 @@ import tempfile import threading import time +from collections import OrderedDict +from contextvars import copy_context from pathlib import Path from datetime import datetime from typing import Dict, Optional, Any, List +from agent.account_usage import fetch_account_usage, render_account_usage_lines + +# --- Agent cache tuning --------------------------------------------------- +# Bounds the per-session AIAgent cache to prevent unbounded growth in +# long-lived gateways (each AIAgent holds LLM clients, tool schemas, +# memory providers, etc.). LRU order + idle TTL eviction are enforced +# from _enforce_agent_cache_cap() and _session_expiry_watcher() below. +_AGENT_CACHE_MAX_SIZE = 128 +_AGENT_CACHE_IDLE_TTL_SECS = 3600.0 # evict agents idle for >1h + # --------------------------------------------------------------------------- # SSL certificate auto-detection for NixOS and other non-standard systems. # Must run BEFORE any HTTP library (discord, aiohttp, etc.) is imported. @@ -76,7 +88,7 @@ def _ensure_ssl_certs() -> None: # Resolve Hermes home directory (respects HERMES_HOME override) from hermes_constants import get_hermes_home -from utils import atomic_yaml_write, is_truthy_value +from utils import atomic_yaml_write, base_url_host_matches, is_truthy_value _hermes_home = get_hermes_home() # Load environment variables from ~/.hermes/.env first. @@ -86,6 +98,10 @@ def _ensure_ssl_certs() -> None: _env_path = _hermes_home / '.env' load_hermes_dotenv(hermes_home=_hermes_home, project_env=Path(__file__).resolve().parents[1] / '.env') + +_DOCKER_VOLUME_SPEC_RE = re.compile(r"^(?P.+):(?P/[^:]+?)(?::(?P[^:]+))?$") +_DOCKER_MEDIA_OUTPUT_CONTAINER_PATHS = {"/output", "/outputs"} + # Bridge config.yaml values into the environment so os.getenv() picks them up. # config.yaml is authoritative for terminal settings — overrides .env. _config_path = _hermes_home / 'config.yaml' @@ -130,6 +146,12 @@ def _ensure_ssl_certs() -> None: for _cfg_key, _env_var in _terminal_env_map.items(): if _cfg_key in _terminal_cfg: _val = _terminal_cfg[_cfg_key] + # Skip cwd placeholder values (".", "auto", "cwd") — the + # gateway resolves these to Path.home() later (line ~255). + # Writing the raw placeholder here would just be noise. + # Only bridge explicit absolute paths from config.yaml. + if _cfg_key == "cwd" and str(_val) in (".", "auto", "cwd"): + continue if isinstance(_val, list): os.environ[_env_var] = json.dumps(_val) else: @@ -224,6 +246,13 @@ def _ensure_ssl_certs() -> None: except Exception: pass +# Warn if user has deprecated MESSAGING_CWD / TERMINAL_CWD in .env +try: + from hermes_cli.config import warn_deprecated_cwd_env_vars + warn_deprecated_cwd_env_vars() +except Exception: + pass + # Gateway runs in quiet mode - suppress debug output and use cwd directly (no temp dirs) os.environ["HERMES_QUIET"] = "1" @@ -231,12 +260,14 @@ def _ensure_ssl_certs() -> None: os.environ["HERMES_EXEC_ASK"] = "1" # Set terminal working directory for messaging platforms. -# If the user set an explicit path in config.yaml (not "." or "auto"), -# respect it. Otherwise use MESSAGING_CWD or default to home directory. +# config.yaml terminal.cwd is the canonical source (bridged to TERMINAL_CWD +# by the config bridge above). When it's unset or a placeholder, default +# to home directory. MESSAGING_CWD is accepted as a backward-compat +# fallback (deprecated — the warning above tells users to migrate). _configured_cwd = os.environ.get("TERMINAL_CWD", "") if not _configured_cwd or _configured_cwd in (".", "auto", "cwd"): - messaging_cwd = os.getenv("MESSAGING_CWD") or str(Path.home()) - os.environ["TERMINAL_CWD"] = messaging_cwd + _fallback = os.getenv("MESSAGING_CWD") or str(Path.home()) + os.environ["TERMINAL_CWD"] = _fallback from gateway.config import ( Platform, @@ -250,6 +281,7 @@ def _ensure_ssl_certs() -> None: build_session_context, build_session_context_prompt, build_session_key, + is_shared_multi_user_session, ) from gateway.delivery import DeliveryRouter from gateway.platforms.base import ( @@ -373,6 +405,33 @@ def _dequeue_pending_event(adapter, session_key: str) -> MessageEvent | None: return adapter.get_pending_message(session_key) +_INTERRUPT_REASON_STOP = "Stop requested" +_INTERRUPT_REASON_RESET = "Session reset requested" +_INTERRUPT_REASON_TIMEOUT = "Execution timed out (inactivity)" +_INTERRUPT_REASON_SSE_DISCONNECT = "SSE client disconnected" +_INTERRUPT_REASON_GATEWAY_SHUTDOWN = "Gateway shutting down" +_INTERRUPT_REASON_GATEWAY_RESTART = "Gateway restarting" + +_CONTROL_INTERRUPT_MESSAGES = frozenset( + { + _INTERRUPT_REASON_STOP.lower(), + _INTERRUPT_REASON_RESET.lower(), + _INTERRUPT_REASON_TIMEOUT.lower(), + _INTERRUPT_REASON_SSE_DISCONNECT.lower(), + _INTERRUPT_REASON_GATEWAY_SHUTDOWN.lower(), + _INTERRUPT_REASON_GATEWAY_RESTART.lower(), + } +) + + +def _is_control_interrupt_message(message: Optional[str]) -> bool: + """Return True when an interrupt message is internal control flow.""" + if not message: + return False + normalized = " ".join(str(message).strip().split()).lower() + return normalized in _CONTROL_INTERRUPT_MESSAGES + + def _check_unavailable_skill(command_name: str) -> str | None: """Check if a command matches a known-but-inactive skill. @@ -482,6 +541,32 @@ def _resolve_hermes_bin() -> Optional[list[str]]: return None +def _parse_session_key(session_key: str) -> "dict | None": + """Parse a session key into its component parts. + + Session keys follow the format + ``agent:main:{platform}:{chat_type}:{chat_id}[:{extra}...]``. + Returns a dict with ``platform``, ``chat_type``, ``chat_id``, and + optionally ``thread_id`` keys, or None if the key doesn't match. + + The 6th element is only returned as ``thread_id`` for chat types where + it is unambiguous (``dm`` and ``thread``). For group/channel sessions + the suffix may be a user_id (per-user isolation) rather than a + thread_id, so we leave ``thread_id`` out to avoid mis-routing. + """ + parts = session_key.split(":") + if len(parts) >= 5 and parts[0] == "agent" and parts[1] == "main": + result = { + "platform": parts[2], + "chat_type": parts[3], + "chat_id": parts[4], + } + if len(parts) > 5 and parts[3] in ("dm", "thread"): + result["thread_id"] = parts[5] + return result + return None + + def _format_gateway_process_notification(evt: dict) -> "str | None": """Format a watch pattern event from completion_queue into a [SYSTEM:] message.""" evt_type = evt.get("type", "completion") @@ -534,6 +619,7 @@ class GatewayRunner: def __init__(self, config: Optional[GatewayConfig] = None): self.config = config or load_gateway_config() self.adapters: Dict[Platform, BasePlatformAdapter] = {} + self._warn_if_docker_media_delivery_is_risky() # Load ephemeral config from config.yaml / env vars. # Both are injected at API-call time only and never persisted. @@ -546,7 +632,6 @@ def __init__(self, config: Optional[GatewayConfig] = None): self._restart_drain_timeout = self._load_restart_drain_timeout() self._provider_routing = self._load_provider_routing() self._fallback_model = self._load_fallback_model() - self._smart_model_routing = self._load_smart_model_routing() # Wire process registry into session store for reset protection from tools.process_registry import process_registry @@ -573,14 +658,21 @@ def __init__(self, config: Optional[GatewayConfig] = None): self._running_agents: Dict[str, Any] = {} self._running_agents_ts: Dict[str, float] = {} # start timestamp per session self._pending_messages: Dict[str, str] = {} # Queued messages during interrupt + self._busy_ack_ts: Dict[str, float] = {} # last busy-ack timestamp per session (debounce) + self._session_run_generation: Dict[str, int] = {} # Cache AIAgent instances per session to preserve prompt caching. # Without this, a new AIAgent is created per message, rebuilding the # system prompt (including memory) every turn — breaking prefix cache # and costing ~10x more on providers with prompt caching (Anthropic). # Key: session_key, Value: (AIAgent, config_signature_str) + # + # OrderedDict so _enforce_agent_cache_cap() can pop the least-recently- + # used entry (move_to_end() on cache hits, popitem(last=False) for + # eviction). Hard cap via _AGENT_CACHE_MAX_SIZE, idle TTL enforced + # from _session_expiry_watcher(). import threading as _threading - self._agent_cache: Dict[str, tuple] = {} + self._agent_cache: "OrderedDict[str, tuple]" = OrderedDict() self._agent_cache_lock = _threading.Lock() # Per-session model overrides from /model command. @@ -618,7 +710,26 @@ def __init__(self, config: Optional[GatewayConfig] = None): self._session_db = SessionDB() except Exception as e: logger.debug("SQLite session store not available: %s", e) - + + # Opportunistic state.db maintenance: prune ended sessions older + # than sessions.retention_days + optional VACUUM. Tracks last-run + # in state_meta so it only actually executes once per + # sessions.min_interval_hours. Gateway is long-lived so blocking + # a few seconds once per day is acceptable; failures are logged + # but never raised. + if self._session_db is not None: + try: + from hermes_cli.config import load_config as _load_full_config + _sess_cfg = (_load_full_config().get("sessions") or {}) + if _sess_cfg.get("auto_prune", False): + self._session_db.maybe_auto_prune_and_vacuum( + retention_days=int(_sess_cfg.get("retention_days", 90)), + min_interval_hours=int(_sess_cfg.get("min_interval_hours", 24)), + vacuum=bool(_sess_cfg.get("vacuum_after_prune", True)), + ) + except Exception as exc: + logger.debug("state.db auto-maintenance skipped: %s", exc) + # DM pairing store for code-based user authorization from gateway.pairing import PairingStore self.pairing_store = PairingStore() @@ -634,6 +745,53 @@ def __init__(self, config: Optional[GatewayConfig] = None): self._background_tasks: set = set() + def _warn_if_docker_media_delivery_is_risky(self) -> None: + """Warn when Docker-backed gateways lack an explicit export mount. + + MEDIA delivery happens in the gateway process, so paths emitted by the model + must be readable from the host. A plain container-local path like + `/workspace/report.txt` or `/output/report.txt` often exists only inside + Docker, so users commonly need a dedicated export mount such as + `host-dir:/output`. + """ + if os.getenv("TERMINAL_ENV", "").strip().lower() != "docker": + return + + connected = self.config.get_connected_platforms() + messaging_platforms = [p for p in connected if p not in {Platform.LOCAL, Platform.API_SERVER, Platform.WEBHOOK}] + if not messaging_platforms: + return + + raw_volumes = os.getenv("TERMINAL_DOCKER_VOLUMES", "").strip() + volumes: List[str] = [] + if raw_volumes: + try: + parsed = json.loads(raw_volumes) + if isinstance(parsed, list): + volumes = [str(v) for v in parsed if isinstance(v, str)] + except Exception: + logger.debug("Could not parse TERMINAL_DOCKER_VOLUMES for gateway media warning", exc_info=True) + + has_explicit_output_mount = False + for spec in volumes: + match = _DOCKER_VOLUME_SPEC_RE.match(spec) + if not match: + continue + container_path = match.group("container") + if container_path in _DOCKER_MEDIA_OUTPUT_CONTAINER_PATHS: + has_explicit_output_mount = True + break + + if has_explicit_output_mount: + return + + logger.warning( + "Docker backend is enabled for the messaging gateway but no explicit host-visible " + "output mount (for example '/home/user/.hermes/cache/documents:/output') is configured. " + "This is fine if the model already emits host-visible paths, but MEDIA file delivery can fail " + "for container-local paths like '/workspace/...' or '/output/...'." + ) + # -- Setup skill availability ---------------------------------------- @@ -650,6 +808,10 @@ def _has_setup_skill(self) -> bool: _VOICE_MODE_PATH = _hermes_home / "gateway_voice_mode.json" + def _voice_key(self, platform: Platform, chat_id: str) -> str: + """Return a platform-namespaced key for voice mode state.""" + return f"{platform.value}:{chat_id}" + def _load_voice_modes(self) -> Dict[str, str]: try: data = json.loads(self._VOICE_MODE_PATH.read_text()) @@ -660,11 +822,21 @@ def _load_voice_modes(self) -> Dict[str, str]: return {} valid_modes = {"off", "voice_only", "all"} - return { - str(chat_id): mode - for chat_id, mode in data.items() - if mode in valid_modes - } + result = {} + for chat_id, mode in data.items(): + if mode not in valid_modes: + continue + key = str(chat_id) + # Skip legacy unprefixed keys (warn and skip) + if ":" not in key: + logger.warning( + "Skipping legacy unprefixed voice mode key %r during migration. " + "Re-enable voice mode on that chat to rebuild the prefixed key.", + key, + ) + continue + result[key] = mode + return result def _save_voice_modes(self) -> None: try: @@ -690,11 +862,36 @@ def _sync_voice_mode_state_to_adapter(self, adapter) -> None: disabled_chats = getattr(adapter, "_auto_tts_disabled_chats", None) if not isinstance(disabled_chats, set): return + platform = getattr(adapter, "platform", None) + if not isinstance(platform, Platform): + return disabled_chats.clear() + prefix = f"{platform.value}:" disabled_chats.update( - chat_id for chat_id, mode in self._voice_mode.items() if mode == "off" + key[len(prefix):] for key, mode in self._voice_mode.items() + if mode == "off" and key.startswith(prefix) ) + async def _safe_adapter_disconnect(self, adapter, platform) -> None: + """Call adapter.disconnect() defensively, swallowing any error. + + Used when adapter.connect() failed or raised — the adapter may + have allocated partial resources (aiohttp.ClientSession, poll + tasks, child subprocesses) that would otherwise leak and surface + as "Unclosed client session" warnings at process exit. + + Must tolerate partial-init state and never raise, since callers + use it inside error-handling blocks. + """ + try: + await adapter.disconnect() + except Exception as e: + logger.debug( + "Defensive %s disconnect after failed connect raised: %s", + platform.value if platform is not None else "adapter", + e, + ) + # ----------------------------------------------------------------- def _flush_memories_for_session( @@ -734,69 +931,72 @@ def _flush_memories_for_session( enabled_toolsets=["memory", "skills"], session_id=old_session_id, ) - # Fully silence the flush agent — quiet_mode only suppresses init - # messages; tool call output still leaks to the terminal through - # _safe_print → _print_fn. Set a no-op to prevent that. - tmp_agent._print_fn = lambda *a, **kw: None + try: + # Fully silence the flush agent — quiet_mode only suppresses init + # messages; tool call output still leaks to the terminal through + # _safe_print → _print_fn. Set a no-op to prevent that. + tmp_agent._print_fn = lambda *a, **kw: None + + # Build conversation history from transcript + msgs = [ + {"role": m.get("role"), "content": m.get("content")} + for m in history + if m.get("role") in ("user", "assistant") and m.get("content") + ] - # Build conversation history from transcript - msgs = [ - {"role": m.get("role"), "content": m.get("content")} - for m in history - if m.get("role") in ("user", "assistant") and m.get("content") - ] + # Read live memory state from disk so the flush agent can see + # what's already saved and avoid overwriting newer entries. + _current_memory = "" + try: + from tools.memory_tool import get_memory_dir + _mem_dir = get_memory_dir() + for fname, label in [ + ("MEMORY.md", "MEMORY (your personal notes)"), + ("USER.md", "USER PROFILE (who the user is)"), + ]: + fpath = _mem_dir / fname + if fpath.exists(): + content = fpath.read_text(encoding="utf-8").strip() + if content: + _current_memory += f"\n\n## Current {label}:\n{content}" + except Exception: + pass # Non-fatal — flush still works, just without the guard + + # Give the agent a real turn to think about what to save + flush_prompt = ( + "[System: This session is about to be automatically reset due to " + "inactivity or a scheduled daily reset. The conversation context " + "will be cleared after this turn.\n\n" + "Review the conversation above and:\n" + "1. Save any important facts, preferences, or decisions to memory " + "(user profile or your notes) that would be useful in future sessions.\n" + "2. If you discovered a reusable workflow or solved a non-trivial " + "problem, consider saving it as a skill.\n" + "3. If nothing is worth saving, that's fine — just skip.\n\n" + ) - # Read live memory state from disk so the flush agent can see - # what's already saved and avoid overwriting newer entries. - _current_memory = "" - try: - from tools.memory_tool import get_memory_dir - _mem_dir = get_memory_dir() - for fname, label in [ - ("MEMORY.md", "MEMORY (your personal notes)"), - ("USER.md", "USER PROFILE (who the user is)"), - ]: - fpath = _mem_dir / fname - if fpath.exists(): - content = fpath.read_text(encoding="utf-8").strip() - if content: - _current_memory += f"\n\n## Current {label}:\n{content}" - except Exception: - pass # Non-fatal — flush still works, just without the guard - - # Give the agent a real turn to think about what to save - flush_prompt = ( - "[System: This session is about to be automatically reset due to " - "inactivity or a scheduled daily reset. The conversation context " - "will be cleared after this turn.\n\n" - "Review the conversation above and:\n" - "1. Save any important facts, preferences, or decisions to memory " - "(user profile or your notes) that would be useful in future sessions.\n" - "2. If you discovered a reusable workflow or solved a non-trivial " - "problem, consider saving it as a skill.\n" - "3. If nothing is worth saving, that's fine — just skip.\n\n" - ) + if _current_memory: + flush_prompt += ( + "IMPORTANT — here is the current live state of memory. Other " + "sessions, cron jobs, or the user may have updated it since this " + "conversation ended. Do NOT overwrite or remove entries unless " + "the conversation above reveals something that genuinely " + "supersedes them. Only add new information that is not already " + "captured below." + f"{_current_memory}\n\n" + ) - if _current_memory: flush_prompt += ( - "IMPORTANT — here is the current live state of memory. Other " - "sessions, cron jobs, or the user may have updated it since this " - "conversation ended. Do NOT overwrite or remove entries unless " - "the conversation above reveals something that genuinely " - "supersedes them. Only add new information that is not already " - "captured below." - f"{_current_memory}\n\n" + "Do NOT respond to the user. Just use the memory and skill_manage " + "tools if needed, then stop.]" ) - flush_prompt += ( - "Do NOT respond to the user. Just use the memory and skill_manage " - "tools if needed, then stop.]" - ) - - tmp_agent.run_conversation( - user_message=flush_prompt, - conversation_history=msgs, - ) + tmp_agent.run_conversation( + user_message=flush_prompt, + conversation_history=msgs, + ) + finally: + self._cleanup_agent_resources(tmp_agent) logger.info("Pre-reset memory flush completed for session %s", old_session_id) except Exception as e: logger.debug("Pre-reset memory flush failed for session %s: %s", old_session_id, e) @@ -807,7 +1007,7 @@ async def _async_flush_memories( session_key: Optional[str] = None, ): """Run the sync memory flush in a thread pool so it won't block the event loop.""" - loop = asyncio.get_event_loop() + loop = asyncio.get_running_loop() await loop.run_in_executor( None, self._flush_memories_for_session, @@ -922,11 +1122,16 @@ def _resolve_session_agent_runtime( return model, runtime_kwargs def _resolve_turn_agent_config(self, user_message: str, model: str, runtime_kwargs: dict) -> dict: - from agent.smart_model_routing import resolve_turn_route + """Build the effective model/runtime config for a single turn. + + Always uses the session's primary model/provider. If `/fast` is + enabled and the model supports Priority Processing / Anthropic fast + mode, attach `request_overrides` so the API call is marked + accordingly. + """ from hermes_cli.models import resolve_fast_mode_overrides - primary = { - "model": model, + runtime = { "api_key": runtime_kwargs.get("api_key"), "base_url": runtime_kwargs.get("base_url"), "provider": runtime_kwargs.get("provider"), @@ -935,7 +1140,18 @@ def _resolve_turn_agent_config(self, user_message: str, model: str, runtime_kwar "args": list(runtime_kwargs.get("args") or []), "credential_pool": runtime_kwargs.get("credential_pool"), } - route = resolve_turn_route(user_message, getattr(self, "_smart_model_routing", {}), primary) + route = { + "model": model, + "runtime": runtime, + "signature": ( + model, + runtime["provider"], + runtime["base_url"], + runtime["api_mode"], + runtime["command"], + tuple(runtime["args"]), + ), + } service_tier = getattr(self, "_service_tier", None) if not service_tier: @@ -943,7 +1159,7 @@ def _resolve_turn_agent_config(self, user_message: str, model: str, runtime_kwar return route try: - overrides = resolve_fast_mode_overrides(route.get("model")) + overrides = resolve_fast_mode_overrides(route["model"]) except Exception: overrides = None route["request_overrides"] = overrides @@ -1072,7 +1288,6 @@ def _load_prefill_messages() -> List[Dict[str, Any]]: the prefill_messages_file key in ~/.hermes/config.yaml. Relative paths are resolved from ~/.hermes/. """ - import json as _json file_path = os.getenv("HERMES_PREFILL_MESSAGES_FILE", "") if not file_path: try: @@ -1094,7 +1309,7 @@ def _load_prefill_messages() -> List[Dict[str, Any]]: return [] try: with open(path, "r", encoding="utf-8") as f: - data = _json.load(f) + data = json.load(f) if not isinstance(data, list): logger.warning("Prefill messages file must contain a JSON array: %s", path) return [] @@ -1301,20 +1516,6 @@ def _load_fallback_model() -> list | dict | None: pass return None - @staticmethod - def _load_smart_model_routing() -> dict: - """Load optional smart cheap-vs-strong model routing config.""" - try: - import yaml as _y - cfg_path = _hermes_home / "config.yaml" - if cfg_path.exists(): - with open(cfg_path, encoding="utf-8") as _f: - cfg = _y.safe_load(_f) or {} - return cfg.get("smart_model_routing", {}) or {} - except Exception: - pass - return {} - def _snapshot_running_agents(self) -> Dict[str, Any]: return { session_key: agent @@ -1329,26 +1530,102 @@ def _queue_or_replace_pending_event(self, session_key: str, event: MessageEvent) merge_pending_message_event(adapter._pending_messages, session_key, event) async def _handle_active_session_busy_message(self, event: MessageEvent, session_key: str) -> bool: - if not self._draining: - return False + # --- Draining case (gateway restarting/stopping) --- + if self._draining: + adapter = self.adapters.get(event.source.platform) + if not adapter: + return True + + thread_meta = {"thread_id": event.source.thread_id} if event.source.thread_id else None + if self._queue_during_drain_enabled(): + self._queue_or_replace_pending_event(session_key, event) + message = f"⏳ Gateway {self._status_action_gerund()} — queued for the next turn after it comes back." + else: + message = f"⏳ Gateway is {self._status_action_gerund()} and is not accepting another turn right now." + await adapter._send_with_retry( + chat_id=event.source.chat_id, + content=message, + reply_to=event.message_id, + metadata=thread_meta, + ) + return True + + # Normal busy case (agent actively running a task) adapter = self.adapters.get(event.source.platform) if not adapter: - return True + return False # let default path handle it - thread_meta = {"thread_id": event.source.thread_id} if event.source.thread_id else None - if self._queue_during_drain_enabled(): - self._queue_or_replace_pending_event(session_key, event) - message = f"⏳ Gateway {self._status_action_gerund()} — queued for the next turn after it comes back." + # Store the message so it's processed as the next turn after the + # current run finishes (or is interrupted). + from gateway.platforms.base import merge_pending_message_event + merge_pending_message_event(adapter._pending_messages, session_key, event) + + is_queue_mode = self._busy_input_mode == "queue" + + # If not in queue mode, interrupt the running agent immediately. + # This aborts in-flight tool calls and causes the agent loop to exit + # at the next check point. + running_agent = self._running_agents.get(session_key) + if not is_queue_mode and running_agent and running_agent is not _AGENT_PENDING_SENTINEL: + try: + running_agent.interrupt(event.text) + except Exception: + pass # don't let interrupt failure block the ack + + # Debounce: only send an acknowledgment once every 30 seconds per session + # to avoid spamming the user when they send multiple messages quickly + _BUSY_ACK_COOLDOWN = 30 + now = time.time() + last_ack = self._busy_ack_ts.get(session_key, 0) + if now - last_ack < _BUSY_ACK_COOLDOWN: + return True # interrupt sent (if not queue), ack already delivered recently + + self._busy_ack_ts[session_key] = now + + # Build a status-rich acknowledgment + status_parts = [] + if running_agent and running_agent is not _AGENT_PENDING_SENTINEL: + try: + summary = running_agent.get_activity_summary() + iteration = summary.get("api_call_count", 0) + max_iter = summary.get("max_iterations", 0) + current_tool = summary.get("current_tool") + start_ts = self._running_agents_ts.get(session_key, 0) + if start_ts: + elapsed_min = int((now - start_ts) / 60) + if elapsed_min > 0: + status_parts.append(f"{elapsed_min} min elapsed") + if max_iter: + status_parts.append(f"iteration {iteration}/{max_iter}") + if current_tool: + status_parts.append(f"running: {current_tool}") + except Exception: + pass + + status_detail = f" ({', '.join(status_parts)})" if status_parts else "" + if is_queue_mode: + message = ( + f"⏳ Queued for the next turn{status_detail}. " + f"I'll respond once the current task finishes." + ) else: - message = f"⏳ Gateway is {self._status_action_gerund()} and is not accepting another turn right now." + message = ( + f"⚡ Interrupting current task{status_detail}. " + f"I'll respond to your message shortly." + ) + + thread_meta = {"thread_id": event.source.thread_id} if event.source.thread_id else None + try: + await adapter._send_with_retry( + chat_id=event.source.chat_id, + content=message, + reply_to=event.message_id, + metadata=thread_meta, + ) + except Exception as e: + logger.debug("Failed to send busy-ack: %s", e) - await adapter._send_with_retry( - chat_id=event.source.chat_id, - content=message, - reply_to=event.message_id, - metadata=thread_meta, - ) return True async def _drain_active_agents(self, timeout: float) -> tuple[Dict[str, Any], bool]: @@ -1391,6 +1668,83 @@ def _interrupt_running_agents(self, reason: str) -> None: except Exception as e: logger.debug("Failed interrupting agent during shutdown: %s", e) + async def _notify_active_sessions_of_shutdown(self) -> None: + """Send a notification to every chat with an active agent. + + Called at the very start of stop() — adapters are still connected so + messages can be delivered. Best-effort: individual send failures are + logged and swallowed so they never block the shutdown sequence. + """ + active = self._snapshot_running_agents() + if not active: + return + + action = "restarting" if self._restart_requested else "shutting down" + hint = ( + "Your current task will be interrupted. " + "Send any message after restart and I'll try to resume where you left off." + if self._restart_requested + else "Your current task will be interrupted." + ) + msg = f"⚠️ Gateway {action} — {hint}" + + notified: set = set() + for session_key in active: + source = None + try: + if getattr(self, "session_store", None) is not None: + self.session_store._ensure_loaded() + entry = self.session_store._entries.get(session_key) + source = getattr(entry, "origin", None) if entry else None + except Exception as e: + logger.debug( + "Failed to load session origin for shutdown notification %s: %s", + session_key, + e, + ) + + if source is not None: + platform_str = source.platform.value + chat_id = source.chat_id + thread_id = source.thread_id + else: + # Fall back to parsing the session key when no persisted + # origin is available (legacy sessions/tests). + _parsed = _parse_session_key(session_key) + if not _parsed: + continue + platform_str = _parsed["platform"] + chat_id = _parsed["chat_id"] + thread_id = _parsed.get("thread_id") + + # Deduplicate: one notification per chat, even if multiple + # sessions (different users/threads) share the same chat. + dedup_key = (platform_str, chat_id) + if dedup_key in notified: + continue + + try: + platform = Platform(platform_str) + adapter = self.adapters.get(platform) + if not adapter: + continue + + # Include thread_id if present so the message lands in the + # correct forum topic / thread. + metadata = {"thread_id": thread_id} if thread_id else None + + await adapter.send(chat_id, msg, metadata=metadata) + notified.add(dedup_key) + logger.info( + "Sent shutdown notification to %s:%s", + platform_str, chat_id, + ) + except Exception as e: + logger.debug( + "Failed to send shutdown notification to %s:%s: %s", + platform_str, chat_id, e, + ) + def _finalize_shutdown_agents(self, active_agents: Dict[str, Any]) -> None: for agent in active_agents.values(): try: @@ -1402,20 +1756,126 @@ def _finalize_shutdown_agents(self, active_agents: Dict[str, Any]) -> None: ) except Exception: pass + self._cleanup_agent_resources(agent) + + def _cleanup_agent_resources(self, agent: Any) -> None: + """Best-effort cleanup for temporary or cached agent instances.""" + if agent is None: + return + try: + if hasattr(agent, "shutdown_memory_provider"): + agent.shutdown_memory_provider() + except Exception: + pass + # Close tool resources (terminal sandboxes, browser daemons, + # background processes, httpx clients) to prevent zombie + # process accumulation. + try: + if hasattr(agent, "close"): + agent.close() + except Exception: + pass + + _STUCK_LOOP_THRESHOLD = 3 # restarts while active before auto-suspend + _STUCK_LOOP_FILE = ".restart_failure_counts" + + def _increment_restart_failure_counts(self, active_session_keys: set) -> None: + """Increment restart-failure counters for sessions active at shutdown. + + Persists to a JSON file so counters survive across restarts. + Sessions NOT in active_session_keys are removed (they completed + successfully, so the loop is broken). + """ + import json + + path = _hermes_home / self._STUCK_LOOP_FILE + try: + counts = json.loads(path.read_text()) if path.exists() else {} + except Exception: + counts = {} + + # Increment active sessions, remove inactive ones (loop broken) + new_counts = {} + for key in active_session_keys: + new_counts[key] = counts.get(key, 0) + 1 + # Keep any entries that are still above 0 even if not active now + # (they might become active again next restart) + + try: + path.write_text(json.dumps(new_counts)) + except Exception: + pass + + def _suspend_stuck_loop_sessions(self) -> int: + """Suspend sessions that have been active across too many restarts. + + Returns the number of sessions suspended. Called on gateway startup + AFTER suspend_recently_active() to catch the stuck-loop pattern: + session loads → agent gets stuck → gateway restarts → repeat. + """ + import json + + path = _hermes_home / self._STUCK_LOOP_FILE + if not path.exists(): + return 0 + + try: + counts = json.loads(path.read_text()) + except Exception: + return 0 + + suspended = 0 + stuck_keys = [k for k, v in counts.items() if v >= self._STUCK_LOOP_THRESHOLD] + + for session_key in stuck_keys: try: - if hasattr(agent, "shutdown_memory_provider"): - agent.shutdown_memory_provider() + entry = self.session_store._entries.get(session_key) + if entry and not entry.suspended: + entry.suspended = True + suspended += 1 + logger.warning( + "Auto-suspended stuck session %s (active across %d " + "consecutive restarts — likely a stuck loop)", + session_key[:30], counts[session_key], + ) except Exception: pass - # Close tool resources (terminal sandboxes, browser daemons, - # background processes, httpx clients) to prevent zombie - # process accumulation. + + if suspended: try: - if hasattr(agent, 'close'): - agent.close() + self.session_store._save() except Exception: pass + # Clear the file — counters start fresh after suspension + try: + path.unlink(missing_ok=True) + except Exception: + pass + + return suspended + + def _clear_restart_failure_count(self, session_key: str) -> None: + """Clear the restart-failure counter for a session that completed OK. + + Called after a successful agent turn to signal the loop is broken. + """ + import json + + path = _hermes_home / self._STUCK_LOOP_FILE + if not path.exists(): + return + try: + counts = json.loads(path.read_text()) + if session_key in counts: + del counts[session_key] + if counts: + path.write_text(json.dumps(counts)) + else: + path.unlink(missing_ok=True) + except Exception: + pass + async def _launch_detached_restart_command(self) -> None: import shutil import subprocess @@ -1499,6 +1959,7 @@ async def start(self) -> bool: "WECOM_CALLBACK_ALLOWED_USERS", "WEIXIN_ALLOWED_USERS", "BLUEBUBBLES_ALLOWED_USERS", + "QQ_ALLOWED_USERS", "GATEWAY_ALLOWED_USERS") ) _allow_all = os.getenv("GATEWAY_ALLOW_ALL_USERS", "").lower() in ("true", "1", "yes") or any( @@ -1512,7 +1973,8 @@ async def start(self) -> bool: "WECOM_ALLOW_ALL_USERS", "WECOM_CALLBACK_ALLOW_ALL_USERS", "WEIXIN_ALLOW_ALL_USERS", - "BLUEBUBBLES_ALLOW_ALL_USERS") + "BLUEBUBBLES_ALLOW_ALL_USERS", + "QQ_ALLOW_ALL_USERS") ) if not _any_allowlist and not _allow_all: logger.warning( @@ -1521,6 +1983,39 @@ async def start(self) -> bool: "or configure platform allowlists (e.g., TELEGRAM_ALLOWED_USERS=your_id)." ) + # Discover Python plugins before shell hooks so plugin block + # decisions take precedence in tie cases. The CLI startup path + # does this via an explicit call in hermes_cli/main.py; the + # gateway lazily imports run_agent inside per-request handlers, + # so the discover_plugins() side-effect in model_tools.py is NOT + # guaranteed to have run by the time we reach this point. + try: + from hermes_cli.plugins import discover_plugins + discover_plugins() + except Exception: + logger.debug( + "plugin discovery failed at gateway startup", exc_info=True, + ) + + # Register declarative shell hooks from cli-config.yaml. Gateway + # has no TTY, so consent has to come from one of the three opt-in + # channels (--accept-hooks on launch, HERMES_ACCEPT_HOOKS env var, + # or hooks_auto_accept: true in config.yaml). We pass + # accept_hooks=False here and let register_from_config resolve + # the effective value from env + config itself — the CLI-side + # registration already honored --accept-hooks, and re-reading + # hooks_auto_accept here would just duplicate that lookup. + # Failures are logged but must never block gateway startup. + try: + from hermes_cli.config import load_config + from agent.shell_hooks import register_from_config + register_from_config(load_config(), accept_hooks=False) + except Exception: + logger.debug( + "shell-hook registration failed at gateway startup", + exc_info=True, + ) + # Discover and load event hooks self.hooks.discover_and_load() @@ -1557,6 +2052,17 @@ async def start(self) -> bool: except Exception as e: logger.warning("Session suspension on startup failed: %s", e) + # Stuck-loop detection (#7536): if a session has been active across + # 3+ consecutive restarts, it's probably stuck in a loop (the same + # history keeps causing the agent to hang). Auto-suspend it so the + # user gets a clean slate on the next message. + try: + stuck = self._suspend_stuck_loop_sessions() + if stuck: + logger.warning("Auto-suspended %d stuck-loop session(s)", stuck) + except Exception as e: + logger.debug("Stuck-loop detection failed: %s", e) + connected_count = 0 enabled_platform_count = 0 startup_nonretryable_errors: list[str] = [] @@ -1602,6 +2108,15 @@ async def start(self) -> bool: logger.info("✓ %s connected", platform.value) else: logger.warning("✗ %s failed to connect", platform.value) + # Defensive cleanup: a failed connect() may have + # allocated resources (aiohttp.ClientSession, poll + # tasks, bridge subprocesses) before giving up. + # Without this call, those resources are orphaned + # and Python logs "Unclosed client session" at + # process exit. Adapter disconnect() implementations + # are expected to be idempotent and tolerate + # partial-init state. + await self._safe_adapter_disconnect(adapter, platform) if adapter.has_fatal_error: self._update_platform_runtime_status( platform.value, @@ -1642,6 +2157,10 @@ async def start(self) -> bool: } except Exception as e: logger.error("✗ %s error: %s", platform.value, e) + # Same defensive cleanup path for exceptions — an adapter + # that raised mid-connect may still have a live + # aiohttp.ClientSession or child subprocess. + await self._safe_adapter_disconnect(adapter, platform) self._update_platform_runtime_status( platform.value, platform_state="retrying", @@ -1804,16 +2323,12 @@ async def _session_expiry_watcher(self, interval: int = 300): if _cached_agent is None: _cached_agent = self._running_agents.get(key) if _cached_agent and _cached_agent is not _AGENT_PENDING_SENTINEL: - try: - if hasattr(_cached_agent, 'shutdown_memory_provider'): - _cached_agent.shutdown_memory_provider() - except Exception: - pass - try: - if hasattr(_cached_agent, 'close'): - _cached_agent.close() - except Exception: - pass + self._cleanup_agent_resources(_cached_agent) + # Drop the cache entry so the AIAgent (and its LLM + # clients, tool schemas, memory provider refs) can + # be garbage-collected. Otherwise the cache grows + # unbounded across the gateway's lifetime. + self._evict_cached_agent(key) # Mark as flushed and persist to disk so the flag # survives gateway restarts. with self.session_store._lock: @@ -1857,6 +2372,44 @@ async def _session_expiry_watcher(self, interval: int = 300): logger.info( "Session expiry done: %d flushed", _flushed, ) + + # Sweep agents that have been idle beyond the TTL regardless + # of session reset policy. This catches sessions with very + # long / "never" reset windows, whose cached AIAgents would + # otherwise pin memory for the gateway's entire lifetime. + try: + _idle_evicted = self._sweep_idle_cached_agents() + if _idle_evicted: + logger.info( + "Agent cache idle sweep: evicted %d agent(s)", + _idle_evicted, + ) + except Exception as _e: + logger.debug("Idle agent sweep failed: %s", _e) + + # Periodically prune stale SessionStore entries. The + # in-memory dict (and sessions.json) would otherwise grow + # unbounded in gateways serving many rotating chats / + # threads / users over long time windows. Pruning is + # invisible to users — a resumed session just gets a + # fresh session_id, exactly as if the reset policy fired. + _last_prune_ts = getattr(self, "_last_session_store_prune_ts", 0.0) + _prune_interval = 3600.0 # once per hour + if time.time() - _last_prune_ts > _prune_interval: + try: + _max_age = int( + getattr(self.config, "session_store_max_age_days", 0) or 0 + ) + if _max_age > 0: + _pruned = self.session_store.prune_old_entries(_max_age) + if _pruned: + logger.info( + "SessionStore prune: dropped %d stale entries", + _pruned, + ) + except Exception as _e: + logger.debug("SessionStore prune failed: %s", _e) + self._last_session_store_prune_ts = time.time() except Exception as e: logger.debug("Session expiry watcher error: %s", e) # Sleep in small increments so we can stop quickly @@ -2009,6 +2562,40 @@ async def stop( return async def _stop_impl() -> None: + def _kill_tool_subprocesses(phase: str) -> None: + """Kill tool subprocesses + tear down terminal envs + browsers. + + Called twice in the shutdown path: once eagerly after a + drain timeout forces agent interrupt (so we reclaim bash/ + sleep children before systemd TimeoutStopSec escalates to + SIGKILL on the cgroup — #8202), and once as a final + catch-all at the end of _stop_impl() for the graceful + path or anything respawned mid-teardown. + + All steps are best-effort; exceptions are swallowed so + one subsystem's failure doesn't block the rest. + """ + try: + from tools.process_registry import process_registry + _killed = process_registry.kill_all() + if _killed: + logger.info( + "Shutdown (%s): killed %d tool subprocess(es)", + phase, _killed, + ) + except Exception as _e: + logger.debug("process_registry.kill_all (%s) error: %s", phase, _e) + try: + from tools.terminal_tool import cleanup_all_environments + cleanup_all_environments() + except Exception as _e: + logger.debug("cleanup_all_environments (%s) error: %s", phase, _e) + try: + from tools.browser_tool import cleanup_all_browsers + cleanup_all_browsers() + except Exception as _e: + logger.debug("cleanup_all_browsers (%s) error: %s", phase, _e) + logger.info( "Stopping gateway%s...", " for restart" if self._restart_requested else "", @@ -2016,6 +2603,10 @@ async def _stop_impl() -> None: self._running = False self._draining = True + # Notify all chats with active agents BEFORE draining. + # Adapters are still connected here, so messages can be sent. + await self._notify_active_sessions_of_shutdown() + timeout = self._restart_drain_timeout active_agents, timed_out = await self._drain_active_agents(timeout) if timed_out: @@ -2024,13 +2615,57 @@ async def _stop_impl() -> None: timeout, self._running_agent_count(), ) - self._interrupt_running_agents( - "Gateway restarting" if self._restart_requested else "Gateway shutting down" + # Mark forcibly-interrupted sessions as resume_pending BEFORE + # interrupting the agents. This preserves each session's + # session_id + transcript so the next message on the same + # session_key auto-resumes from the existing conversation + # instead of getting routed through suspend_recently_active() + # and converted into a fresh session. Terminal escalation + # for genuinely stuck sessions still flows through the + # existing ``.restart_failure_counts`` stuck-loop counter + # (incremented below, threshold 3), which sets + # ``suspended=True`` and overrides resume_pending. + # + # Iterate self._running_agents (current) rather than the + # drain-start ``active_agents`` snapshot — the snapshot + # may include sessions that finished gracefully during + # the drain window, and marking those falsely would give + # them a stray restart-interruption system note on their + # next turn even though their previous turn completed + # cleanly. Skip pending sentinels for the same reason + # _interrupt_running_agents() does: their agent hasn't + # started yet, there's nothing to interrupt, and the + # session shouldn't carry a misleading resume flag. + _resume_reason = ( + "restart_timeout" if self._restart_requested else "shutdown_timeout" ) - interrupt_deadline = asyncio.get_running_loop().time() + 5.0 - while self._running_agents and asyncio.get_running_loop().time() < interrupt_deadline: - self._update_runtime_status("draining") - await asyncio.sleep(0.1) + for _sk, _agent in list(self._running_agents.items()): + if _agent is _AGENT_PENDING_SENTINEL: + continue + try: + self.session_store.mark_resume_pending(_sk, _resume_reason) + except Exception as _e: + logger.debug( + "mark_resume_pending failed for %s: %s", + _sk[:20], _e, + ) + self._interrupt_running_agents( + _INTERRUPT_REASON_GATEWAY_RESTART if self._restart_requested else _INTERRUPT_REASON_GATEWAY_SHUTDOWN + ) + interrupt_deadline = asyncio.get_running_loop().time() + 5.0 + while self._running_agents and asyncio.get_running_loop().time() < interrupt_deadline: + self._update_runtime_status("draining") + await asyncio.sleep(0.1) + + # Kill lingering tool subprocesses NOW, before we spend more + # budget on adapter disconnect / session DB close. Under + # systemd (TimeoutStopSec bounded by drain_timeout+headroom), + # deferring this to the end of stop() risks systemd escalating + # to SIGKILL on the cgroup first — at which point bash/sleep + # children left behind by an interrupted terminal tool get + # killed by systemd instead of us (issue #8202). The final + # catch-all cleanup below still runs for the graceful path. + _kill_tool_subprocesses("post-interrupt") if self._restart_requested and self._restart_detached: try: @@ -2059,39 +2694,67 @@ async def _stop_impl() -> None: self.adapters.clear() self._running_agents.clear() + self._running_agents_ts.clear() self._pending_messages.clear() self._pending_approvals.clear() + if hasattr(self, '_busy_ack_ts'): + self._busy_ack_ts.clear() self._shutdown_event.set() # Global cleanup: kill any remaining tool subprocesses not tied - # to a specific agent (catch-all for zombie prevention). - try: - from tools.process_registry import process_registry - process_registry.kill_all() - except Exception: - pass - try: - from tools.terminal_tool import cleanup_all_environments - cleanup_all_environments() - except Exception: - pass - try: - from tools.browser_tool import cleanup_all_browsers - cleanup_all_browsers() - except Exception: - pass + # to a specific agent (catch-all for zombie prevention). On the + # drain-timeout path we already did this earlier after agent + # interrupt — this second call catches (a) the graceful path + # where drain succeeded without interrupt, and (b) anything + # that got respawned between the earlier call and adapter + # disconnect (defense in depth; safe to call repeatedly). + _kill_tool_subprocesses("final-cleanup") + + # Close SQLite session DBs so the WAL write lock is released. + # Without this, --replace and similar restart flows leave the + # old gateway's connection holding the WAL lock until Python + # actually exits — causing 'database is locked' errors when + # the new gateway tries to open the same file. + for _db_holder in (self, getattr(self, "session_store", None)): + _db = getattr(_db_holder, "_db", None) if _db_holder else None + if _db is None or not hasattr(_db, "close"): + continue + try: + _db.close() + except Exception as _e: + logger.debug("SessionDB close error: %s", _e) - from gateway.status import remove_pid_file + from gateway.status import remove_pid_file, release_gateway_runtime_lock remove_pid_file() + release_gateway_runtime_lock() # Write a clean-shutdown marker so the next startup knows this # wasn't a crash. suspend_recently_active() only needs to run - # after unexpected exits — graceful shutdowns already drain - # active agents, so there's no stuck-session risk. - try: - (_hermes_home / ".clean_shutdown").touch() - except Exception: - pass + # after unexpected exits. However, if the drain timed out and + # agents were force-interrupted, their sessions may be in an + # incomplete state (trailing tool response, no final assistant + # message). Skip the marker in that case so the next startup + # suspends those sessions — giving users a clean slate instead + # of resuming a half-finished tool loop. + if not timed_out: + try: + (_hermes_home / ".clean_shutdown").touch() + except Exception: + pass + else: + logger.info( + "Skipping .clean_shutdown marker — drain timed out with " + "interrupted agents; next startup will suspend recently " + "active sessions." + ) + + # Track sessions that were active at shutdown for stuck-loop + # detection (#7536). On each restart, the counter increments + # for sessions that were running. If a session hits the + # threshold (3 consecutive restarts while active), the next + # startup auto-suspends it — breaking the loop. + if active_agents: + self._increment_restart_failure_counts(set(active_agents.keys())) if self._restart_requested and self._restart_via_service: self._exit_code = GATEWAY_SERVICE_RESTART_EXIT_CODE @@ -2255,8 +2918,15 @@ def _create_adapter( return None return BlueBubblesAdapter(config) + elif platform == Platform.QQBOT: + from gateway.platforms.qqbot import QQAdapter, check_qq_requirements + if not check_qq_requirements(): + logger.warning("QQBot: aiohttp/httpx missing or QQ_APP_ID/QQ_CLIENT_SECRET not configured") + return None + return QQAdapter(config) + return None - + def _is_user_authorized(self, source: SessionSource) -> bool: """ Check if a user is authorized to use the bot. @@ -2296,6 +2966,10 @@ def _is_user_authorized(self, source: SessionSource) -> bool: Platform.WECOM_CALLBACK: "WECOM_CALLBACK_ALLOWED_USERS", Platform.WEIXIN: "WEIXIN_ALLOWED_USERS", Platform.BLUEBUBBLES: "BLUEBUBBLES_ALLOWED_USERS", + Platform.QQBOT: "QQ_ALLOWED_USERS", + } + platform_group_env_map = { + Platform.QQBOT: "QQ_GROUP_ALLOWED_USERS", } platform_allow_all_map = { Platform.TELEGRAM: "TELEGRAM_ALLOW_ALL_USERS", @@ -2313,6 +2987,7 @@ def _is_user_authorized(self, source: SessionSource) -> bool: Platform.WECOM_CALLBACK: "WECOM_CALLBACK_ALLOW_ALL_USERS", Platform.WEIXIN: "WEIXIN_ALLOW_ALL_USERS", Platform.BLUEBUBBLES: "BLUEBUBBLES_ALLOW_ALL_USERS", + Platform.QQBOT: "QQ_ALLOW_ALL_USERS", } # Per-platform allow-all flag (e.g., DISCORD_ALLOW_ALL_USERS=true) @@ -2320,6 +2995,28 @@ def _is_user_authorized(self, source: SessionSource) -> bool: if platform_allow_all_var and os.getenv(platform_allow_all_var, "").lower() in ("true", "1", "yes"): return True + # Discord bot senders that passed the DISCORD_ALLOW_BOTS platform + # filter are already authorized at the platform level — skip the + # user allowlist. Without this, bot messages allowed by + # DISCORD_ALLOW_BOTS=mentions/all would be rejected here with + # "Unauthorized user" (fixes #4466). + if source.platform == Platform.DISCORD and getattr(source, "is_bot", False): + allow_bots = os.getenv("DISCORD_ALLOW_BOTS", "none").lower().strip() + if allow_bots in ("mentions", "all"): + return True + + # Discord role-based access (DISCORD_ALLOWED_ROLES): the adapter's + # on_message pre-filter already verified role membership — if the + # message reached here, the user passed that check. Authorize + # directly to avoid the "no allowlists configured" branch below + # rejecting role-only setups where DISCORD_ALLOWED_USERS is empty + # (issue #7871). + if ( + source.platform == Platform.DISCORD + and os.getenv("DISCORD_ALLOWED_ROLES", "").strip() + ): + return True + # Check pairing store (always checked, regardless of allowlists) platform_name = source.platform.value if source.platform else "" if self.pairing_store.is_approved(platform_name, user_id): @@ -2327,12 +3024,23 @@ def _is_user_authorized(self, source: SessionSource) -> bool: # Check platform-specific and global allowlists platform_allowlist = os.getenv(platform_env_map.get(source.platform, ""), "").strip() + group_allowlist = "" + if source.chat_type == "group": + group_allowlist = os.getenv(platform_group_env_map.get(source.platform, ""), "").strip() global_allowlist = os.getenv("GATEWAY_ALLOWED_USERS", "").strip() - if not platform_allowlist and not global_allowlist: + if not platform_allowlist and not group_allowlist and not global_allowlist: # No allowlists configured -- check global allow-all flag return os.getenv("GATEWAY_ALLOW_ALL_USERS", "").lower() in ("true", "1", "yes") + # Some platforms authorize group traffic by chat ID rather than sender ID. + if group_allowlist and source.chat_type == "group" and source.chat_id: + allowed_group_ids = { + chat_id.strip() for chat_id in group_allowlist.split(",") if chat_id.strip() + } + if "*" in allowed_group_ids or source.chat_id in allowed_group_ids: + return True + # Check if user is in any allowlist allowed_ids = set() if platform_allowlist: @@ -2365,10 +3073,59 @@ def _is_user_authorized(self, source: SessionSource) -> bool: return bool(check_ids & allowed_ids) def _get_unauthorized_dm_behavior(self, platform: Optional[Platform]) -> str: - """Return how unauthorized DMs should be handled for a platform.""" + """Return how unauthorized DMs should be handled for a platform. + + Resolution order: + 1. Explicit per-platform ``unauthorized_dm_behavior`` in config — always wins. + 2. Explicit global ``unauthorized_dm_behavior`` in config — wins when no per-platform. + 3. When an allowlist (``PLATFORM_ALLOWED_USERS`` or ``GATEWAY_ALLOWED_USERS``) is + configured, default to ``"ignore"`` — the allowlist signals that the owner has + deliberately restricted access; spamming unknown contacts with pairing codes + is both noisy and a potential info-leak. (#9337) + 4. No allowlist and no explicit config → ``"pair"`` (open-gateway default). + """ config = getattr(self, "config", None) - if config and hasattr(config, "get_unauthorized_dm_behavior"): - return config.get_unauthorized_dm_behavior(platform) + + # Check for an explicit per-platform override first. + if config and hasattr(config, "get_unauthorized_dm_behavior") and platform: + platform_cfg = config.platforms.get(platform) if hasattr(config, "platforms") else None + if platform_cfg and "unauthorized_dm_behavior" in getattr(platform_cfg, "extra", {}): + # Operator explicitly configured behavior for this platform — respect it. + return config.get_unauthorized_dm_behavior(platform) + + # Check for an explicit global config override. + if config and hasattr(config, "unauthorized_dm_behavior"): + if config.unauthorized_dm_behavior != "pair": # non-default → explicit override + return config.unauthorized_dm_behavior + + # No explicit override. Fall back to allowlist-aware default: + # if any allowlist is configured for this platform, silently drop + # unauthorized messages instead of sending pairing codes. + if platform: + platform_env_map = { + Platform.TELEGRAM: "TELEGRAM_ALLOWED_USERS", + Platform.DISCORD: "DISCORD_ALLOWED_USERS", + Platform.WHATSAPP: "WHATSAPP_ALLOWED_USERS", + Platform.SLACK: "SLACK_ALLOWED_USERS", + Platform.SIGNAL: "SIGNAL_ALLOWED_USERS", + Platform.EMAIL: "EMAIL_ALLOWED_USERS", + Platform.SMS: "SMS_ALLOWED_USERS", + Platform.MATTERMOST: "MATTERMOST_ALLOWED_USERS", + Platform.MATRIX: "MATRIX_ALLOWED_USERS", + Platform.DINGTALK: "DINGTALK_ALLOWED_USERS", + Platform.FEISHU: "FEISHU_ALLOWED_USERS", + Platform.WECOM: "WECOM_ALLOWED_USERS", + Platform.WECOM_CALLBACK: "WECOM_CALLBACK_ALLOWED_USERS", + Platform.WEIXIN: "WEIXIN_ALLOWED_USERS", + Platform.BLUEBUBBLES: "BLUEBUBBLES_ALLOWED_USERS", + Platform.QQBOT: "QQ_ALLOWED_USERS", + } + if os.getenv(platform_env_map.get(platform, ""), "").strip(): + return "ignore" + + if os.getenv("GATEWAY_ALLOWED_USERS", "").strip(): + return "ignore" + return "pair" async def _handle_message(self, event: MessageEvent) -> Optional[str]: @@ -2515,15 +3272,21 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: _quick_key[:30], _stale_age, _stale_idle, _raw_stale_timeout, _stale_detail, ) - del self._running_agents[_quick_key] - self._running_agents_ts.pop(_quick_key, None) + self._invalidate_session_run_generation( + _quick_key, + reason="stale_running_agent_eviction", + ) + self._release_running_agent_state(_quick_key) if _quick_key in self._running_agents: if event.get_command() == "status": return await self._handle_status_command(event) # Resolve the command once for all early-intercept checks below. - from hermes_cli.commands import resolve_command as _resolve_cmd_inner + from hermes_cli.commands import ( + ACTIVE_SESSION_BYPASS_COMMANDS as _DEDICATED_HANDLERS, + resolve_command as _resolve_cmd_inner, + ) _evt_cmd = event.get_command() _cmd_def_inner = _resolve_cmd_inner(_evt_cmd) if _evt_cmd else None @@ -2536,21 +3299,14 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: # _interrupt_requested. Force-clean _running_agents so the session # is unlocked and subsequent messages are processed normally. if _cmd_def_inner and _cmd_def_inner.name == "stop": - running_agent = self._running_agents.get(_quick_key) - if running_agent and running_agent is not _AGENT_PENDING_SENTINEL: - running_agent.interrupt("Stop requested") - # Force-clean: remove the session lock regardless of agent state - adapter = self.adapters.get(source.platform) - if adapter and hasattr(adapter, 'get_pending_message'): - adapter.get_pending_message(_quick_key) # consume and discard - self._pending_messages.pop(_quick_key, None) - if _quick_key in self._running_agents: - del self._running_agents[_quick_key] - # Mark session suspended so the next message starts fresh - # instead of resuming the stuck context (#7536). - self.session_store.suspend_session(_quick_key) - logger.info("HARD STOP for session %s — suspended, session lock released", _quick_key[:20]) - return "⚡ Force-stopped. The session is suspended — your next message will start fresh." + await self._interrupt_and_clear_session( + _quick_key, + source, + interrupt_reason=_INTERRUPT_REASON_STOP, + invalidation_reason="stop_command", + ) + logger.info("STOP for session %s — agent interrupted, session lock released", _quick_key[:20]) + return "⚡ Stopped. You can continue this session." # /reset and /new must bypass the running-agent guard so they # actually dispatch as commands instead of being queued as user @@ -2560,18 +3316,15 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: # doesn't get re-processed as a user message after the # interrupt completes. if _cmd_def_inner and _cmd_def_inner.name == "new": - running_agent = self._running_agents.get(_quick_key) - if running_agent and running_agent is not _AGENT_PENDING_SENTINEL: - running_agent.interrupt("Session reset requested") # Clear any pending messages so the old text doesn't replay - adapter = self.adapters.get(source.platform) - if adapter and hasattr(adapter, 'get_pending_message'): - adapter.get_pending_message(_quick_key) # consume and discard - self._pending_messages.pop(_quick_key, None) + await self._interrupt_and_clear_session( + _quick_key, + source, + interrupt_reason=_INTERRUPT_REASON_RESET, + invalidation_reason="new_command", + ) # Clean up the running agent entry so the reset handler # doesn't think an agent is still active. - if _quick_key in self._running_agents: - del self._running_agents[_quick_key] return await self._handle_reset_command(event) # /queue — queue without interrupting @@ -2581,16 +3334,62 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: return "Usage: /queue " adapter = self.adapters.get(source.platform) if adapter: - from gateway.platforms.base import MessageEvent as _ME, MessageType as _MT - queued_event = _ME( + queued_event = MessageEvent( text=queued_text, - message_type=_MT.TEXT, + message_type=MessageType.TEXT, source=event.source, message_id=event.message_id, + channel_prompt=event.channel_prompt, ) adapter._pending_messages[_quick_key] = queued_event return "Queued for the next turn." + # /steer — inject mid-run after the next tool call. + # Unlike /queue (turn boundary), /steer lands BETWEEN tool-call + # iterations inside the same agent run, by appending to the + # last tool result's content. No interrupt, no new user turn, + # no role-alternation violation. + if _cmd_def_inner and _cmd_def_inner.name == "steer": + steer_text = event.get_command_args().strip() + if not steer_text: + return "Usage: /steer " + running_agent = self._running_agents.get(_quick_key) + if running_agent is _AGENT_PENDING_SENTINEL: + # Agent hasn't started yet — queue as turn-boundary fallback. + adapter = self.adapters.get(source.platform) + if adapter: + queued_event = MessageEvent( + text=steer_text, + message_type=MessageType.TEXT, + source=event.source, + message_id=event.message_id, + channel_prompt=event.channel_prompt, + ) + adapter._pending_messages[_quick_key] = queued_event + return "Agent still starting — /steer queued for the next turn." + if running_agent and hasattr(running_agent, "steer"): + try: + accepted = running_agent.steer(steer_text) + except Exception as exc: + logger.warning("Steer failed for session %s: %s", _quick_key[:20], exc) + return f"⚠️ Steer failed: {exc}" + if accepted: + preview = steer_text[:60] + ("..." if len(steer_text) > 60 else "") + return f"⏩ Steer queued — arrives after the next tool call: '{preview}'" + return "Steer rejected (empty payload)." + # Running agent is missing or lacks steer() — fall back to queue. + adapter = self.adapters.get(source.platform) + if adapter: + queued_event = MessageEvent( + text=steer_text, + message_type=MessageType.TEXT, + source=event.source, + message_id=event.message_id, + channel_prompt=event.channel_prompt, + ) + adapter._pending_messages[_quick_key] = queued_event + return "No active agent — /steer queued for the next turn." + # /model must not be used while the agent is running. if _cmd_def_inner and _cmd_def_inner.name == "model": return "Agent is running — wait or /stop first, then switch models." @@ -2604,11 +3403,56 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: return await self._handle_approve_command(event) return await self._handle_deny_command(event) + # /agents (/tasks alias) should be query-only and never interrupt. + if _cmd_def_inner and _cmd_def_inner.name == "agents": + return await self._handle_agents_command(event) + # /background must bypass the running-agent guard — it starts a # parallel task and must never interrupt the active conversation. if _cmd_def_inner and _cmd_def_inner.name == "background": return await self._handle_background_command(event) + # Session-level toggles that are safe to run mid-agent — + # /yolo can unblock a pending approval prompt, /verbose cycles + # the tool-progress display mode for the ongoing stream. + # Both modify session state without needing agent interaction + # and must not be queued (the safety net would discard them). + # /fast and /reasoning are config-only and take effect next + # message, so they fall through to the catch-all busy response + # below — users should wait and set them between turns. + if _cmd_def_inner and _cmd_def_inner.name in ("yolo", "verbose"): + if _cmd_def_inner.name == "yolo": + return await self._handle_yolo_command(event) + if _cmd_def_inner.name == "verbose": + return await self._handle_verbose_command(event) + + # Gateway-handled info/control commands with dedicated + # running-agent handlers. + if _cmd_def_inner and _cmd_def_inner.name in _DEDICATED_HANDLERS: + if _cmd_def_inner.name == "help": + return await self._handle_help_command(event) + if _cmd_def_inner.name == "commands": + return await self._handle_commands_command(event) + if _cmd_def_inner.name == "profile": + return await self._handle_profile_command(event) + if _cmd_def_inner.name == "update": + return await self._handle_update_command(event) + + # Catch-all: any other recognized slash command reached the + # running-agent guard. Reject gracefully rather than falling + # through to interrupt + discard. Without this, commands + # like /model, /reasoning, /voice, /insights, /title, + # /resume, /retry, /undo, /compress, /usage, /provider, + # /reload-mcp, /sethome, /reset (all registered as Discord + # slash commands) would interrupt the agent AND get + # silently discarded by the slash-command safety net, + # producing a zero-char response. See #5057, #6252, #10370. + if _cmd_def_inner: + return ( + f"⏳ Agent is running — `/{_cmd_def_inner.name}` can't run " + f"mid-turn. Wait for the current response or `/stop` first." + ) + if event.message_type == MessageType.PHOTO: logger.debug("PRIORITY photo follow-up for session %s — queueing without interrupt", _quick_key[:20]) adapter = self.adapters.get(source.platform) @@ -2616,20 +3460,50 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: merge_pending_message_event(adapter._pending_messages, _quick_key, event) return None + _telegram_followup_grace = float( + os.getenv("HERMES_TELEGRAM_FOLLOWUP_GRACE_SECONDS", "3.0") + ) + _started_at = self._running_agents_ts.get(_quick_key, 0) + if ( + source.platform == Platform.TELEGRAM + and event.message_type == MessageType.TEXT + and _telegram_followup_grace > 0 + and _started_at + and (time.time() - _started_at) <= _telegram_followup_grace + ): + logger.debug( + "Telegram follow-up arrived %.2fs after run start for %s — queueing without interrupt", + time.time() - _started_at, + _quick_key[:20], + ) + adapter = self.adapters.get(source.platform) + if adapter: + merge_pending_message_event( + adapter._pending_messages, + _quick_key, + event, + merge_text=True, + ) + return None + running_agent = self._running_agents.get(_quick_key) if running_agent is _AGENT_PENDING_SENTINEL: # Agent is being set up but not ready yet. if event.get_command() == "stop": # Force-clean the sentinel so the session is unlocked. - if _quick_key in self._running_agents: - del self._running_agents[_quick_key] + self._release_running_agent_state(_quick_key) logger.info("HARD STOP (pending) for session %s — sentinel cleared", _quick_key[:20]) return "⚡ Force-stopped. The agent was still starting — session unlocked." # Queue the message so it will be picked up after the # agent starts. adapter = self.adapters.get(source.platform) if adapter: - adapter._pending_messages[_quick_key] = event + merge_pending_message_event( + adapter._pending_messages, + _quick_key, + event, + merge_text=True, + ) return None if self._draining: if self._queue_during_drain_enabled(): @@ -2649,23 +3523,73 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: # Check for commands command = event.get_command() - - # Emit command:* hook for any recognized slash command. - # GATEWAY_KNOWN_COMMANDS is derived from the central COMMAND_REGISTRY - # in hermes_cli/commands.py — no hardcoded set to maintain here. - from hermes_cli.commands import GATEWAY_KNOWN_COMMANDS, resolve_command as _resolve_cmd - if command and command in GATEWAY_KNOWN_COMMANDS: - await self.hooks.emit(f"command:{command}", { - "platform": source.platform.value if source.platform else "", - "user_id": source.user_id, - "command": command, - "args": event.get_command_args().strip(), - }) - # Resolve aliases to canonical name so dispatch only checks canonicals. + from hermes_cli.commands import ( + GATEWAY_KNOWN_COMMANDS, + is_gateway_known_command, + resolve_command as _resolve_cmd, + ) + + # Resolve aliases to canonical name so dispatch and hook names + # don't depend on the exact alias the user typed. _cmd_def = _resolve_cmd(command) if command else None canonical = _cmd_def.name if _cmd_def else command + # Fire the ``command:`` hook for any recognized slash + # command — built-in OR plugin-registered. Handlers can return a + # dict with ``{"decision": "deny" | "handled" | "rewrite", ...}`` + # to intercept dispatch before core handling runs. This replaces + # the previous fire-and-forget emit(): return values are now + # honored, but handlers that return nothing behave exactly as + # before (telemetry-style hooks keep working). + if command and is_gateway_known_command(canonical): + raw_args = event.get_command_args().strip() + hook_ctx = { + "platform": source.platform.value if source.platform else "", + "user_id": source.user_id, + "command": canonical, + "raw_command": command, + "args": raw_args, + "raw_args": raw_args, + } + try: + hook_results = await self.hooks.emit_collect( + f"command:{canonical}", hook_ctx + ) + except Exception as _hook_err: + logger.debug( + "command:%s hook dispatch failed (non-fatal): %s", + canonical, _hook_err, + ) + hook_results = [] + + for hook_result in hook_results: + if not isinstance(hook_result, dict): + continue + decision = str(hook_result.get("decision", "")).strip().lower() + if not decision or decision == "allow": + continue + if decision == "deny": + message = hook_result.get("message") + if isinstance(message, str) and message: + return message + return f"Command `/{command}` was blocked by a hook." + if decision == "handled": + message = hook_result.get("message") + return message if isinstance(message, str) and message else None + if decision == "rewrite": + new_command = str( + hook_result.get("command_name", "") + ).strip().lstrip("/") + if not new_command: + continue + new_args = str(hook_result.get("raw_args", "")).strip() + event.text = f"/{new_command} {new_args}".strip() + command = event.get_command() + _cmd_def = _resolve_cmd(command) if command else None + canonical = _cmd_def.name if _cmd_def else command + break + if canonical == "new": return await self._handle_reset_command(event) @@ -2681,6 +3605,9 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: if canonical == "status": return await self._handle_status_command(event) + if canonical == "agents": + return await self._handle_agents_command(event) + if canonical == "restart": return await self._handle_restart_command(event) @@ -2781,6 +3708,21 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: if canonical == "btw": return await self._handle_btw_command(event) + if canonical == "steer": + # No active agent — /steer has no tool call to inject into. + # Strip the prefix so downstream treats it as a normal user + # message. If the payload is empty, surface the usage hint. + steer_payload = event.get_command_args().strip() + if not steer_payload: + return "Usage: /steer (no agent is running; sending as a normal message)" + try: + event.text = steer_payload + except Exception: + pass + # Do NOT return — fall through to _handle_message_with_agent + # at the end of this function so the rewritten text is sent + # to the agent as a regular user turn. + if canonical == "voice": return await self._handle_voice_command(event) @@ -2839,9 +3781,8 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: plugin_handler = get_plugin_command_handler(command.replace("_", "-")) if plugin_handler: user_args = event.get_command_args().strip() - import asyncio as _aio result = plugin_handler(user_args) - if _aio.iscoroutine(result): + if asyncio.iscoroutine(result): result = await result return str(result) if result else None except Exception as e: @@ -2924,17 +3865,23 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: # same session — corrupting the transcript. self._running_agents[_quick_key] = _AGENT_PENDING_SENTINEL self._running_agents_ts[_quick_key] = time.time() + _run_generation = self._begin_session_run_generation(_quick_key) try: - return await self._handle_message_with_agent(event, source, _quick_key) + return await self._handle_message_with_agent(event, source, _quick_key, _run_generation) finally: # If _run_agent replaced the sentinel with a real agent and # then cleaned it up, this is a no-op. If we exited early # (exception, command fallthrough, etc.) the sentinel must # not linger or the session would be permanently locked out. if self._running_agents.get(_quick_key) is _AGENT_PENDING_SENTINEL: - del self._running_agents[_quick_key] - self._running_agents_ts.pop(_quick_key, None) + self._release_running_agent_state(_quick_key) + else: + # Agent path already cleaned _running_agents; make sure + # the paired metadata dicts are gone too. + self._running_agents_ts.pop(_quick_key, None) + if hasattr(self, "_busy_ack_ts"): + self._busy_ack_ts.pop(_quick_key, None) async def _prepare_inbound_message_text( self, @@ -2952,12 +3899,12 @@ async def _prepare_inbound_message_text( history = history or [] message_text = event.text or "" - _is_shared_thread = ( - source.chat_type != "dm" - and source.thread_id - and not getattr(self.config, "thread_sessions_per_user", False) + _is_shared_multi_user = is_shared_multi_user_session( + source, + group_sessions_per_user=getattr(self.config, "group_sessions_per_user", True), + thread_sessions_per_user=getattr(self.config, "thread_sessions_per_user", False), ) - if _is_shared_thread and source.user_name: + if _is_shared_multi_user and source.user_name: message_text = f"[{source.user_name}] {message_text}" if event.media_urls: @@ -3017,9 +3964,7 @@ async def _prepare_inbound_message_text( for i, path in enumerate(event.media_urls): mtype = event.media_types[i] if i < len(event.media_types) else "" if mtype in ("", "application/octet-stream"): - import os as _os2 - - _ext = _os2.path.splitext(path)[1].lower() + _ext = os.path.splitext(path)[1].lower() if _ext in _TEXT_EXTENSIONS: mtype = "text/plain" else: @@ -3029,13 +3974,10 @@ async def _prepare_inbound_message_text( if not mtype.startswith(("application/", "text/")): continue - import os as _os - import re as _re - - basename = _os.path.basename(path) + basename = os.path.basename(path) parts = basename.split("_", 2) display_name = parts[2] if len(parts) >= 3 else basename - display_name = _re.sub(r'[^\w.\- ]', '_', display_name) + display_name = re.sub(r'[^\w.\- ]', '_', display_name) if mtype.startswith("text/"): context_note = ( @@ -3052,24 +3994,26 @@ async def _prepare_inbound_message_text( message_text = f"{context_note}\n\n{message_text}" if getattr(event, "reply_to_text", None) and event.reply_to_message_id: + # Always inject the reply-to pointer — even when the quoted text + # already appears in history. The prefix isn't deduplication, it's + # disambiguation: it tells the agent *which* prior message the user + # is referencing. History can contain the same or similar text + # multiple times, and without an explicit pointer the agent has to + # guess (or answer for both subjects). Token overhead is minimal. reply_snippet = event.reply_to_text[:500] - found_in_history = any( - reply_snippet[:200] in (msg.get("content") or "") - for msg in history - if msg.get("role") in ("assistant", "user", "tool") - ) - if not found_in_history: - message_text = f'[Replying to: "{reply_snippet}"]\n\n{message_text}' + message_text = f'[Replying to: "{reply_snippet}"]\n\n{message_text}' if "@" in message_text: try: from agent.context_references import preprocess_context_references_async from agent.model_metadata import get_model_context_length - _msg_cwd = os.environ.get("MESSAGING_CWD", os.path.expanduser("~")) + _msg_cwd = os.environ.get("TERMINAL_CWD", os.path.expanduser("~")) + _msg_runtime = _resolve_runtime_agent_kwargs() _msg_ctx_len = get_model_context_length( self._model, - base_url=self._base_url or "", + base_url=self._base_url or _msg_runtime.get("base_url") or "", + api_key=_msg_runtime.get("api_key") or "", ) _ctx_result = await preprocess_context_references_async( message_text, @@ -3092,7 +4036,7 @@ async def _prepare_inbound_message_text( return message_text - async def _handle_message_with_agent(self, event, source, _quick_key: str): + async def _handle_message_with_agent(self, event, source, _quick_key: str, run_generation: int): """Inner handler that runs under the _running_agents sentinel guard.""" _msg_start_time = time.time() _platform_name = source.platform.value if hasattr(source.platform, "value") else str(source.platform) @@ -3434,54 +4378,58 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): model=_hyg_model, max_iterations=4, quiet_mode=True, + skip_memory=True, enabled_toolsets=["memory"], session_id=session_entry.session_id, ) - _hyg_agent._print_fn = lambda *a, **kw: None - - loop = asyncio.get_event_loop() - _compressed, _ = await loop.run_in_executor( - None, - lambda: _hyg_agent._compress_context( - _hyg_msgs, "", - approx_tokens=_approx_tokens, - ), - ) - - # _compress_context ends the old session and creates - # a new session_id. Write compressed messages into - # the NEW session so the old transcript stays intact - # and searchable via session_search. - _hyg_new_sid = _hyg_agent.session_id - if _hyg_new_sid != session_entry.session_id: - session_entry.session_id = _hyg_new_sid - self.session_store._save() - - self.session_store.rewrite_transcript( - session_entry.session_id, _compressed - ) - # Reset stored token count — transcript was rewritten - session_entry.last_prompt_tokens = 0 - history = _compressed - _new_count = len(_compressed) - _new_tokens = estimate_messages_tokens_rough( - _compressed - ) + try: + _hyg_agent._print_fn = lambda *a, **kw: None + + loop = asyncio.get_running_loop() + _compressed, _ = await loop.run_in_executor( + None, + lambda: _hyg_agent._compress_context( + _hyg_msgs, "", + approx_tokens=_approx_tokens, + ), + ) - logger.info( - "Session hygiene: compressed %s → %s msgs, " - "~%s → ~%s tokens", - _msg_count, _new_count, - f"{_approx_tokens:,}", f"{_new_tokens:,}", - ) + # _compress_context ends the old session and creates + # a new session_id. Write compressed messages into + # the NEW session so the old transcript stays intact + # and searchable via session_search. + _hyg_new_sid = _hyg_agent.session_id + if _hyg_new_sid != session_entry.session_id: + session_entry.session_id = _hyg_new_sid + self.session_store._save() + + self.session_store.rewrite_transcript( + session_entry.session_id, _compressed + ) + # Reset stored token count — transcript was rewritten + session_entry.last_prompt_tokens = 0 + history = _compressed + _new_count = len(_compressed) + _new_tokens = estimate_messages_tokens_rough( + _compressed + ) - if _new_tokens >= _warn_token_threshold: - logger.warning( - "Session hygiene: still ~%s tokens after " - "compression", - f"{_new_tokens:,}", + logger.info( + "Session hygiene: compressed %s → %s msgs, " + "~%s → ~%s tokens", + _msg_count, _new_count, + f"{_approx_tokens:,}", f"{_new_tokens:,}", ) + if _new_tokens >= _warn_token_threshold: + logger.warning( + "Session hygiene: still ~%s tokens after " + "compression", + f"{_new_tokens:,}", + ) + finally: + self._cleanup_agent_resources(_hyg_agent) + except Exception as e: logger.warning( "Session hygiene auto-compress failed: %s", e @@ -3545,6 +4493,15 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): if message_text is None: return + # Bind this gateway run generation to the adapter's active-session + # event so deferred post-delivery callbacks can be released by the + # same run that registered them. + self._bind_adapter_run_generation( + self.adapters.get(source.platform), + session_key, + run_generation, + ) + try: # Emit agent:start hook hook_ctx = { @@ -3563,7 +4520,9 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): source=source, session_id=session_entry.session_id, session_key=session_key, + run_generation=run_generation, event_message_id=event.message_id, + channel_prompt=event.channel_prompt, ) # Stop persistent typing indicator now that the agent is done @@ -3574,7 +4533,35 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): except Exception: pass + if not self._is_session_run_current(_quick_key, run_generation): + logger.info( + "Discarding stale agent result for %s — generation %d is no longer current", + _quick_key[:20] if _quick_key else "?", + run_generation, + ) + _stale_adapter = self.adapters.get(source.platform) + if getattr(type(_stale_adapter), "pop_post_delivery_callback", None) is not None: + _stale_adapter.pop_post_delivery_callback( + _quick_key, + generation=run_generation, + ) + elif _stale_adapter and hasattr(_stale_adapter, "_post_delivery_callbacks"): + _stale_adapter._post_delivery_callbacks.pop(_quick_key, None) + return None + response = agent_result.get("final_response") or "" + + # Convert the agent's internal "(empty)" sentinel into a + # user-friendly message. "(empty)" means the model failed to + # produce visible content after exhausting all retries (nudge, + # prefill, empty-retry, fallback). Sending the raw sentinel + # looks like a bug; a short explanation is more helpful. + if response == "(empty)": + response = ( + "⚠️ The model returned no response after processing tool " + "results. This can happen with some models — try again or " + "rephrase your question." + ) agent_messages = agent_result.get("messages", []) _response_time = time.time() - _msg_start_time _api_calls = agent_result.get("api_calls", 0) @@ -3585,6 +4572,24 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): _response_time, _api_calls, _resp_len, ) + # Successful turn — clear any stuck-loop counter for this session. + # This ensures the counter only accumulates across CONSECUTIVE + # restarts where the session was active (never completed). + # + # Also clear the resume_pending flag (set by drain-timeout + # shutdown) — the turn ran to completion, so recovery + # succeeded and subsequent messages should no longer receive + # the restart-interruption system note. + if session_key: + self._clear_restart_failure_count(session_key) + try: + self.session_store.clear_resume_pending(session_key) + except Exception as _e: + logger.debug( + "clear_resume_pending failed for %s: %s", + session_key[:20], _e, + ) + # Surface error details when the agent failed silently (final_response=None) if not response and agent_result.get("failed"): error_detail = agent_result.get("error", "unknown error") @@ -3673,7 +4678,7 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): synth_text = _format_gateway_process_notification(evt) if synth_text: try: - await self._inject_watch_notification(synth_text, event) + await self._inject_watch_notification(synth_text, evt) except Exception as e2: logger.error("Watch notification injection error: %s", e2) except Exception as e: @@ -3691,14 +4696,11 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): # intermediate reasoning) so sessions can be resumed with full context # and transcripts are useful for debugging and training data. # - # IMPORTANT: When the agent failed before producing any response - # (e.g. context-overflow 400), do NOT persist the user's message. + # IMPORTANT: When the agent failed (e.g. context-overflow 400, + # compression exhausted), do NOT persist the user's message. # Persisting it would make the session even larger, causing the - # same failure on the next attempt — an infinite loop. (#1630) - agent_failed_early = ( - agent_result.get("failed") - and not agent_result.get("final_response") - ) + # same failure on the next attempt — an infinite loop. (#1630, #9893) + agent_failed_early = bool(agent_result.get("failed")) if agent_failed_early: logger.info( "Skipping transcript persistence for failed request in " @@ -3706,6 +4708,24 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): session_entry.session_id, ) + # When compression is exhausted, the session is permanently too + # large to process. Auto-reset it so the next message starts + # fresh instead of replaying the same oversized context in an + # infinite fail loop. (#9893) + if agent_result.get("compression_exhausted") and session_entry and session_key: + logger.info( + "Auto-resetting session %s after compression exhaustion.", + session_entry.session_id, + ) + self.session_store.reset_session(session_key) + self._evict_cached_agent(session_key) + self._session_model_overrides.pop(session_key, None) + response = (response or "") + ( + "\n\n🔄 Session auto-reset — the conversation exceeded the " + "maximum context size and could not be compressed further. " + "Your next message will start a fresh session." + ) + ts = datetime.now().isoformat() # If this is a fresh session (no history), write the full tool @@ -3813,6 +4833,8 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str): _hist_len = len(history) if 'history' in locals() else 0 if status_code == 401: status_hint = " Check your API key or run `claude /login` to refresh OAuth credentials." + elif status_code == 402: + status_hint = " Your API balance or quota is exhausted. Check your provider dashboard." elif status_code == 429: # Check if this is a plan usage limit (resets on a schedule) vs a transient rate limit _err_body = getattr(e, "response", None) @@ -3941,6 +4963,7 @@ async def _handle_reset_command(self, event: MessageEvent) -> str: # Get existing session key session_key = self._session_key_for_source(source) + self._invalidate_session_run_generation(session_key, reason="session_reset") # Flush memories in the background (fire-and-forget) so the user # gets the "Session reset!" response immediately. @@ -3963,11 +4986,7 @@ async def _handle_reset_command(self, event: MessageEvent) -> str: _cached = self._agent_cache.get(session_key) _old_agent = _cached[0] if isinstance(_cached, tuple) else _cached if _cached else None if _old_agent is not None: - try: - if hasattr(_old_agent, "close"): - _old_agent.close() - except Exception: - pass + self._cleanup_agent_resources(_old_agent) self._evict_cached_agent(session_key) try: @@ -3989,6 +5008,11 @@ async def _handle_reset_command(self, event: MessageEvent) -> str: # the configured default instead of the previously switched model. self._session_model_overrides.pop(session_key, None) + # Clear session-scoped dangerous-command approvals and /yolo state. + # /new is a conversation-boundary operation — approval state from the + # previous conversation must not survive the reset. + self._clear_session_boundary_security_state(session_key) + # Fire plugin on_session_finalize hook (session boundary) try: from hermes_cli.plugins import invoke_hook as _invoke_hook @@ -4047,31 +5071,16 @@ async def _handle_reset_command(self, event: MessageEvent) -> str: async def _handle_profile_command(self, event: MessageEvent) -> str: """Handle /profile — show active profile name and home directory.""" - from hermes_constants import get_hermes_home, display_hermes_home - from pathlib import Path + from hermes_constants import display_hermes_home + from hermes_cli.profiles import get_active_profile_name - home = get_hermes_home() display = display_hermes_home() + profile_name = get_active_profile_name() - # Detect profile name from HERMES_HOME path - # Profile paths look like: ~/.hermes/profiles/ - profiles_parent = Path.home() / ".hermes" / "profiles" - try: - rel = home.relative_to(profiles_parent) - profile_name = str(rel).split("/")[0] - except ValueError: - profile_name = None - - if profile_name: - lines = [ - f"👤 **Profile:** `{profile_name}`", - f"📂 **Home:** `{display}`", - ] - else: - lines = [ - "👤 **Profile:** default", - f"📂 **Home:** `{display}`", - ] + lines = [ + f"👤 **Profile:** `{profile_name}`", + f"📂 **Home:** `{display}`", + ] return "\n".join(lines) @@ -4110,6 +5119,96 @@ async def _handle_status_command(self, event: MessageEvent) -> str: ]) return "\n".join(lines) + + async def _handle_agents_command(self, event: MessageEvent) -> str: + """Handle /agents command - list active agents and running tasks.""" + from tools.process_registry import format_uptime_short, process_registry + + now = time.time() + current_session_key = self._session_key_for_source(event.source) + + running_agents: dict = getattr(self, "_running_agents", {}) or {} + running_started: dict = getattr(self, "_running_agents_ts", {}) or {} + + agent_rows: list[dict] = [] + for session_key, agent in running_agents.items(): + started = float(running_started.get(session_key, now)) + elapsed = max(0, int(now - started)) + is_pending = agent is _AGENT_PENDING_SENTINEL + agent_rows.append( + { + "session_key": session_key, + "elapsed": elapsed, + "state": "starting" if is_pending else "running", + "session_id": "" if is_pending else str(getattr(agent, "session_id", "") or ""), + "model": "" if is_pending else str(getattr(agent, "model", "") or ""), + } + ) + + agent_rows.sort(key=lambda row: row["elapsed"], reverse=True) + + running_processes: list[dict] = [] + try: + running_processes = [ + p for p in process_registry.list_sessions() + if p.get("status") == "running" + ] + except Exception: + running_processes = [] + + background_tasks = [ + t for t in (getattr(self, "_background_tasks", set()) or set()) + if hasattr(t, "done") and not t.done() + ] + + lines = [ + "🤖 **Active Agents & Tasks**", + "", + f"**Active agents:** {len(agent_rows)}", + ] + + if agent_rows: + for idx, row in enumerate(agent_rows[:12], 1): + current = " · this chat" if row["session_key"] == current_session_key else "" + sid = f" · `{row['session_id']}`" if row["session_id"] else "" + model = f" · `{row['model']}`" if row["model"] else "" + lines.append( + f"{idx}. `{row['session_key']}` · {row['state']} · " + f"{format_uptime_short(row['elapsed'])}{sid}{model}{current}" + ) + if len(agent_rows) > 12: + lines.append(f"... and {len(agent_rows) - 12} more") + + lines.extend( + [ + "", + f"**Running background processes:** {len(running_processes)}", + ] + ) + if running_processes: + for proc in running_processes[:12]: + cmd = " ".join(str(proc.get("command", "")).split()) + if len(cmd) > 90: + cmd = cmd[:87] + "..." + lines.append( + f"- `{proc.get('session_id', '?')}` · " + f"{format_uptime_short(int(proc.get('uptime_seconds', 0)))} · `{cmd}`" + ) + if len(running_processes) > 12: + lines.append(f"... and {len(running_processes) - 12} more") + + lines.extend( + [ + "", + f"**Gateway async jobs:** {len(background_tasks)}", + ] + ) + + if not agent_rows and not running_processes and not background_tasks: + lines.append("") + lines.append("No active agents or running tasks.") + + return "\n".join(lines) async def _handle_stop_command(self, event: MessageEvent) -> str: """Handle /stop command - interrupt a running agent. @@ -4120,9 +5219,7 @@ async def _handle_stop_command(self, event: MessageEvent) -> str: only through normal command dispatch (no running agent) or as a fallback. Force-clean the session lock in all cases for safety. - When there IS a running/pending agent, the session is also marked - as *suspended* so the next message starts a fresh session instead - of resuming the stuck context (#7536). + The session is preserved so the user can continue the conversation. """ source = event.source session_entry = self.session_store.get_or_create_session(source) @@ -4131,24 +5228,49 @@ async def _handle_stop_command(self, event: MessageEvent) -> str: agent = self._running_agents.get(session_key) if agent is _AGENT_PENDING_SENTINEL: # Force-clean the sentinel so the session is unlocked. - if session_key in self._running_agents: - del self._running_agents[session_key] - self.session_store.suspend_session(session_key) - logger.info("HARD STOP (pending) for session %s — suspended, sentinel cleared", session_key[:20]) - return "⚡ Force-stopped. The agent was still starting — your next message will start fresh." + await self._interrupt_and_clear_session( + session_key, + source, + interrupt_reason=_INTERRUPT_REASON_STOP, + invalidation_reason="stop_command_pending", + ) + logger.info("STOP (pending) for session %s — sentinel cleared", session_key[:20]) + return "⚡ Stopped. The agent hadn't started yet — you can continue this session." if agent: - agent.interrupt("Stop requested") # Force-clean the session lock so a truly hung agent doesn't # keep it locked forever. - if session_key in self._running_agents: - del self._running_agents[session_key] - self.session_store.suspend_session(session_key) - return "⚡ Force-stopped. Your next message will start a fresh session." + await self._interrupt_and_clear_session( + session_key, + source, + interrupt_reason=_INTERRUPT_REASON_STOP, + invalidation_reason="stop_command_handler", + ) + return "⚡ Stopped. You can continue this session." else: return "No active task to stop." async def _handle_restart_command(self, event: MessageEvent) -> str: """Handle /restart command - drain active work, then restart the gateway.""" + # Defensive idempotency check: if the previous gateway process + # recorded this same /restart (same platform + update_id) and the new + # process is seeing it *again*, this is a re-delivery caused by PTB's + # graceful-shutdown `get_updates` ACK failing on the way out ("Error + # while calling `get_updates` one more time to mark all fetched + # updates. Suppressing error to ensure graceful shutdown. When + # polling for updates is restarted, updates may be received twice." + # in gateway.log). Ignoring the stale redelivery prevents a + # self-perpetuating restart loop where every fresh gateway + # re-processes the same /restart command and immediately restarts + # again. + if self._is_stale_restart_redelivery(event): + logger.info( + "Ignoring redelivered /restart (platform=%s, update_id=%s) — " + "already processed by a previous gateway instance.", + event.source.platform.value if event.source and event.source.platform else "?", + event.platform_update_id, + ) + return "" + if self._restart_requested or self._draining: count = self._running_agent_count() if count: @@ -4158,7 +5280,6 @@ async def _handle_restart_command(self, event: MessageEvent) -> str: # Save the requester's routing info so the new gateway process can # notify them once it comes back online. try: - import json as _json notify_data = { "platform": event.source.platform.value if event.source.platform else None, "chat_id": event.source.chat_id, @@ -4166,11 +5287,29 @@ async def _handle_restart_command(self, event: MessageEvent) -> str: if event.source.thread_id: notify_data["thread_id"] = event.source.thread_id (_hermes_home / ".restart_notify.json").write_text( - _json.dumps(notify_data) + json.dumps(notify_data) ) except Exception as e: logger.debug("Failed to write restart notify file: %s", e) + # Record the triggering platform + update_id in a dedicated dedup + # marker. Unlike .restart_notify.json (which gets unlinked once the + # new gateway sends the "gateway restarted" notification), this + # marker persists so the new gateway can still detect a delayed + # /restart redelivery from Telegram. Overwritten on every /restart. + try: + dedup_data = { + "platform": event.source.platform.value if event.source.platform else None, + "requested_at": time.time(), + } + if event.platform_update_id is not None: + dedup_data["update_id"] = event.platform_update_id + (_hermes_home / ".restart_last_processed.json").write_text( + json.dumps(dedup_data) + ) + except Exception as e: + logger.debug("Failed to write restart dedup marker: %s", e) + active_agents = self._running_agent_count() # When running under a service manager (systemd/launchd), use the # service restart path: exit with code 75 so the service manager @@ -4186,6 +5325,56 @@ async def _handle_restart_command(self, event: MessageEvent) -> str: return f"⏳ Draining {active_agents} active agent(s) before restart..." return "♻ Restarting gateway. If you aren't notified within 60 seconds, restart from the console with `hermes gateway restart`." + def _is_stale_restart_redelivery(self, event: MessageEvent) -> bool: + """Return True if this /restart is a Telegram re-delivery we already handled. + + The previous gateway wrote ``.restart_last_processed.json`` with the + triggering platform + update_id when it processed the /restart. If + we now see a /restart on the same platform with an update_id <= that + recorded value AND the marker is recent (< 5 minutes), it's a + redelivery and should be ignored. + + Only applies to Telegram today (the only platform that exposes a + numeric cross-session update ordering); other platforms return False. + """ + if event is None or event.source is None: + return False + if event.platform_update_id is None: + return False + if event.source.platform is None: + return False + # Only Telegram populates platform_update_id currently; be explicit + # so future platforms aren't accidentally gated by this check. + try: + platform_value = event.source.platform.value + except Exception: + return False + if platform_value != "telegram": + return False + + try: + marker_path = _hermes_home / ".restart_last_processed.json" + if not marker_path.exists(): + return False + data = json.loads(marker_path.read_text()) + except Exception: + return False + + if data.get("platform") != platform_value: + return False + recorded_uid = data.get("update_id") + if not isinstance(recorded_uid, int): + return False + # Staleness guard: ignore markers older than 5 minutes. A legitimately + # old marker (e.g. crash recovery where notify never fired) should not + # swallow a fresh /restart from the user. + requested_at = data.get("requested_at") + if isinstance(requested_at, (int, float)): + if time.time() - requested_at > 300: + return False + return event.platform_update_id <= recorded_uid + + async def _handle_help_command(self, event: MessageEvent) -> str: """Handle /help command - list available commands.""" from hermes_cli.commands import gateway_help_lines @@ -4332,6 +5521,7 @@ async def _handle_model_command(self, event: MessageEvent) -> Optional[str]: try: providers = list_authenticated_providers( current_provider=current_provider, + current_base_url=current_base_url, user_providers=user_provs, custom_providers=custom_provs, max_models=50, @@ -4443,6 +5633,7 @@ async def _on_model_selected( try: providers = list_authenticated_providers( current_provider=current_provider, + current_base_url=current_base_url, user_providers=user_provs, custom_providers=custom_provs, max_models=5, @@ -4572,7 +5763,7 @@ async def _on_model_selected( # Cache notice cache_enabled = ( - ("openrouter" in (result.base_url or "").lower() and "claude" in result.new_model.lower()) + (base_url_host_matches(result.base_url or "", "openrouter.ai") and "claude" in result.new_model.lower()) or result.api_mode == "anthropic_messages" ) if cache_enabled: @@ -4648,6 +5839,7 @@ async def _handle_provider_command(self, event: MessageEvent) -> str: async def _handle_personality_command(self, event: MessageEvent) -> str: """Handle /personality command - list or set a personality.""" import yaml + from hermes_constants import display_hermes_home args = event.get_command_args().strip().lower() config_path = _hermes_home / 'config.yaml' @@ -4665,7 +5857,7 @@ async def _handle_personality_command(self, event: MessageEvent) -> str: personalities = {} if not personalities: - return "No personalities configured in `~/.hermes/config.yaml`" + return f"No personalities configured in `{display_hermes_home()}/config.yaml`" if not args: lines = ["🎭 **Available Personalities**\n"] @@ -4749,6 +5941,7 @@ async def _handle_retry_command(self, event: MessageEvent) -> str: message_type=MessageType.TEXT, source=source, raw_message=event.raw_message, + channel_prompt=event.channel_prompt, ) # Let the normal message handler process it @@ -4826,11 +6019,13 @@ async def _handle_voice_command(self, event: MessageEvent) -> str: """Handle /voice [on|off|tts|channel|leave|status] command.""" args = event.get_command_args().strip().lower() chat_id = event.source.chat_id + platform = event.source.platform + voice_key = self._voice_key(platform, chat_id) - adapter = self.adapters.get(event.source.platform) + adapter = self.adapters.get(platform) if args in ("on", "enable"): - self._voice_mode[chat_id] = "voice_only" + self._voice_mode[voice_key] = "voice_only" self._save_voice_modes() if adapter: self._set_adapter_auto_tts_disabled(adapter, chat_id, disabled=False) @@ -4840,13 +6035,13 @@ async def _handle_voice_command(self, event: MessageEvent) -> str: "Use /voice tts to get voice replies for all messages." ) elif args in ("off", "disable"): - self._voice_mode[chat_id] = "off" + self._voice_mode[voice_key] = "off" self._save_voice_modes() if adapter: self._set_adapter_auto_tts_disabled(adapter, chat_id, disabled=True) return "Voice mode disabled. Text-only replies." elif args == "tts": - self._voice_mode[chat_id] = "all" + self._voice_mode[voice_key] = "all" self._save_voice_modes() if adapter: self._set_adapter_auto_tts_disabled(adapter, chat_id, disabled=False) @@ -4859,7 +6054,7 @@ async def _handle_voice_command(self, event: MessageEvent) -> str: elif args == "leave": return await self._handle_voice_channel_leave(event) elif args == "status": - mode = self._voice_mode.get(chat_id, "off") + mode = self._voice_mode.get(voice_key, "off") labels = { "off": "Off (text only)", "voice_only": "On (voice reply to voice messages)", @@ -4883,15 +6078,15 @@ async def _handle_voice_command(self, event: MessageEvent) -> str: return f"Voice mode: {labels.get(mode, mode)}" else: # Toggle: off → on, on/all → off - current = self._voice_mode.get(chat_id, "off") + current = self._voice_mode.get(voice_key, "off") if current == "off": - self._voice_mode[chat_id] = "voice_only" + self._voice_mode[voice_key] = "voice_only" self._save_voice_modes() if adapter: self._set_adapter_auto_tts_disabled(adapter, chat_id, disabled=False) return "Voice mode enabled." else: - self._voice_mode[chat_id] = "off" + self._voice_mode[voice_key] = "off" self._save_voice_modes() if adapter: self._set_adapter_auto_tts_disabled(adapter, chat_id, disabled=True) @@ -4929,8 +6124,7 @@ async def _handle_voice_channel_join(self, event: MessageEvent) -> str: if "pynacl" in err_lower or "nacl" in err_lower or "davey" in err_lower: return ( "Voice dependencies are missing (PyNaCl / davey). " - "Install or reinstall Hermes with the messaging extra, e.g. " - "`pip install hermes-agent[messaging]`." + f"Install with: `{sys.executable} -m pip install PyNaCl`" ) return f"Failed to join voice channel: {e}" @@ -4938,7 +6132,7 @@ async def _handle_voice_channel_join(self, event: MessageEvent) -> str: adapter._voice_text_channels[guild_id] = int(event.source.chat_id) if hasattr(adapter, "_voice_sources"): adapter._voice_sources[guild_id] = event.source.to_dict() - self._voice_mode[event.source.chat_id] = "all" + self._voice_mode[self._voice_key(event.source.platform, event.source.chat_id)] = "all" self._save_voice_modes() self._set_adapter_auto_tts_disabled(adapter, event.source.chat_id, disabled=False) return ( @@ -4965,7 +6159,7 @@ async def _handle_voice_channel_leave(self, event: MessageEvent) -> str: except Exception as e: logger.warning("Error leaving voice channel: %s", e) # Always clean up state even if leave raised an exception - self._voice_mode[event.source.chat_id] = "off" + self._voice_mode[self._voice_key(event.source.platform, event.source.chat_id)] = "off" self._save_voice_modes() self._set_adapter_auto_tts_disabled(adapter, event.source.chat_id, disabled=True) if hasattr(adapter, "_voice_input_callback"): @@ -4977,7 +6171,7 @@ def _handle_voice_timeout_cleanup(self, chat_id: str) -> None: Cleans up runner-side voice_mode state that the adapter cannot reach. """ - self._voice_mode[chat_id] = "off" + self._voice_mode[self._voice_key(Platform.DISCORD, chat_id)] = "off" self._save_voice_modes() adapter = self.adapters.get(Platform.DISCORD) self._set_adapter_auto_tts_disabled(adapter, chat_id, disabled=True) @@ -5063,7 +6257,7 @@ def _should_send_voice_reply( return False chat_id = event.source.chat_id - voice_mode = self._voice_mode.get(chat_id, "off") + voice_mode = self._voice_mode.get(self._voice_key(event.source.platform, chat_id), "off") is_voice_input = (event.message_type == MessageType.VOICE) should = ( @@ -5258,7 +6452,7 @@ async def _handle_rollback_command(self, event: MessageEvent) -> str: max_snapshots=cp_cfg.get("max_snapshots", 50), ) - cwd = os.getenv("MESSAGING_CWD", str(Path.home())) + cwd = os.getenv("TERMINAL_CWD", str(Path.home())) arg = event.get_command_args().strip() if not arg: @@ -5376,17 +6570,23 @@ def run_sync(): session_id=task_id, platform=platform_key, user_id=source.user_id, + user_name=source.user_name, + chat_id=source.chat_id, + chat_name=source.chat_name, + chat_type=source.chat_type, + thread_id=source.thread_id, session_db=self._session_db, fallback_model=self._fallback_model, ) + try: + return agent.run_conversation( + user_message=prompt, + task_id=task_id, + ) + finally: + self._cleanup_agent_resources(agent) - return agent.run_conversation( - user_message=prompt, - task_id=task_id, - ) - - loop = asyncio.get_event_loop() - result = await loop.run_in_executor(None, run_sync) + result = await self._run_in_executor_with_context(run_sync) response = result.get("final_response", "") if result else "" if not response and result and result.get("error"): @@ -5425,7 +6625,7 @@ def run_sync(): pass # Send media files - for media_path in (media_files or []): + for media_path, _is_voice in (media_files or []): try: await adapter.send_document( chat_id=source.chat_id, @@ -5562,14 +6762,16 @@ def run_sync(): skip_context_files=True, persist_session=False, ) - return agent.run_conversation( - user_message=btw_prompt, - conversation_history=history_snapshot, - task_id=task_id, - ) + try: + return agent.run_conversation( + user_message=btw_prompt, + conversation_history=history_snapshot, + task_id=task_id, + ) + finally: + self._cleanup_agent_resources(agent) - loop = asyncio.get_event_loop() - result = await loop.run_in_executor(None, run_sync) + result = await self._run_in_executor_with_context(run_sync) response = (result.get("final_response") or "") if result else "" if not response and result and result.get("error"): @@ -5601,7 +6803,7 @@ def run_sync(): except Exception: pass - for media_path in (media_files or []): + for media_path, _is_voice in (media_files or []): try: await adapter.send_file(chat_id=source.chat_id, file_path=media_path) except Exception: @@ -5892,45 +7094,49 @@ async def _handle_compress_command(self, event: MessageEvent) -> str: model=model, max_iterations=4, quiet_mode=True, + skip_memory=True, enabled_toolsets=["memory"], session_id=session_entry.session_id, ) - tmp_agent._print_fn = lambda *a, **kw: None - - compressor = tmp_agent.context_compressor - compress_start = compressor.protect_first_n - compress_start = compressor._align_boundary_forward(msgs, compress_start) - compress_end = compressor._find_tail_cut_by_tokens(msgs, compress_start) - if compress_start >= compress_end: - return "Nothing to compress yet (the transcript is still all protected context)." - - loop = asyncio.get_event_loop() - compressed, _ = await loop.run_in_executor( - None, - lambda: tmp_agent._compress_context(msgs, "", approx_tokens=approx_tokens, focus_topic=focus_topic) - ) + try: + tmp_agent._print_fn = lambda *a, **kw: None + + compressor = tmp_agent.context_compressor + compress_start = compressor.protect_first_n + compress_start = compressor._align_boundary_forward(msgs, compress_start) + compress_end = compressor._find_tail_cut_by_tokens(msgs, compress_start) + if compress_start >= compress_end: + return "Nothing to compress yet (the transcript is still all protected context)." + + loop = asyncio.get_running_loop() + compressed, _ = await loop.run_in_executor( + None, + lambda: tmp_agent._compress_context(msgs, "", approx_tokens=approx_tokens, focus_topic=focus_topic) + ) - # _compress_context already calls end_session() on the old session - # (preserving its full transcript in SQLite) and creates a new - # session_id for the continuation. Write the compressed messages - # into the NEW session so the original history stays searchable. - new_session_id = tmp_agent.session_id - if new_session_id != session_entry.session_id: - session_entry.session_id = new_session_id - self.session_store._save() + # _compress_context already calls end_session() on the old session + # (preserving its full transcript in SQLite) and creates a new + # session_id for the continuation. Write the compressed messages + # into the NEW session so the original history stays searchable. + new_session_id = tmp_agent.session_id + if new_session_id != session_entry.session_id: + session_entry.session_id = new_session_id + self.session_store._save() - self.session_store.rewrite_transcript(new_session_id, compressed) - # Reset stored token count — transcript changed, old value is stale - self.session_store.update_session( - session_entry.session_key, last_prompt_tokens=0 - ) - new_tokens = estimate_messages_tokens_rough(compressed) - summary = summarize_manual_compression( - msgs, - compressed, - approx_tokens, - new_tokens, - ) + self.session_store.rewrite_transcript(new_session_id, compressed) + # Reset stored token count — transcript changed, old value is stale + self.session_store.update_session( + session_entry.session_key, last_prompt_tokens=0 + ) + new_tokens = estimate_messages_tokens_rough(compressed) + summary = summarize_manual_compression( + msgs, + compressed, + approx_tokens, + new_tokens, + ) + finally: + self._cleanup_agent_resources(tmp_agent) lines = [f"🗜️ {summary['headline']}"] if focus_topic: lines.append(f"Focus: \"{focus_topic}\"") @@ -6049,13 +7255,13 @@ async def _handle_resume_command(self, event: MessageEvent) -> str: logger.debug("Memory flush on resume failed: %s", e) # Clear any running agent for this session key - if session_key in self._running_agents: - del self._running_agents[session_key] + self._release_running_agent_state(session_key) # Switch the session entry to point at the old session new_entry = self.session_store.switch_session(session_key, target_id) if not new_entry: return "Failed to switch session." + self._clear_session_boundary_security_state(session_key) # Get the title for confirmation title = self._session_db.get_session_title(target_id) or name @@ -6130,6 +7336,7 @@ async def _handle_branch_command(self, event: MessageEvent) -> str: tool_calls=msg.get("tool_calls"), tool_call_id=msg.get("tool_call_id"), reasoning=msg.get("reasoning"), + reasoning_content=msg.get("reasoning_content"), ) except Exception: pass # Best-effort copy @@ -6144,6 +7351,7 @@ async def _handle_branch_command(self, event: MessageEvent) -> str: new_entry = self.session_store.switch_session(session_key, new_session_id) if not new_entry: return "Branch created but failed to switch to it." + self._clear_session_boundary_security_state(session_key) # Evict any cached agent for this session self._evict_cached_agent(session_key) @@ -6178,6 +7386,38 @@ async def _handle_usage_command(self, event: MessageEvent) -> str: if cached: agent = cached[0] + # Resolve provider/base_url/api_key for the account-usage fetch. + # Prefer the live agent; fall back to persisted billing data on the + # SessionDB row so `/usage` still returns account info between turns + # when no agent is resident. + provider = getattr(agent, "provider", None) if agent and agent is not _AGENT_PENDING_SENTINEL else None + base_url = getattr(agent, "base_url", None) if agent and agent is not _AGENT_PENDING_SENTINEL else None + api_key = getattr(agent, "api_key", None) if agent and agent is not _AGENT_PENDING_SENTINEL else None + if not provider and getattr(self, "_session_db", None) is not None: + try: + _entry_for_billing = self.session_store.get_or_create_session(source) + persisted = self._session_db.get_session(_entry_for_billing.session_id) or {} + except Exception: + persisted = {} + provider = provider or persisted.get("billing_provider") + base_url = base_url or persisted.get("billing_base_url") + + # Fetch account usage off the event loop so slow provider APIs don't + # block the gateway. Failures are non-fatal -- account_lines stays []. + account_lines: list[str] = [] + if provider: + try: + account_snapshot = await asyncio.to_thread( + fetch_account_usage, + provider, + base_url=base_url, + api_key=api_key, + ) + except Exception: + account_snapshot = None + if account_snapshot: + account_lines = render_account_usage_lines(account_snapshot, markdown=True) + if agent and hasattr(agent, "session_total_tokens") and agent.session_api_calls > 0: lines = [] @@ -6235,6 +7475,10 @@ async def _handle_usage_command(self, event: MessageEvent) -> str: if ctx.compression_count: lines.append(f"Compressions: {ctx.compression_count}") + if account_lines: + lines.append("") + lines.extend(account_lines) + return "\n".join(lines) # No agent at all -- check session history for a rough count @@ -6244,19 +7488,27 @@ async def _handle_usage_command(self, event: MessageEvent) -> str: from agent.model_metadata import estimate_messages_tokens_rough msgs = [m for m in history if m.get("role") in ("user", "assistant") and m.get("content")] approx = estimate_messages_tokens_rough(msgs) - return ( - f"📊 **Session Info**\n" - f"Messages: {len(msgs)}\n" - f"Estimated context: ~{approx:,} tokens\n" - f"_(Detailed usage available after the first agent response)_" - ) + lines = [ + "📊 **Session Info**", + f"Messages: {len(msgs)}", + f"Estimated context: ~{approx:,} tokens", + "_(Detailed usage available after the first agent response)_", + ] + if account_lines: + lines.append("") + lines.extend(account_lines) + return "\n".join(lines) + if account_lines: + return "\n".join(account_lines) return "No usage data available for this session." async def _handle_insights_command(self, event: MessageEvent) -> str: """Handle /insights command -- show usage insights and analytics.""" - import asyncio as _asyncio - args = event.get_command_args().strip() + + # Normalize Unicode dashes (Telegram/iOS auto-converts -- to em/en dash) + args = re.sub(r'[\u2012\u2013\u2014\u2015](days|source)', r'--\1', args) + days = 30 source = None @@ -6284,7 +7536,7 @@ async def _handle_insights_command(self, event: MessageEvent) -> str: from hermes_state import SessionDB from agent.insights import InsightsEngine - loop = _asyncio.get_event_loop() + loop = asyncio.get_running_loop() def _run_insights(): db = SessionDB() @@ -6301,9 +7553,9 @@ def _run_insights(): async def _handle_reload_mcp_command(self, event: MessageEvent) -> str: """Handle /reload-mcp command -- disconnect and reconnect all MCP servers.""" - loop = asyncio.get_event_loop() + loop = asyncio.get_running_loop() try: - from tools.mcp_tool import shutdown_mcp_servers, discover_mcp_tools, _load_mcp_config, _servers, _lock + from tools.mcp_tool import shutdown_mcp_servers, discover_mcp_tools, _servers, _lock # Capture old server names before shutdown with _lock: @@ -6476,60 +7728,49 @@ async def _handle_deny_command(self, event: MessageEvent) -> str: Platform.TELEGRAM, Platform.DISCORD, Platform.SLACK, Platform.WHATSAPP, Platform.SIGNAL, Platform.MATTERMOST, Platform.MATRIX, Platform.HOMEASSISTANT, Platform.EMAIL, Platform.SMS, Platform.DINGTALK, - Platform.FEISHU, Platform.WECOM, Platform.WECOM_CALLBACK, Platform.WEIXIN, Platform.BLUEBUBBLES, Platform.LOCAL, + Platform.FEISHU, Platform.WECOM, Platform.WECOM_CALLBACK, Platform.WEIXIN, Platform.BLUEBUBBLES, Platform.QQBOT, Platform.LOCAL, }) async def _handle_debug_command(self, event: MessageEvent) -> str: - """Handle /debug — upload debug report + logs and return paste URLs.""" + """Handle /debug — upload debug report (summary only) and return paste URLs. + + Gateway uploads ONLY the summary report (system info + log tails), + NOT full log files, to protect conversation privacy. Users who need + full log uploads should use ``hermes debug share`` from the CLI. + """ import asyncio from hermes_cli.debug import ( - _capture_dump, collect_debug_report, _read_full_log, - upload_to_pastebin, + _capture_dump, collect_debug_report, + upload_to_pastebin, _schedule_auto_delete, + _GATEWAY_PRIVACY_NOTICE, _best_effort_sweep_expired_pastes, ) loop = asyncio.get_running_loop() # Run blocking I/O (dump capture, log reads, uploads) in a thread. def _collect_and_upload(): + _best_effort_sweep_expired_pastes() dump_text = _capture_dump() report = collect_debug_report(log_lines=200, dump_text=dump_text) - agent_log = _read_full_log("agent") - gateway_log = _read_full_log("gateway") - - if agent_log: - agent_log = dump_text + "\n\n--- full agent.log ---\n" + agent_log - if gateway_log: - gateway_log = dump_text + "\n\n--- full gateway.log ---\n" + gateway_log urls = {} - failures = [] - try: urls["Report"] = upload_to_pastebin(report) except Exception as exc: return f"✗ Failed to upload debug report: {exc}" - if agent_log: - try: - urls["agent.log"] = upload_to_pastebin(agent_log) - except Exception: - failures.append("agent.log") + # Schedule auto-deletion after 6 hours + _schedule_auto_delete(list(urls.values())) - if gateway_log: - try: - urls["gateway.log"] = upload_to_pastebin(gateway_log) - except Exception: - failures.append("gateway.log") - - lines = ["**Debug report uploaded:**", ""] + lines = [_GATEWAY_PRIVACY_NOTICE, "", "**Debug report uploaded:**", ""] label_width = max(len(k) for k in urls) for label, url in urls.items(): lines.append(f"`{label:<{label_width}}` {url}") - if failures: - lines.append(f"\n_(failed to upload: {', '.join(failures)})_") - - lines.append("\nShare these links with the Hermes team for support.") + lines.append("") + lines.append("⏱ Pastes will auto-delete in 6 hours.") + lines.append("For full log uploads, use `hermes debug share` from the CLI.") + lines.append("Share these links with the Hermes team for support.") return "\n".join(lines) return await loop.run_in_executor(None, _collect_and_upload) @@ -6654,9 +7895,6 @@ async def _watch_update_progress( the messenger. The user's next message is intercepted by ``_handle_message`` and written to ``.update_response``. """ - import json - import re as _re - pending_path = _hermes_home / ".update_pending.json" claimed_path = _hermes_home / ".update_pending.claimed.json" output_path = _hermes_home / ".update_output.txt" @@ -6701,7 +7939,7 @@ async def _watch_update_progress( return def _strip_ansi(text: str) -> str: - return _re.sub(r'\x1b\[[0-9;]*[A-Za-z]', '', text) + return re.sub(r'\x1b\[[0-9;]*[A-Za-z]', '', text) bytes_sent = 0 last_stream_time = loop.time() @@ -6849,9 +8087,6 @@ async def _send_update_notification(self) -> bool: cannot resolve the adapter (e.g. after a gateway restart where the platform hasn't reconnected yet). """ - import json - import re as _re - pending_path = _hermes_home / ".update_pending.json" claimed_path = _hermes_home / ".update_pending.claimed.json" output_path = _hermes_home / ".update_output.txt" @@ -6897,7 +8132,7 @@ async def _send_update_notification(self) -> bool: if adapter and chat_id: # Strip ANSI escape codes for clean display - output = _re.sub(r'\x1b\[[0-9;]*m', '', output).strip() + output = re.sub(r'\x1b\[[0-9;]*m', '', output).strip() if output: if len(output) > 3500: output = "…" + output[-3500:] @@ -6930,14 +8165,12 @@ async def _send_update_notification(self) -> bool: async def _send_restart_notification(self) -> None: """Notify the chat that initiated /restart that the gateway is back.""" - import json as _json - notify_path = _hermes_home / ".restart_notify.json" if not notify_path.exists(): return try: - data = _json.loads(notify_path.read_text()) + data = json.loads(notify_path.read_text()) platform_str = data.get("platform") chat_id = data.get("chat_id") thread_id = data.get("thread_id") @@ -6994,7 +8227,13 @@ def _clear_session_env(self, tokens: list) -> None: """Restore session context variables to their pre-handler values.""" from gateway.session_context import clear_session_vars clear_session_vars(tokens) - + + async def _run_in_executor_with_context(self, func, *args): + """Run blocking work in the thread pool while preserving session contextvars.""" + loop = asyncio.get_running_loop() + ctx = copy_context() + return await loop.run_in_executor(None, ctx.run, func, *args) + async def _enrich_message_with_vision( self, user_text: str, @@ -7017,7 +8256,6 @@ async def _enrich_message_with_vision( The enriched message string with vision descriptions prepended. """ from tools.vision_tools import vision_analyze_tool - import json as _json analysis_prompt = ( "Describe everything visible in this image in thorough detail. " @@ -7033,7 +8271,7 @@ async def _enrich_message_with_vision( image_url=path, user_prompt=analysis_prompt, ) - result = _json.loads(result_json) + result = json.loads(result_json) if result.get("success"): description = result.get("analysis", "") enriched_parts.append( @@ -7092,7 +8330,6 @@ async def _enrich_message_with_transcription( return disabled_note from tools.transcription_tools import transcribe_audio - import asyncio enriched_parts = [] for path in audio_paths: @@ -7149,14 +8386,75 @@ async def _enrich_message_with_transcription( return prefix return user_text - async def _inject_watch_notification(self, synth_text: str, original_event) -> None: + def _build_process_event_source(self, evt: dict): + """Resolve the canonical source for a synthetic background-process event. + + Prefer the persisted session-store origin for the event's session key. + Falling back to the currently active foreground event is what causes + cross-topic bleed, so don't do that. + """ + from gateway.session import SessionSource + + session_key = str(evt.get("session_key") or "").strip() + derived_platform = "" + derived_chat_type = "" + derived_chat_id = "" + + if session_key: + try: + self.session_store._ensure_loaded() + entry = self.session_store._entries.get(session_key) + if entry and getattr(entry, "origin", None): + return entry.origin + except Exception as exc: + logger.debug( + "Synthetic process-event session-store lookup failed for %s: %s", + session_key, + exc, + ) + + _parsed = _parse_session_key(session_key) + if _parsed: + derived_platform = _parsed["platform"] + derived_chat_type = _parsed["chat_type"] + derived_chat_id = _parsed["chat_id"] + + platform_name = str(evt.get("platform") or derived_platform or "").strip().lower() + chat_type = str(evt.get("chat_type") or derived_chat_type or "").strip().lower() + chat_id = str(evt.get("chat_id") or derived_chat_id or "").strip() + if not platform_name or not chat_type or not chat_id: + return None + + try: + platform = Platform(platform_name) + except Exception: + logger.warning( + "Synthetic process event has invalid platform metadata: %r", + platform_name, + ) + return None + + return SessionSource( + platform=platform, + chat_id=chat_id, + chat_type=chat_type, + thread_id=str(evt.get("thread_id") or "").strip() or None, + user_id=str(evt.get("user_id") or "").strip() or None, + user_name=str(evt.get("user_name") or "").strip() or None, + ) + + async def _inject_watch_notification(self, synth_text: str, evt: dict) -> None: """Inject a watch-pattern notification as a synthetic message event. - Uses the source from the original user event to route the notification - back to the correct chat/adapter. + Routing must come from the queued watch event itself, not from whatever + foreground message happened to be active when the queue was drained. """ - source = getattr(original_event, "source", None) + source = self._build_process_event_source(evt) if not source: + logger.warning( + "Dropping watch notification with no routing metadata for process %s", + evt.get("session_id", "unknown"), + ) return platform_name = source.platform.value if hasattr(source.platform, "value") else str(source.platform) adapter = None @@ -7167,14 +8465,18 @@ async def _inject_watch_notification(self, synth_text: str, original_event) -> N if not adapter: return try: - from gateway.platforms.base import MessageEvent, MessageType synth_event = MessageEvent( text=synth_text, message_type=MessageType.TEXT, source=source, internal=True, ) - logger.info("Watch pattern notification — injecting for %s", platform_name) + logger.info( + "Watch pattern notification — injecting for %s chat=%s thread=%s", + platform_name, + source.chat_id, + source.thread_id, + ) await adapter.handle_message(synth_event) except Exception as e: logger.error("Watch notification injection error: %s", e) @@ -7244,33 +8546,41 @@ async def _run_process_watcher(self, watcher: dict) -> None: f"Command: {session.command}\n" f"Output:\n{_out}]" ) + source = self._build_process_event_source({ + "session_id": session_id, + "session_key": session_key, + "platform": platform_name, + "chat_id": chat_id, + "thread_id": thread_id, + "user_id": user_id, + "user_name": user_name, + }) + if not source: + logger.warning( + "Dropping completion notification with no routing metadata for process %s", + session_id, + ) + break + adapter = None for p, a in self.adapters.items(): - if p.value == platform_name: + if p == source.platform: adapter = a break - if adapter and chat_id: + if adapter and source.chat_id: try: - from gateway.platforms.base import MessageEvent, MessageType - from gateway.session import SessionSource - from gateway.config import Platform - _platform_enum = Platform(platform_name) - _source = SessionSource( - platform=_platform_enum, - chat_id=chat_id, - thread_id=thread_id or None, - user_id=user_id or None, - user_name=user_name or None, - ) synth_event = MessageEvent( text=synth_text, message_type=MessageType.TEXT, - source=_source, + source=source, internal=True, ) logger.info( - "Process %s finished — injecting agent notification for session %s", - session_id, session_key, + "Process %s finished — injecting agent notification for session %s chat=%s thread=%s", + session_id, + session_key, + source.chat_id, + source.thread_id, ) await adapter.handle_message(synth_event) except Exception as e: @@ -7392,6 +8702,148 @@ def _is_intentional_model_switch(self, session_key: str, agent_model: str) -> bo override = self._session_model_overrides.get(session_key) return override is not None and override.get("model") == agent_model + def _release_running_agent_state( + self, + session_key: str, + *, + run_generation: Optional[int] = None, + ) -> bool: + """Pop ALL per-running-agent state entries for ``session_key``. + + Replaces ad-hoc ``del self._running_agents[key]`` calls scattered + across the gateway. Those sites had drifted: some popped only + ``_running_agents``; some also ``_running_agents_ts``; only one + path also cleared ``_busy_ack_ts``. Each missed entry was a + small, persistent leak — a (str_key → float) tuple per session + per gateway lifetime. + + Use this at every site that ends a running turn, regardless of + cause (normal completion, /stop, /reset, /resume, sentinel + cleanup, stale-eviction). Per-session state that PERSISTS + across turns (``_session_model_overrides``, ``_voice_mode``, + ``_pending_approvals``, ``_update_prompt_pending``) is NOT + touched here — those have their own lifecycles. + + When ``run_generation`` is provided, only clear the slot if that + generation is still current for the session. This prevents an + older async run whose generation was bumped by /stop or /new from + clobbering a newer run's state during its own unwind. Returns + True when the slot was cleared, False when an ownership guard + blocked it. + """ + if not session_key: + return False + if run_generation is not None and not self._is_session_run_current( + session_key, run_generation + ): + return False + self._running_agents.pop(session_key, None) + self._running_agents_ts.pop(session_key, None) + if hasattr(self, "_busy_ack_ts"): + self._busy_ack_ts.pop(session_key, None) + return True + + def _clear_session_boundary_security_state(self, session_key: str) -> None: + """Clear approval state that must not survive a real conversation switch.""" + if not session_key: + return + + pending_approvals = getattr(self, "_pending_approvals", None) + if isinstance(pending_approvals, dict): + pending_approvals.pop(session_key, None) + + try: + from tools.approval import clear_session as _clear_approval_session + except Exception: + return + + try: + _clear_approval_session(session_key) + except Exception as e: + logger.debug( + "Failed to clear approval state for session boundary %s: %s", + session_key, + e, + ) + + def _begin_session_run_generation(self, session_key: str) -> int: + """Claim a fresh run generation token for ``session_key``. + + Every top-level gateway turn gets a monotonically increasing token. + If a later command like /stop or /new invalidates that token while the + old worker is still unwinding, the late result can be recognized and + dropped instead of bleeding into the fresh session. + """ + if not session_key: + return 0 + generations = self.__dict__.get("_session_run_generation") + if generations is None: + generations = {} + self._session_run_generation = generations + next_generation = int(generations.get(session_key, 0)) + 1 + generations[session_key] = next_generation + return next_generation + + def _invalidate_session_run_generation(self, session_key: str, *, reason: str = "") -> int: + """Invalidate any in-flight run token for ``session_key``.""" + generation = self._begin_session_run_generation(session_key) + if reason: + logger.info( + "Invalidated run generation for %s → %d (%s)", + session_key[:20], + generation, + reason, + ) + return generation + + def _is_session_run_current(self, session_key: str, generation: int) -> bool: + """Return True when ``generation`` is still current for ``session_key``.""" + if not session_key: + return True + generations = self.__dict__.get("_session_run_generation") or {} + return int(generations.get(session_key, 0)) == int(generation) + + def _bind_adapter_run_generation( + self, + adapter: Any, + session_key: str, + generation: int | None, + ) -> None: + """Bind a gateway run generation to the adapter's active-session event.""" + if not adapter or not session_key or generation is None: + return + try: + interrupt_event = getattr(adapter, "_active_sessions", {}).get(session_key) + if interrupt_event is not None: + setattr(interrupt_event, "_hermes_run_generation", int(generation)) + except Exception: + pass + + async def _interrupt_and_clear_session( + self, + session_key: str, + source: SessionSource, + *, + interrupt_reason: str, + invalidation_reason: str, + release_running_state: bool = True, + ) -> None: + """Interrupt the current run and clear queued session state consistently.""" + if not session_key: + return + running_agent = self._running_agents.get(session_key) + if running_agent and running_agent is not _AGENT_PENDING_SENTINEL: + running_agent.interrupt(interrupt_reason) + self._invalidate_session_run_generation(session_key, reason=invalidation_reason) + adapter = self.adapters.get(source.platform) + if adapter and hasattr(adapter, "interrupt_session_activity"): + await adapter.interrupt_session_activity(session_key, source.chat_id) + if adapter and hasattr(adapter, "get_pending_message"): + adapter.get_pending_message(session_key) # consume and discard + self._pending_messages.pop(session_key, None) + if release_running_state: + self._release_running_agent_state(session_key) + def _evict_cached_agent(self, session_key: str) -> None: """Remove a cached agent for a session (called on /new, /model, etc).""" _lock = getattr(self, "_agent_cache_lock", None) @@ -7399,6 +8851,448 @@ def _evict_cached_agent(self, session_key: str) -> None: with _lock: self._agent_cache.pop(session_key, None) + def _release_evicted_agent_soft(self, agent: Any) -> None: + """Soft cleanup for cache-evicted agents — preserves session tool state. + + Called from _enforce_agent_cache_cap and _sweep_idle_cached_agents. + Distinct from _cleanup_agent_resources (full teardown) because a + cache-evicted session may resume at any time — its terminal + sandbox, browser daemon, and tracked bg processes must outlive + the Python AIAgent instance so the next agent built for the + same task_id inherits them. + """ + if agent is None: + return + try: + if hasattr(agent, "release_clients"): + agent.release_clients() + else: + # Older agent instance (shouldn't happen in practice) — + # fall back to the legacy full-close path. + self._cleanup_agent_resources(agent) + except Exception: + pass + + def _enforce_agent_cache_cap(self) -> None: + """Evict oldest cached agents when cache exceeds _AGENT_CACHE_MAX_SIZE. + + Must be called with _agent_cache_lock held. Resource cleanup + (memory provider shutdown, tool resource close) is scheduled + on a daemon thread so the caller doesn't block on slow teardown + while holding the cache lock. + + Agents currently in _running_agents are SKIPPED — their clients, + terminal sandboxes, background processes, and child subagents + are all in active use by the running turn. Evicting them would + tear down those resources mid-turn and crash the request. If + every candidate in the LRU order is active, we simply leave the + cache over the cap; it will be re-checked on the next insert. + """ + _cache = getattr(self, "_agent_cache", None) + if _cache is None: + return + # OrderedDict.popitem(last=False) pops oldest; plain dict lacks the + # arg so skip enforcement if a test fixture swapped the cache type. + if not hasattr(_cache, "move_to_end"): + return + + # Snapshot of agent instances that are actively mid-turn. Use id() + # so the lookup is O(1) and doesn't depend on AIAgent.__eq__ (which + # MagicMock overrides in tests). + running_ids = { + id(a) + for a in getattr(self, "_running_agents", {}).values() + if a is not None and a is not _AGENT_PENDING_SENTINEL + } + + # Walk LRU → MRU and evict excess-LRU entries that aren't mid-turn. + # We only consider entries in the first (size - cap) LRU positions + # as eviction candidates. If one of those slots is held by an + # active agent, we SKIP it without compensating by evicting a + # newer entry — that would penalise a freshly-inserted session + # (which has no cache history to retain) while protecting an + # already-cached long-running one. The cache may therefore stay + # temporarily over cap; it will re-check on the next insert, + # after active turns have finished. + excess = max(0, len(_cache) - _AGENT_CACHE_MAX_SIZE) + evict_plan: List[tuple] = [] # [(key, agent), ...] + if excess > 0: + ordered_keys = list(_cache.keys()) + for key in ordered_keys[:excess]: + entry = _cache.get(key) + agent = entry[0] if isinstance(entry, tuple) and entry else None + if agent is not None and id(agent) in running_ids: + continue # active mid-turn; don't evict, don't substitute + evict_plan.append((key, agent)) + + for key, _ in evict_plan: + _cache.pop(key, None) + + remaining_over_cap = len(_cache) - _AGENT_CACHE_MAX_SIZE + if remaining_over_cap > 0: + logger.warning( + "Agent cache over cap (%d > %d); %d excess slot(s) held by " + "mid-turn agents — will re-check on next insert.", + len(_cache), _AGENT_CACHE_MAX_SIZE, remaining_over_cap, + ) + + for key, agent in evict_plan: + logger.info( + "Agent cache at cap; evicting LRU session=%s (cache_size=%d)", + key, len(_cache), + ) + if agent is not None: + threading.Thread( + target=self._release_evicted_agent_soft, + args=(agent,), + daemon=True, + name=f"agent-cache-evict-{key[:24]}", + ).start() + + def _sweep_idle_cached_agents(self) -> int: + """Evict cached agents whose AIAgent has been idle > _AGENT_CACHE_IDLE_TTL_SECS. + + Safe to call from the session expiry watcher without holding the + cache lock — acquires it internally. Returns the number of entries + evicted. Resource cleanup is scheduled on daemon threads. + + Agents currently in _running_agents are SKIPPED for the same reason + as _enforce_agent_cache_cap: tearing down an active turn's clients + mid-flight would crash the request. + """ + _cache = getattr(self, "_agent_cache", None) + _lock = getattr(self, "_agent_cache_lock", None) + if _cache is None or _lock is None: + return 0 + now = time.time() + to_evict: List[tuple] = [] + running_ids = { + id(a) + for a in getattr(self, "_running_agents", {}).values() + if a is not None and a is not _AGENT_PENDING_SENTINEL + } + with _lock: + for key, entry in list(_cache.items()): + agent = entry[0] if isinstance(entry, tuple) and entry else None + if agent is None: + continue + if id(agent) in running_ids: + continue # mid-turn — don't tear it down + last_activity = getattr(agent, "_last_activity_ts", None) + if last_activity is None: + continue + if (now - last_activity) > _AGENT_CACHE_IDLE_TTL_SECS: + to_evict.append((key, agent)) + for key, _ in to_evict: + _cache.pop(key, None) + for key, agent in to_evict: + logger.info( + "Agent cache idle-TTL evict: session=%s (idle=%.0fs)", + key, now - getattr(agent, "_last_activity_ts", now), + ) + threading.Thread( + target=self._release_evicted_agent_soft, + args=(agent,), + daemon=True, + name=f"agent-cache-idle-{key[:24]}", + ).start() + return len(to_evict) + + # ------------------------------------------------------------------ + # Proxy mode: forward messages to a remote Hermes API server + # ------------------------------------------------------------------ + + def _get_proxy_url(self) -> Optional[str]: + """Return the proxy URL if proxy mode is configured, else None. + + Checks GATEWAY_PROXY_URL env var first (convenient for Docker), + then ``gateway.proxy_url`` in config.yaml. + """ + url = os.getenv("GATEWAY_PROXY_URL", "").strip() + if url: + return url.rstrip("/") + cfg = _load_gateway_config() + url = (cfg.get("gateway") or {}).get("proxy_url", "").strip() + if url: + return url.rstrip("/") + return None + + async def _run_agent_via_proxy( + self, + message: str, + context_prompt: str, + history: List[Dict[str, Any]], + source: "SessionSource", + session_id: str, + session_key: str = None, + run_generation: Optional[int] = None, + event_message_id: Optional[str] = None, + ) -> Dict[str, Any]: + """Forward the message to a remote Hermes API server instead of + running a local AIAgent. + + When ``GATEWAY_PROXY_URL`` (or ``gateway.proxy_url`` in config.yaml) + is set, the gateway becomes a thin relay: it handles platform I/O + (encryption, threading, media) and delegates all agent work to the + remote server via ``POST /v1/chat/completions`` with SSE streaming. + + This lets a Docker container handle Matrix E2EE while the actual + agent runs on the host with full access to local files, memory, + skills, and a unified session store. + """ + try: + from aiohttp import ClientSession as _AioClientSession, ClientTimeout + except ImportError: + return { + "final_response": "⚠️ Proxy mode requires aiohttp. Install with: pip install aiohttp", + "messages": [], + "api_calls": 0, + "tools": [], + } + + proxy_url = self._get_proxy_url() + if not proxy_url: + return { + "final_response": "⚠️ Proxy URL not configured (GATEWAY_PROXY_URL or gateway.proxy_url)", + "messages": [], + "api_calls": 0, + "tools": [], + } + + proxy_key = os.getenv("GATEWAY_PROXY_KEY", "").strip() + + def _run_still_current() -> bool: + if run_generation is None or not session_key: + return True + return self._is_session_run_current(session_key, run_generation) + + # Build messages in OpenAI chat format -------------------------- + # + # The remote api_server can maintain session continuity via + # X-Hermes-Session-Id, so it loads its own history. We only + # need to send the current user message. If the remote has + # no history for this session yet, include what we have locally + # so the first exchange has context. + # + # We always include the current message. For history, send a + # compact version (text-only user/assistant turns) — the remote + # handles tool replay and system prompts. + api_messages: List[Dict[str, str]] = [] + + if context_prompt: + api_messages.append({"role": "system", "content": context_prompt}) + + for msg in history: + role = msg.get("role") + content = msg.get("content") + if role in ("user", "assistant") and content: + api_messages.append({"role": role, "content": content}) + + api_messages.append({"role": "user", "content": message}) + + # HTTP headers --------------------------------------------------- + headers: Dict[str, str] = {"Content-Type": "application/json"} + if proxy_key: + headers["Authorization"] = f"Bearer {proxy_key}" + if session_id: + headers["X-Hermes-Session-Id"] = session_id + + body = { + "model": "hermes-agent", + "messages": api_messages, + "stream": True, + } + + # Set up platform streaming if available ------------------------- + _stream_consumer = None + _scfg = getattr(getattr(self, "config", None), "streaming", None) + if _scfg is None: + from gateway.config import StreamingConfig + _scfg = StreamingConfig() + + platform_key = _platform_config_key(source.platform) + user_config = _load_gateway_config() + from gateway.display_config import resolve_display_setting + _plat_streaming = resolve_display_setting( + user_config, platform_key, "streaming" + ) + _streaming_enabled = ( + _scfg.enabled and _scfg.transport != "off" + if _plat_streaming is None + else bool(_plat_streaming) + ) + + if source.thread_id: + _thread_metadata: Optional[Dict[str, Any]] = {"thread_id": source.thread_id} + else: + _thread_metadata = None + + if _streaming_enabled: + try: + from gateway.stream_consumer import GatewayStreamConsumer, StreamConsumerConfig + _adapter = self.adapters.get(source.platform) + if _adapter: + _adapter_supports_edit = getattr(_adapter, "SUPPORTS_MESSAGE_EDITING", True) + _effective_cursor = _scfg.cursor if _adapter_supports_edit else "" + _buffer_only = False + if source.platform == Platform.MATRIX: + _effective_cursor = "" + _buffer_only = True + _consumer_cfg = StreamConsumerConfig( + edit_interval=_scfg.edit_interval, + buffer_threshold=_scfg.buffer_threshold, + cursor=_effective_cursor, + buffer_only=_buffer_only, + ) + _stream_consumer = GatewayStreamConsumer( + adapter=_adapter, + chat_id=source.chat_id, + config=_consumer_cfg, + metadata=_thread_metadata, + ) + except Exception as _sc_err: + logger.debug("Proxy: could not set up stream consumer: %s", _sc_err) + + # Run the stream consumer task in the background + stream_task = None + if _stream_consumer: + stream_task = asyncio.create_task(_stream_consumer.run()) + + # Send typing indicator + _adapter = self.adapters.get(source.platform) + if _adapter: + try: + await _adapter.send_typing(source.chat_id, metadata=_thread_metadata) + except Exception: + pass + + # Make the HTTP request with SSE streaming ----------------------- + full_response = "" + _start = time.time() + + try: + _timeout = ClientTimeout(total=0, sock_read=1800) + async with _AioClientSession(timeout=_timeout) as session: + async with session.post( + f"{proxy_url}/v1/chat/completions", + json=body, + headers=headers, + ) as resp: + if resp.status != 200: + error_text = await resp.text() + logger.warning( + "Proxy error (%d) from %s: %s", + resp.status, proxy_url, error_text[:500], + ) + return { + "final_response": f"⚠️ Proxy error ({resp.status}): {error_text[:300]}", + "messages": [], + "api_calls": 0, + "tools": [], + } + + # Parse SSE stream + buffer = "" + async for chunk in resp.content.iter_any(): + if not _run_still_current(): + logger.info( + "Discarding stale proxy stream for %s — generation %d is no longer current", + session_key[:20] if session_key else "?", + run_generation or 0, + ) + return { + "final_response": "", + "messages": [], + "api_calls": 0, + "tools": [], + "history_offset": len(history), + "session_id": session_id, + "response_previewed": False, + } + text = chunk.decode("utf-8", errors="replace") + buffer += text + + # Process complete SSE lines + while "\n" in buffer: + line, buffer = buffer.split("\n", 1) + line = line.strip() + if not line: + continue + if line.startswith("data: "): + data = line[6:] + if data.strip() == "[DONE]": + break + try: + obj = json.loads(data) + choices = obj.get("choices", []) + if choices: + delta = choices[0].get("delta", {}) + content = delta.get("content", "") + if content: + full_response += content + if _stream_consumer: + _stream_consumer.on_delta(content) + except json.JSONDecodeError: + pass + + except asyncio.CancelledError: + raise + except Exception as e: + logger.error("Proxy connection error to %s: %s", proxy_url, e) + if not full_response: + return { + "final_response": f"⚠️ Proxy connection error: {e}", + "messages": [], + "api_calls": 0, + "tools": [], + } + # Partial response — return what we got + finally: + # Finalize stream consumer + if _stream_consumer: + _stream_consumer.finish() + if stream_task: + try: + await asyncio.wait_for(stream_task, timeout=5.0) + except (asyncio.TimeoutError, asyncio.CancelledError): + stream_task.cancel() + + _elapsed = time.time() - _start + if not _run_still_current(): + logger.info( + "Discarding stale proxy result for %s — generation %d is no longer current", + session_key[:20] if session_key else "?", + run_generation or 0, + ) + return { + "final_response": "", + "messages": [], + "api_calls": 0, + "tools": [], + "history_offset": len(history), + "session_id": session_id, + "response_previewed": False, + } + logger.info( + "proxy response: url=%s session=%s time=%.1fs response=%d chars", + proxy_url, (session_id or "")[:20], _elapsed, len(full_response), + ) + + return { + "final_response": full_response or "(No response from remote agent)", + "messages": [ + {"role": "user", "content": message}, + {"role": "assistant", "content": full_response}, + ], + "api_calls": 1, + "tools": [], + "history_offset": len(history), + "session_id": session_id, + "response_previewed": _stream_consumer is not None and bool(full_response), + } + + # ------------------------------------------------------------------ + async def _run_agent( self, message: str, @@ -7407,8 +9301,10 @@ async def _run_agent( source: SessionSource, session_id: str, session_key: str = None, + run_generation: Optional[int] = None, _interrupt_depth: int = 0, event_message_id: Optional[str] = None, + channel_prompt: Optional[str] = None, ) -> Dict[str, Any]: """ Run the agent with the given message and context. @@ -7422,8 +9318,26 @@ async def _run_agent( This is run in a thread pool to not block the event loop. Supports interruption via new messages. """ + # ---- Proxy mode: delegate to remote API server ---- + if self._get_proxy_url(): + return await self._run_agent_via_proxy( + message=message, + context_prompt=context_prompt, + history=history, + source=source, + session_id=session_id, + session_key=session_key, + run_generation=run_generation, + event_message_id=event_message_id, + ) + from run_agent import AIAgent import queue + + def _run_still_current() -> bool: + if run_generation is None or not session_key: + return True + return self._is_session_run_current(session_key, run_generation) user_config = _load_gateway_config() platform_key = _platform_config_key(source.platform) @@ -7478,7 +9392,7 @@ async def _run_agent( def progress_callback(event_type: str, tool_name: str = None, preview: str = None, args: dict = None, **kwargs): """Callback invoked by agent on tool lifecycle events.""" - if not progress_queue: + if not progress_queue or not _run_still_current(): return # Only act on tool.started events (ignore tool.completed, reasoning.available, etc.) @@ -7499,8 +9413,7 @@ def progress_callback(event_type: str, tool_name: str = None, preview: str = Non if args: from agent.display import get_tool_preview_max_len _pl = get_tool_preview_max_len() - import json as _json - args_str = _json.dumps(args, ensure_ascii=False, default=str) + args_str = json.dumps(args, ensure_ascii=False, default=str) # When tool_preview_length is 0 (default), don't truncate # in verbose mode — the user explicitly asked for full # detail. Platform message-length limits handle the rest. @@ -7566,8 +9479,7 @@ async def send_progress_messages(): # Skip tool progress for platforms that don't support message # editing (e.g. iMessage/BlueBubbles) — each progress update # would become a separate message bubble, which is noisy. - from gateway.platforms.base import BasePlatformAdapter as _BaseAdapter - if type(adapter).edit_message is _BaseAdapter.edit_message: + if type(adapter).edit_message is BasePlatformAdapter.edit_message: while not progress_queue.empty(): try: progress_queue.get_nowait() @@ -7583,6 +9495,14 @@ async def send_progress_messages(): while True: try: + if not _run_still_current(): + while not progress_queue.empty(): + try: + progress_queue.get_nowait() + except Exception: + break + return + raw = progress_queue.get_nowait() # Handle dedup messages: update last line with repeat counter @@ -7608,6 +9528,9 @@ async def send_progress_messages(): await asyncio.sleep(_remaining) continue + if not _run_still_current(): + return + if can_edit and progress_msg_id is not None: # Try to edit the existing progress message full_text = "\n".join(progress_lines) @@ -7643,7 +9566,8 @@ async def send_progress_messages(): # Restore typing indicator await asyncio.sleep(0.3) - await adapter.send_typing(source.chat_id, metadata=_progress_metadata) + if _run_still_current(): + await adapter.send_typing(source.chat_id, metadata=_progress_metadata) except queue.Empty: await asyncio.sleep(0.3) @@ -7683,10 +9607,12 @@ async def send_progress_messages(): stream_consumer_holder = [None] # Mutable container for stream consumer # Bridge sync step_callback → async hooks.emit for agent:step events - _loop_for_step = asyncio.get_event_loop() + _loop_for_step = asyncio.get_running_loop() _hooks_ref = self.hooks def _step_callback_sync(iteration: int, prev_tools: list) -> None: + if not _run_still_current(): + return try: # prev_tools may be list[str] or list[dict] with "name"/"result" # keys. Normalise to keep "tool_names" backward-compatible for @@ -7717,7 +9643,7 @@ def _step_callback_sync(iteration: int, prev_tools: list) -> None: _status_thread_metadata = {"thread_id": _progress_thread_id} if _progress_thread_id else None def _status_callback_sync(event_type: str, message: str) -> None: - if not _status_adapter: + if not _status_adapter or not _run_still_current(): return try: asyncio.run_coroutine_threadsafe( @@ -7751,8 +9677,12 @@ def run_sync(): # Platform.LOCAL ("local") maps to "cli"; others pass through as-is. platform_key = "cli" if source.platform == Platform.LOCAL else source.platform.value - # Combine platform context with user-configured ephemeral system prompt + # Combine platform context, per-channel context, and the user-configured + # ephemeral system prompt. combined_ephemeral = context_prompt or "" + event_channel_prompt = (channel_prompt or "").strip() + if event_channel_prompt: + combined_ephemeral = (combined_ephemeral + "\n\n" + event_channel_prompt).strip() if self._ephemeral_system_prompt: combined_ephemeral = (combined_ephemeral + "\n\n" + self._ephemeral_system_prompt).strip() @@ -7816,17 +9746,26 @@ def run_sync(): _adapter = self.adapters.get(source.platform) if _adapter: # Platforms that don't support editing sent messages - # (e.g. WeChat) must not show a cursor in intermediate - # sends — the cursor would be permanently visible because - # it can never be edited away. Use an empty cursor for - # such platforms so streaming still delivers the final - # response, just without the typing indicator. + # (e.g. QQ, WeChat) should skip streaming entirely — + # without edit support, the consumer sends a partial + # first message that can never be updated, resulting in + # duplicate messages (partial + final). _adapter_supports_edit = getattr(_adapter, "SUPPORTS_MESSAGE_EDITING", True) - _effective_cursor = _scfg.cursor if _adapter_supports_edit else "" + if not _adapter_supports_edit: + raise RuntimeError("skip streaming for non-editable platform") + _effective_cursor = _scfg.cursor + # Some Matrix clients render the streaming cursor + # as a visible tofu/white-box artifact. Keep + # streaming text on Matrix, but suppress the cursor. + _buffer_only = False + if source.platform == Platform.MATRIX: + _effective_cursor = "" + _buffer_only = True _consumer_cfg = StreamConsumerConfig( edit_interval=_scfg.edit_interval, buffer_threshold=_scfg.buffer_threshold, cursor=_effective_cursor, + buffer_only=_buffer_only, ) _stream_consumer = GatewayStreamConsumer( adapter=_adapter, @@ -7835,12 +9774,16 @@ def run_sync(): metadata={"thread_id": _progress_thread_id} if _progress_thread_id else None, ) if _want_stream_deltas: - _stream_delta_cb = _stream_consumer.on_delta + def _stream_delta_cb(text: str) -> None: + if _run_still_current(): + _stream_consumer.on_delta(text) stream_consumer_holder[0] = _stream_consumer except Exception as _sc_err: logger.debug("Could not set up stream consumer: %s", _sc_err) def _interim_assistant_cb(text: str, *, already_streamed: bool = False) -> None: + if not _run_still_current(): + return if _stream_consumer is not None: if already_streamed: _stream_consumer.on_segment_break() @@ -7880,6 +9823,19 @@ def _interim_assistant_cb(text: str, *, already_streamed: bool = False) -> None: cached = _cache.get(session_key) if cached and cached[1] == _sig: agent = cached[0] + # Refresh LRU order so the cap enforcement evicts + # truly-oldest entries, not the one we just used. + if hasattr(_cache, "move_to_end"): + try: + _cache.move_to_end(session_key) + except KeyError: + pass + # Reset activity timestamp so the inactivity timeout + # handler doesn't see stale idle time from the previous + # turn and immediately kill this agent. (#9051) + agent._last_activity_ts = time.time() + agent._last_activity_desc = "starting new turn (cached)" + agent._api_call_count = 0 logger.debug("Reusing cached agent for session %s", session_key) if agent is None: @@ -7905,12 +9861,19 @@ def _interim_assistant_cb(text: str, *, already_streamed: bool = False) -> None: session_id=session_id, platform=platform_key, user_id=source.user_id, + user_name=source.user_name, + chat_id=source.chat_id, + chat_name=source.chat_name, + chat_type=source.chat_type, + thread_id=source.thread_id, + gateway_session_key=session_key, session_db=self._session_db, fallback_model=self._fallback_model, ) if _cache_lock and _cache is not None: with _cache_lock: _cache[session_key] = (agent, _sig) + self._enforce_agent_cache_cap() logger.debug("Created new agent for session %s (sig=%s)", session_key, _sig) # Per-message state — callbacks and reasoning config change every @@ -7924,9 +9887,12 @@ def _interim_assistant_cb(text: str, *, already_streamed: bool = False) -> None: agent.service_tier = self._service_tier agent.request_overrides = turn_route.get("request_overrides") - # Background review delivery — send "💾 Memory updated" etc. to user - def _bg_review_send(message: str) -> None: - if not _status_adapter: + _bg_review_release = threading.Event() + _bg_review_pending: list[str] = [] + _bg_review_pending_lock = threading.Lock() + + def _deliver_bg_review_message(message: str) -> None: + if not _status_adapter or not _run_still_current(): return try: asyncio.run_coroutine_threadsafe( @@ -7940,7 +9906,39 @@ def _bg_review_send(message: str) -> None: except Exception as _e: logger.debug("background_review_callback error: %s", _e) + def _release_bg_review_messages() -> None: + _bg_review_release.set() + with _bg_review_pending_lock: + pending = list(_bg_review_pending) + _bg_review_pending.clear() + for queued in pending: + _deliver_bg_review_message(queued) + + # Background review delivery — send "💾 Memory updated" etc. to user + def _bg_review_send(message: str) -> None: + if not _status_adapter or not _run_still_current(): + return + if not _bg_review_release.is_set(): + with _bg_review_pending_lock: + if not _bg_review_release.is_set(): + _bg_review_pending.append(message) + return + _deliver_bg_review_message(message) + agent.background_review_callback = _bg_review_send + # Register the release hook on the adapter so base.py's finally + # block can fire it after delivering the main response. + if _status_adapter and session_key: + if getattr(type(_status_adapter), "register_post_delivery_callback", None) is not None: + _status_adapter.register_post_delivery_callback( + session_key, + _release_bg_review_messages, + generation=run_generation, + ) + else: + _pdc = getattr(_status_adapter, "_post_delivery_callbacks", None) + if _pdc is not None: + _pdc[session_key] = _release_bg_review_messages # Store agent reference for interrupt support agent_holder[0] = agent @@ -8049,7 +10047,7 @@ def _approval_notify_sync(approval_data: dict) -> None: # false positives from MagicMock auto-attribute creation in tests. if getattr(type(_status_adapter), "send_exec_approval", None) is not None: try: - asyncio.run_coroutine_threadsafe( + _approval_result = asyncio.run_coroutine_threadsafe( _status_adapter.send_exec_approval( chat_id=_status_chat_id, command=cmd, @@ -8059,7 +10057,12 @@ def _approval_notify_sync(approval_data: dict) -> None: ), _loop_for_step, ).result(timeout=15) - return + if _approval_result.success: + return + logger.warning( + "Button-based approval failed (send returned error), falling back to text: %s", + _approval_result.error, + ) except Exception as _e: logger.warning( "Button-based approval failed, falling back to text: %s", _e @@ -8092,6 +10095,54 @@ def _approval_notify_sync(approval_data: dict) -> None: if _msn: message = _msn + "\n\n" + message + # Auto-continue: if the loaded history ends with a tool result, + # the previous agent turn was interrupted mid-work (gateway + # restart, crash, SIGTERM). Prepend a system note so the model + # finishes processing the pending tool results before addressing + # the user's new message. (#4493) + # + # Session-level resume_pending (set on drain-timeout shutdown) + # escalates the wording — the transcript's last role may be + # anything (tool, assistant with unfinished work, etc.), so we + # give a stronger, reason-aware instruction that subsumes the + # tool-tail case. + _resume_entry = None + if session_key: + try: + _resume_entry = self.session_store._entries.get(session_key) + except Exception: + _resume_entry = None + _is_resume_pending = bool( + _resume_entry is not None and getattr(_resume_entry, "resume_pending", False) + ) + + if _is_resume_pending: + _reason = getattr(_resume_entry, "resume_reason", None) or "restart_timeout" + _reason_phrase = ( + "a gateway restart" + if _reason == "restart_timeout" + else "a gateway shutdown" + if _reason == "shutdown_timeout" + else "a gateway interruption" + ) + message = ( + f"[System note: Your previous turn in this session was interrupted " + f"by {_reason_phrase}. The conversation history below is intact. " + f"If it contains unfinished tool result(s), process them first and " + f"summarize what was accomplished, then address the user's new " + f"message below.]\n\n" + + message + ) + elif agent_history and agent_history[-1].get("role") == "tool": + message = ( + "[System note: Your previous turn was interrupted before you could " + "process the last tool result(s). The conversation history contains " + "tool outputs you haven't responded to yet. Please finish processing " + "those results and summarize what was accomplished, then address the " + "user's new message below.]\n\n" + + message + ) + _approval_session_key = session_key or "" _approval_session_token = set_current_session_key(_approval_session_key) register_gateway_notify(_approval_session_key, _approval_notify_sync) @@ -8121,11 +10172,13 @@ def _approval_notify_sync(approval_data: dict) -> None: _resolved_model = getattr(_agent, "model", None) if _agent else None if not final_response: - error_msg = f"⚠️ {result['error']}" if result.get("error") else "(No response generated)" + error_msg = f"⚠️ {result['error']}" if result.get("error") else "" return { "final_response": error_msg, "messages": result.get("messages", []), "api_calls": result.get("api_calls", 0), + "failed": result.get("failed", False), + "compression_exhausted": result.get("compression_exhausted", False), "tools": tools_holder[0] or [], "history_offset": len(agent_history), "last_prompt_tokens": _last_prompt_toks, @@ -8250,10 +10303,24 @@ async def track_agent(): # Wait for agent to be created while agent_holder[0] is None: await asyncio.sleep(0.05) - if session_key: - self._running_agents[session_key] = agent_holder[0] - if self._draining: - self._update_runtime_status("draining") + if not session_key: + return + # Only promote the sentinel to the real agent if this run is still + # current. If /stop or /new bumped the generation while we were + # spinning up, leave the newer run's slot alone — we'll be + # discarded by the stale-result check in _handle_message_with_agent. + if run_generation is not None and not self._is_session_run_current( + session_key, run_generation + ): + logger.info( + "Skipping stale agent promotion for %s — generation %s is no longer current", + (session_key or "")[:20], + run_generation, + ) + return + self._running_agents[session_key] = agent_holder[0] + if self._draining: + self._update_runtime_status("draining") tracking_task = asyncio.create_task(track_agent()) @@ -8308,9 +10375,9 @@ async def monitor_for_interrupt(): # Periodic "still working" notifications for long-running tasks. # Fires every N seconds so the user knows the agent hasn't died. # Config: agent.gateway_notify_interval in config.yaml, or - # HERMES_AGENT_NOTIFY_INTERVAL env var. Default 600s (10 min). + # HERMES_AGENT_NOTIFY_INTERVAL env var. Default 180s (3 min). # 0 = disable notifications. - _NOTIFY_INTERVAL_RAW = float(os.getenv("HERMES_AGENT_NOTIFY_INTERVAL", 600)) + _NOTIFY_INTERVAL_RAW = float(os.getenv("HERMES_AGENT_NOTIFY_INTERVAL", 180)) _NOTIFY_INTERVAL = _NOTIFY_INTERVAL_RAW if _NOTIFY_INTERVAL_RAW > 0 else None _notify_start = time.time() @@ -8363,9 +10430,8 @@ async def _notify_long_running(): _agent_warning_raw = float(os.getenv("HERMES_AGENT_TIMEOUT_WARNING", 900)) _agent_warning = _agent_warning_raw if _agent_warning_raw > 0 else None _warning_fired = False - loop = asyncio.get_event_loop() _executor_task = asyncio.ensure_future( - loop.run_in_executor(None, run_sync) + self._run_in_executor_with_context(run_sync) ) _inactivity_timeout = False @@ -8488,7 +10554,7 @@ async def _notify_long_running(): # Interrupt the agent if it's still running so the thread # pool worker is freed. if _timed_out_agent and hasattr(_timed_out_agent, "interrupt"): - _timed_out_agent.interrupt("Execution timed out (inactivity)") + _timed_out_agent.interrupt(_INTERRUPT_REASON_TIMEOUT) _timeout_mins = int(_agent_timeout // 60) or 1 @@ -8553,11 +10619,29 @@ async def _notify_long_running(): if result and adapter and session_key: pending_event = _dequeue_pending_event(adapter, session_key) if result.get("interrupted") and not pending_event and result.get("interrupt_message"): - pending = result.get("interrupt_message") + interrupt_message = result.get("interrupt_message") + if _is_control_interrupt_message(interrupt_message): + logger.info( + "Ignoring control interrupt message for session %s: %s", + session_key[:20] if session_key else "?", + interrupt_message, + ) + else: + pending = interrupt_message elif pending_event: pending = pending_event.text or _build_media_placeholder(pending_event) logger.debug("Processing queued message after agent completion: '%s...'", pending[:40]) + # Leftover /steer: if a steer arrived after the last tool batch + # (e.g. during the final API call), the agent couldn't inject it + # and returned it in result["pending_steer"]. Deliver it as the + # next user turn so it isn't silently dropped. + if result and not pending and not pending_event: + _leftover_steer = result.get("pending_steer") + if _leftover_steer: + pending = _leftover_steer + logger.debug("Delivering leftover /steer as next turn: '%s...'", pending[:40]) + # Safety net: if the pending text is a slash command (e.g. "/stop", # "/new"), discard it — commands should never be passed to the agent # as user input. The primary fix is in base.py (commands bypass the @@ -8630,20 +10714,18 @@ async def _notify_long_running(): pass except Exception as e: logger.debug("Stream consumer wait before queued message failed: %s", e) - _response_previewed = bool(result.get("response_previewed")) + _previewed = bool(result.get("response_previewed")) _already_streamed = bool( - _sc - and ( - getattr(_sc, "final_response_sent", False) - or ( - _response_previewed - and getattr(_sc, "already_sent", False) - ) - ) + (_sc and getattr(_sc, "final_response_sent", False)) + or _previewed ) first_response = result.get("final_response", "") if first_response and not _already_streamed: try: + logger.info( + "Queued follow-up for session %s: final stream delivery not confirmed; sending first response before continuing.", + session_key[:20] if session_key else "?", + ) await adapter.send( source.chat_id, first_response, @@ -8651,6 +10733,32 @@ async def _notify_long_running(): ) except Exception as e: logger.warning("Failed to send first response before queued message: %s", e) + elif first_response: + logger.info( + "Queued follow-up for session %s: skipping resend because final streamed delivery was confirmed.", + session_key[:20] if session_key else "?", + ) + # Release deferred bg-review notifications now that the + # first response has been delivered. Pop from the + # adapter's callback dict (prevents double-fire in + # base.py's finally block) and call it. + if getattr(type(adapter), "pop_post_delivery_callback", None) is not None: + _bg_cb = adapter.pop_post_delivery_callback( + session_key, + generation=run_generation, + ) + if callable(_bg_cb): + try: + _bg_cb() + except Exception: + pass + elif adapter and hasattr(adapter, "_post_delivery_callbacks"): + _bg_cb = adapter._post_delivery_callbacks.pop(session_key, None) + if callable(_bg_cb): + try: + _bg_cb() + except Exception: + pass # else: interrupted — discard the interrupted response ("Operation # interrupted." is just noise; the user already knows they sent a # new message). @@ -8659,6 +10767,7 @@ async def _notify_long_running(): next_source = source next_message = pending next_message_id = None + next_channel_prompt = None if pending_event is not None: next_source = getattr(pending_event, "source", None) or source next_message = await self._prepare_inbound_message_text( @@ -8669,6 +10778,20 @@ async def _notify_long_running(): if next_message is None: return result next_message_id = getattr(pending_event, "message_id", None) + next_channel_prompt = getattr(pending_event, "channel_prompt", None) + + # Restart typing indicator so the user sees activity while + # the follow-up turn runs. The outer _process_message_background + # typing task is still alive but may be stale. + _followup_adapter = self.adapters.get(source.platform) + if _followup_adapter: + try: + await _followup_adapter.send_typing( + source.chat_id, + metadata=_status_thread_metadata, + ) + except Exception: + pass return await self._run_agent( message=next_message, @@ -8677,8 +10800,10 @@ async def _notify_long_running(): source=next_source, session_id=session_id, session_key=session_key, + run_generation=run_generation, _interrupt_depth=_interrupt_depth + 1, event_message_id=next_message_id, + channel_prompt=next_channel_prompt, ) finally: # Stop progress sender, interrupt monitor, and notification task @@ -8700,10 +10825,15 @@ async def _notify_long_running(): # Clean up tracking tracking_task.cancel() - if session_key and session_key in self._running_agents: - del self._running_agents[session_key] if session_key: - self._running_agents_ts.pop(session_key, None) + # Only release the slot if this run's generation still owns + # it. A /stop or /new that bumped the generation while we + # were unwinding has already installed its own state; this + # guard prevents an old run from clobbering it on the way + # out. + self._release_running_agent_state( + session_key, run_generation=run_generation + ) if self._draining: self._update_runtime_status("draining") @@ -8720,16 +10850,31 @@ async def _notify_long_running(): # BUT: never suppress delivery when the agent failed — the error # message is new content the user hasn't seen, and it must reach # them even if streaming had sent earlier partial output. + # + # Also never suppress when the final response is "(empty)" — this + # means the model failed to produce content after tool calls (common + # with mimo-v2-pro, GLM-5, etc.). The stream consumer may have + # sent intermediate text ("Let me search for that…") alongside the + # tool call, setting already_sent=True, but that text is NOT the + # final answer. Suppressing delivery here leaves the user staring + # at silence. (#10xxx — "agent stops after web search") _sc = stream_consumer_holder[0] - if _sc and isinstance(response, dict) and not response.get("failed"): - _response_previewed = bool(response.get("response_previewed")) - if ( - getattr(_sc, "final_response_sent", False) - or ( - _response_previewed - and getattr(_sc, "already_sent", False) + if isinstance(response, dict) and not response.get("failed"): + _final = response.get("final_response") or "" + _is_empty_sentinel = not _final or _final == "(empty)" + _streamed = bool( + _sc and getattr(_sc, "final_response_sent", False) + ) + # response_previewed means the interim_assistant_callback already + # sent the final text via the adapter (non-streaming path). + _previewed = bool(response.get("response_previewed")) + if not _is_empty_sentinel and (_streamed or _previewed): + logger.info( + "Suppressing normal final send for session %s: final delivery already confirmed (streamed=%s previewed=%s).", + session_key[:20] if session_key else "?", + _streamed, + _previewed, ) - ): response["already_sent"] = True return response @@ -8808,15 +10953,32 @@ async def start_gateway(config: Optional[GatewayConfig] = None, replace: bool = # The PID file is scoped to HERMES_HOME, so future multi-profile # setups (each profile using a distinct HERMES_HOME) will naturally # allow concurrent instances without tripping this guard. - import time as _time - from gateway.status import get_running_pid, remove_pid_file, terminate_pid + from gateway.status import ( + acquire_gateway_runtime_lock, + get_running_pid, + get_process_start_time, + release_gateway_runtime_lock, + remove_pid_file, + terminate_pid, + ) existing_pid = get_running_pid() if existing_pid is not None and existing_pid != os.getpid(): if replace: + existing_start_time = get_process_start_time(existing_pid) logger.info( "Replacing existing gateway instance (PID %d) with --replace.", existing_pid, ) + # Record a takeover marker so the target's shutdown handler + # recognises its SIGTERM as a planned takeover and exits 0 + # (rather than exit 1, which would trigger systemd's + # Restart=on-failure and start a flap loop against us). + # Best-effort — proceed even if the write fails. + try: + from gateway.status import write_takeover_marker + write_takeover_marker(existing_pid) + except Exception as e: + logger.debug("Could not write takeover marker: %s", e) try: terminate_pid(existing_pid, force=False) except ProcessLookupError: @@ -8826,12 +10988,19 @@ async def start_gateway(config: Optional[GatewayConfig] = None, replace: bool = "Permission denied killing PID %d. Cannot replace.", existing_pid, ) + # Marker is scoped to a specific target; clean it up on + # give-up so it doesn't grief an unrelated future shutdown. + try: + from gateway.status import clear_takeover_marker + clear_takeover_marker() + except Exception: + pass return False # Wait up to 10 seconds for the old process to exit for _ in range(20): try: os.kill(existing_pid, 0) - _time.sleep(0.5) + time.sleep(0.5) except (ProcessLookupError, PermissionError): break # Process is gone else: @@ -8842,16 +11011,32 @@ async def start_gateway(config: Optional[GatewayConfig] = None, replace: bool = ) try: terminate_pid(existing_pid, force=True) - _time.sleep(0.5) + time.sleep(0.5) except (ProcessLookupError, PermissionError, OSError): pass remove_pid_file() + # remove_pid_file() is a no-op when the PID doesn't match. + # Force-unlink to cover the old-process-crashed case. + try: + (get_hermes_home() / "gateway.pid").unlink(missing_ok=True) + except Exception: + pass + # Clean up any takeover marker the old process didn't consume + # (e.g. SIGKILL'd before its shutdown handler could read it). + try: + from gateway.status import clear_takeover_marker + clear_takeover_marker() + except Exception: + pass # Also release all scoped locks left by the old process. # Stopped (Ctrl+Z) processes don't release locks on exit, # leaving stale lock files that block the new gateway from starting. try: from gateway.status import release_all_scoped_locks - _released = release_all_scoped_locks() + _released = release_all_scoped_locks( + owner_pid=existing_pid, + owner_start_time=existing_start_time, + ) if _released: logger.info("Released %d stale scoped lock(s) from old gateway.", _released) except Exception: @@ -8903,14 +11088,66 @@ async def start_gateway(config: Optional[GatewayConfig] = None, replace: bool = runner = GatewayRunner(config) + # Track whether a signal initiated the shutdown (vs. internal request). + # When an unexpected SIGTERM kills the gateway, we exit non-zero so + # systemd's Restart=on-failure revives the process. systemctl stop + # is safe: systemd tracks stop-requested state independently of exit + # code, so Restart= never fires for a deliberate stop. + _signal_initiated_shutdown = False + # Set up signal handlers def shutdown_signal_handler(): + nonlocal _signal_initiated_shutdown + # Planned --replace takeover check: when a sibling gateway is + # taking over via --replace, it wrote a marker naming this PID + # before sending SIGTERM. If present, treat the signal as a + # planned shutdown and exit 0 so systemd's Restart=on-failure + # doesn't revive us (which would flap-fight the replacer when + # both services are enabled, e.g. hermes.service + hermes- + # gateway.service from pre-rename installs). + planned_takeover = False + try: + from gateway.status import consume_takeover_marker_for_self + planned_takeover = consume_takeover_marker_for_self() + except Exception as e: + logger.debug("Takeover marker check failed: %s", e) + + if planned_takeover: + logger.info( + "Received SIGTERM as a planned --replace takeover — exiting cleanly" + ) + else: + _signal_initiated_shutdown = True + logger.info("Received SIGTERM/SIGINT — initiating shutdown") + # Diagnostic: log all hermes-related processes so we can identify + # what triggered the signal (hermes update, hermes gateway restart, + # a stale detached subprocess, etc.). + try: + import subprocess as _sp + _ps = _sp.run( + ["ps", "aux"], + capture_output=True, text=True, timeout=3, + ) + _hermes_procs = [ + line for line in _ps.stdout.splitlines() + if ("hermes" in line.lower() or "gateway" in line.lower()) + and str(os.getpid()) not in line.split()[1:2] # exclude self + ] + if _hermes_procs: + logger.warning( + "Shutdown diagnostic — other hermes processes running:\n %s", + "\n ".join(_hermes_procs), + ) + else: + logger.info("Shutdown diagnostic — no other hermes processes found") + except Exception: + pass asyncio.create_task(runner.stop()) def restart_signal_handler(): runner.request_restart(detached=False, via_service=True) - loop = asyncio.get_event_loop() + loop = asyncio.get_running_loop() if threading.current_thread() is threading.main_thread(): for sig in (signal.SIGINT, signal.SIGTERM): try: @@ -8925,6 +11162,37 @@ def restart_signal_handler(): else: logger.info("Skipping signal handlers (not running in main thread).") + # Claim the PID file BEFORE bringing up any platform adapters. + # This closes the --replace race window: two concurrent `gateway run + # --replace` invocations both pass the termination-wait above, but + # only the winner of the O_CREAT|O_EXCL race below will ever open + # Telegram polling, Discord gateway sockets, etc. The loser exits + # cleanly before touching any external service. + import atexit + from gateway.status import write_pid_file, remove_pid_file, get_running_pid + _current_pid = get_running_pid() + if _current_pid is not None and _current_pid != os.getpid(): + logger.error( + "Another gateway instance (PID %d) started during our startup. " + "Exiting to avoid double-running.", _current_pid + ) + return False + if not acquire_gateway_runtime_lock(): + logger.error( + "Gateway runtime lock is already held by another instance. Exiting." + ) + return False + try: + write_pid_file() + except FileExistsError: + release_gateway_runtime_lock() + logger.error( + "PID file race lost to another gateway instance. Exiting." + ) + return False + atexit.register(remove_pid_file) + atexit.register(release_gateway_runtime_lock) + # Start the gateway success = await runner.start() if not success: @@ -8934,12 +11202,6 @@ def restart_signal_handler(): logger.error("Gateway exiting cleanly: %s", runner.exit_reason) return True - # Write PID file so CLI can detect gateway is running - import atexit - from gateway.status import write_pid_file, remove_pid_file - write_pid_file() - atexit.register(remove_pid_file) - # Start background cron ticker so scheduled jobs fire automatically. # Pass the event loop so cron delivery can use live adapters (E2EE support). cron_stop = threading.Event() @@ -8974,6 +11236,21 @@ def restart_signal_handler(): if runner.exit_code is not None: raise SystemExit(runner.exit_code) + # When a signal (SIGTERM/SIGINT) caused the shutdown and it wasn't a + # planned restart (/restart, /update, SIGUSR1), exit non-zero so + # systemd's Restart=on-failure revives the process. This covers: + # - hermes update killing the gateway mid-work + # - External kill commands + # - WSL2/container runtime sending unexpected signals + # systemctl stop is safe: systemd tracks "stop requested" state + # independently of exit code, so Restart= never fires for it. + if _signal_initiated_shutdown and not runner._restart_requested: + logger.info( + "Exiting with code 1 (signal-initiated shutdown without restart " + "request) so systemd Restart=on-failure can revive the gateway." + ) + return False # → sys.exit(1) in the caller + return True @@ -8989,9 +11266,9 @@ def main(): config = None if args.config: - import json + import yaml with open(args.config, encoding="utf-8") as f: - data = json.load(f) + data = yaml.safe_load(f) config = GatewayConfig.from_dict(data) # Run the gateway - exit with code 1 if no platforms connected, diff --git a/gateway/session.py b/gateway/session.py index 62beeffa8498..db90d3121727 100644 --- a/gateway/session.py +++ b/gateway/session.py @@ -12,7 +12,6 @@ import logging import os import json -import re import threading import uuid from pathlib import Path @@ -81,8 +80,9 @@ class SessionSource: user_name: Optional[str] = None thread_id: Optional[str] = None # For forum topics, Discord threads, etc. chat_topic: Optional[str] = None # Channel topic/description (Discord, Slack) - user_id_alt: Optional[str] = None # Signal UUID (alternative to phone number) + user_id_alt: Optional[str] = None # Platform-specific stable alt ID (Signal UUID, Feishu union_id) chat_id_alt: Optional[str] = None # Signal group internal ID + is_bot: bool = False # True when the message author is a bot/webhook (Discord) @property def description(self) -> str: @@ -152,6 +152,7 @@ class SessionContext: source: SessionSource connected_platforms: List[Platform] home_channels: Dict[Platform, HomeChannel] + shared_multi_user_session: bool = False # Session metadata session_key: str = "" @@ -166,6 +167,7 @@ def to_dict(self) -> Dict[str, Any]: "home_channels": { p.value: hc.to_dict() for p, hc in self.home_channels.items() }, + "shared_multi_user_session": self.shared_multi_user_session, "session_key": self.session_key, "session_id": self.session_id, "created_at": self.created_at.isoformat() if self.created_at else None, @@ -240,18 +242,16 @@ def build_session_context_prompt( lines.append(f"**Channel Topic:** {context.source.chat_topic}") # User identity. - # In shared thread sessions (non-DM with thread_id), multiple users - # contribute to the same conversation. Don't pin a single user name - # in the system prompt — it changes per-turn and would bust the prompt - # cache. Instead, note that this is a multi-user thread; individual - # sender names are prefixed on each user message by the gateway. - _is_shared_thread = ( - context.source.chat_type != "dm" - and context.source.thread_id - ) - if _is_shared_thread: + # In shared multi-user sessions (shared threads OR shared non-thread groups + # when group_sessions_per_user=False), multiple users contribute to the same + # conversation. Don't pin a single user name in the system prompt — it + # changes per-turn and would bust the prompt cache. Instead, note that + # this is a multi-user session; individual sender names are prefixed on + # each user message by the gateway. + if context.shared_multi_user_session: + session_label = "Multi-user thread" if context.source.thread_id else "Multi-user session" lines.append( - "**Session type:** Multi-user thread — messages are prefixed " + f"**Session type:** {session_label} — messages are prefixed " "with [sender name]. Multiple users may participate." ) elif context.source.user_name: @@ -302,6 +302,8 @@ def build_session_context_prompt( lines.append("") lines.append("**Delivery options for scheduled tasks:**") + from hermes_constants import display_hermes_home + # Origin delivery if context.source.platform == Platform.LOCAL: lines.append("- `\"origin\"` → Local output (saved to files)") @@ -310,9 +312,11 @@ def build_session_context_prompt( _hash_chat_id(context.source.chat_id) if redact_pii else context.source.chat_id ) lines.append(f"- `\"origin\"` → Back to this chat ({_origin_label})") - + # Local always available - lines.append("- `\"local\"` → Save to local files only (~/.hermes/cron/output/)") + lines.append( + f"- `\"local\"` → Save to local files only ({display_hermes_home()}/cron/output/)" + ) # Platform home channels for platform, home in context.home_channels.items(): @@ -373,7 +377,19 @@ class SessionEntry: # this session (create a new session_id) so the user starts fresh. # Set by /stop to break stuck-resume loops (#7536). suspended: bool = False - + + # When True the session was interrupted by a gateway restart/shutdown + # drain timeout, but recovery is still expected. Unlike ``suspended``, + # ``resume_pending`` preserves the existing session_id on next access — + # the user stays on the same transcript and the agent auto-continues + # from where it left off. Cleared after the next successful turn. + # Escalation to ``suspended`` is handled by the existing + # ``.restart_failure_counts`` stuck-loop counter (#7536), not by a + # parallel counter on this entry. + resume_pending: bool = False + resume_reason: Optional[str] = None # e.g. "restart_timeout" + last_resume_marked_at: Optional[datetime] = None + def to_dict(self) -> Dict[str, Any]: result = { "session_key": self.session_key, @@ -393,6 +409,13 @@ def to_dict(self) -> Dict[str, Any]: "cost_status": self.cost_status, "memory_flushed": self.memory_flushed, "suspended": self.suspended, + "resume_pending": self.resume_pending, + "resume_reason": self.resume_reason, + "last_resume_marked_at": ( + self.last_resume_marked_at.isoformat() + if self.last_resume_marked_at + else None + ), } if self.origin: result["origin"] = self.origin.to_dict() @@ -410,7 +433,15 @@ def from_dict(cls, data: Dict[str, Any]) -> "SessionEntry": platform = Platform(data["platform"]) except ValueError as e: logger.debug("Unknown platform value %r: %s", data["platform"], e) - + + last_resume_marked_at = None + _lrma = data.get("last_resume_marked_at") + if _lrma: + try: + last_resume_marked_at = datetime.fromisoformat(_lrma) + except (TypeError, ValueError): + last_resume_marked_at = None + return cls( session_key=data["session_key"], session_id=data["session_id"], @@ -430,9 +461,33 @@ def from_dict(cls, data: Dict[str, Any]) -> "SessionEntry": cost_status=data.get("cost_status", "unknown"), memory_flushed=data.get("memory_flushed", False), suspended=data.get("suspended", False), + resume_pending=data.get("resume_pending", False), + resume_reason=data.get("resume_reason"), + last_resume_marked_at=last_resume_marked_at, ) +def is_shared_multi_user_session( + source: SessionSource, + *, + group_sessions_per_user: bool = True, + thread_sessions_per_user: bool = False, +) -> bool: + """Return True when a non-DM session is shared across participants. + + Mirrors the isolation rules in :func:`build_session_key`: + - DMs are never shared. + - Threads are shared unless ``thread_sessions_per_user`` is True. + - Non-thread group/channel sessions are shared unless + ``group_sessions_per_user`` is True (default: True = isolated). + """ + if source.chat_type == "dm": + return False + if source.thread_id: + return not thread_sessions_per_user + return not group_sessions_per_user + + def build_session_key( source: SessionSource, group_sessions_per_user: bool = True, @@ -706,9 +761,23 @@ def get_or_create_session( entry = self._entries[session_key] # Auto-reset sessions marked as suspended (e.g. after /stop - # broke a stuck loop — #7536). + # broke a stuck loop — #7536). ``suspended`` is the hard + # forced-wipe signal and always wins over ``resume_pending``, + # so repeated interrupted restarts that escalate via the + # existing ``.restart_failure_counts`` stuck-loop counter + # still converge to a clean slate. if entry.suspended: reset_reason = "suspended" + elif entry.resume_pending: + # Restart-interrupted session: preserve the session_id + # and return the existing entry so the transcript + # reloads intact. ``resume_pending`` is cleared after + # the NEXT successful turn completes (not here), which + # means a re-interrupted retry keeps trying — the + # stuck-loop counter handles terminal escalation. + entry.updated_at = now + self._save() + return entry else: reset_reason = self._should_reset(entry, source) if not reset_reason: @@ -798,6 +867,112 @@ def suspend_session(self, session_key: str) -> bool: return True return False + def mark_resume_pending( + self, + session_key: str, + reason: str = "restart_timeout", + ) -> bool: + """Mark a session as resumable after a restart interruption. + + Unlike ``suspend_session()``, this preserves the existing + ``session_id`` and the transcript. The next call to + ``get_or_create_session()`` for this key returns the same entry + so the user auto-resumes on the same conversation lane. + + Returns True if the session existed and was marked. + """ + with self._lock: + self._ensure_loaded_locked() + if session_key in self._entries: + entry = self._entries[session_key] + # Never override an explicit ``suspended`` — that is a hard + # forced-wipe signal (from /stop or stuck-loop escalation). + if entry.suspended: + return False + entry.resume_pending = True + entry.resume_reason = reason + entry.last_resume_marked_at = _now() + self._save() + return True + return False + + def clear_resume_pending(self, session_key: str) -> bool: + """Clear the resume-pending flag after a successful resumed turn. + + Called from the gateway after ``run_conversation()`` returns a + final response for a session that had ``resume_pending=True``, + signalling that recovery succeeded. + + Returns True if a flag was cleared. + """ + with self._lock: + self._ensure_loaded_locked() + entry = self._entries.get(session_key) + if entry is None or not entry.resume_pending: + return False + entry.resume_pending = False + entry.resume_reason = None + entry.last_resume_marked_at = None + self._save() + return True + + def prune_old_entries(self, max_age_days: int) -> int: + """Drop SessionEntry records older than max_age_days. + + Pruning is based on ``updated_at`` (last activity), not ``created_at``. + A session that's been active within the window is kept regardless of + how old it is. Entries marked ``suspended`` are kept — the user + explicitly paused them for later resume. Entries held by an active + process (via has_active_processes_fn) are also kept so long-running + background work isn't orphaned. + + Pruning is functionally identical to a natural reset-policy expiry: + the transcript in SQLite stays, but the session_key → session_id + mapping is dropped and the user starts a fresh session on return. + + ``max_age_days <= 0`` disables pruning; returns 0 immediately. + Returns the number of entries removed. + """ + if max_age_days is None or max_age_days <= 0: + return 0 + from datetime import timedelta + + cutoff = _now() - timedelta(days=max_age_days) + removed_keys: list[str] = [] + + with self._lock: + self._ensure_loaded_locked() + for key, entry in list(self._entries.items()): + if entry.suspended: + continue + # Never prune sessions with an active background process + # attached — the user may still be waiting on output. + # The callback is keyed by session_key (see process_registry. + # has_active_for_session); passing session_id here used to + # never match, so active sessions got pruned anyway. + if self._has_active_processes_fn is not None: + try: + if self._has_active_processes_fn(entry.session_key): + continue + except Exception as exc: + logger.debug( + "has_active_processes_fn raised during prune for %s: %s", + entry.session_key, exc, + ) + if entry.updated_at < cutoff: + removed_keys.append(key) + for key in removed_keys: + self._entries.pop(key, None) + if removed_keys: + self._save() + + if removed_keys: + logger.info( + "SessionStore pruned %d entries older than %d days", + len(removed_keys), max_age_days, + ) + return len(removed_keys) + def suspend_recently_active(self, max_age_seconds: int = 120) -> int: """Mark recently-active sessions as suspended. @@ -806,6 +981,12 @@ def suspend_recently_active(self, max_age_seconds: int = 120) -> int: (#7536). Only suspends sessions updated within *max_age_seconds* to avoid resetting long-idle sessions that are harmless to resume. Returns the number of sessions that were suspended. + + Entries flagged ``resume_pending=True`` are skipped — those were + marked intentionally by the drain-timeout path as recoverable. + Terminal escalation for genuinely stuck ``resume_pending`` sessions + is handled by the existing ``.restart_failure_counts`` stuck-loop + counter, which runs after this method on startup. """ from datetime import timedelta @@ -814,6 +995,8 @@ def suspend_recently_active(self, max_age_seconds: int = 120) -> int: with self._lock: self._ensure_loaded_locked() for entry in self._entries.values(): + if entry.resume_pending: + continue if not entry.suspended and entry.updated_at >= cutoff: entry.suspended = True count += 1 @@ -964,6 +1147,10 @@ def append_to_transcript(self, session_id: str, message: Dict[str, Any], skip_db tool_name=message.get("tool_name"), tool_calls=message.get("tool_calls"), tool_call_id=message.get("tool_call_id"), + reasoning=message.get("reasoning") if message.get("role") == "assistant" else None, + reasoning_content=message.get("reasoning_content") if message.get("role") == "assistant" else None, + reasoning_details=message.get("reasoning_details") if message.get("role") == "assistant" else None, + codex_reasoning_items=message.get("codex_reasoning_items") if message.get("role") == "assistant" else None, ) except Exception as e: logger.debug("Session DB operation failed: %s", e) @@ -993,6 +1180,7 @@ def rewrite_transcript(self, session_id: str, messages: List[Dict[str, Any]]) -> tool_calls=msg.get("tool_calls"), tool_call_id=msg.get("tool_call_id"), reasoning=msg.get("reasoning") if role == "assistant" else None, + reasoning_content=msg.get("reasoning_content") if role == "assistant" else None, reasoning_details=msg.get("reasoning_details") if role == "assistant" else None, codex_reasoning_items=msg.get("codex_reasoning_items") if role == "assistant" else None, ) @@ -1076,6 +1264,11 @@ def build_session_context( source=source, connected_platforms=connected, home_channels=home_channels, + shared_multi_user_session=is_shared_multi_user_session( + source, + group_sessions_per_user=getattr(config, "group_sessions_per_user", True), + thread_sessions_per_user=getattr(config, "thread_sessions_per_user", False), + ), ) if session_entry: diff --git a/gateway/session_context.py b/gateway/session_context.py index b9fdcdfaf771..9dc051e3a2c4 100644 --- a/gateway/session_context.py +++ b/gateway/session_context.py @@ -37,18 +37,30 @@ """ from contextvars import ContextVar +from typing import Any + +# Sentinel to distinguish "never set in this context" from "explicitly set to empty". +# When a contextvar holds _UNSET, we fall back to os.environ (CLI/cron compat). +# When it holds "" (after clear_session_vars resets it), we return "" — no fallback. +_UNSET: Any = object() # --------------------------------------------------------------------------- # Per-task session variables # --------------------------------------------------------------------------- -_SESSION_PLATFORM: ContextVar[str] = ContextVar("HERMES_SESSION_PLATFORM", default="") -_SESSION_CHAT_ID: ContextVar[str] = ContextVar("HERMES_SESSION_CHAT_ID", default="") -_SESSION_CHAT_NAME: ContextVar[str] = ContextVar("HERMES_SESSION_CHAT_NAME", default="") -_SESSION_THREAD_ID: ContextVar[str] = ContextVar("HERMES_SESSION_THREAD_ID", default="") -_SESSION_USER_ID: ContextVar[str] = ContextVar("HERMES_SESSION_USER_ID", default="") -_SESSION_USER_NAME: ContextVar[str] = ContextVar("HERMES_SESSION_USER_NAME", default="") -_SESSION_KEY: ContextVar[str] = ContextVar("HERMES_SESSION_KEY", default="") +_SESSION_PLATFORM: ContextVar = ContextVar("HERMES_SESSION_PLATFORM", default=_UNSET) +_SESSION_CHAT_ID: ContextVar = ContextVar("HERMES_SESSION_CHAT_ID", default=_UNSET) +_SESSION_CHAT_NAME: ContextVar = ContextVar("HERMES_SESSION_CHAT_NAME", default=_UNSET) +_SESSION_THREAD_ID: ContextVar = ContextVar("HERMES_SESSION_THREAD_ID", default=_UNSET) +_SESSION_USER_ID: ContextVar = ContextVar("HERMES_SESSION_USER_ID", default=_UNSET) +_SESSION_USER_NAME: ContextVar = ContextVar("HERMES_SESSION_USER_NAME", default=_UNSET) +_SESSION_KEY: ContextVar = ContextVar("HERMES_SESSION_KEY", default=_UNSET) + +# Cron auto-delivery vars — set per-job in run_job() so concurrent jobs +# don't clobber each other's delivery targets. +_CRON_AUTO_DELIVER_PLATFORM: ContextVar = ContextVar("HERMES_CRON_AUTO_DELIVER_PLATFORM", default=_UNSET) +_CRON_AUTO_DELIVER_CHAT_ID: ContextVar = ContextVar("HERMES_CRON_AUTO_DELIVER_CHAT_ID", default=_UNSET) +_CRON_AUTO_DELIVER_THREAD_ID: ContextVar = ContextVar("HERMES_CRON_AUTO_DELIVER_THREAD_ID", default=_UNSET) _VAR_MAP = { "HERMES_SESSION_PLATFORM": _SESSION_PLATFORM, @@ -58,6 +70,9 @@ "HERMES_SESSION_USER_ID": _SESSION_USER_ID, "HERMES_SESSION_USER_NAME": _SESSION_USER_NAME, "HERMES_SESSION_KEY": _SESSION_KEY, + "HERMES_CRON_AUTO_DELIVER_PLATFORM": _CRON_AUTO_DELIVER_PLATFORM, + "HERMES_CRON_AUTO_DELIVER_CHAT_ID": _CRON_AUTO_DELIVER_CHAT_ID, + "HERMES_CRON_AUTO_DELIVER_THREAD_ID": _CRON_AUTO_DELIVER_THREAD_ID, } @@ -91,10 +106,17 @@ def set_session_vars( def clear_session_vars(tokens: list) -> None: - """Restore session context variables to their pre-handler values.""" - if not tokens: - return - vars_in_order = [ + """Mark session context variables as explicitly cleared. + + Sets all variables to ``""`` so that ``get_session_env`` returns an empty + string instead of falling back to (potentially stale) ``os.environ`` + values. The *tokens* argument is accepted for API compatibility with + callers that saved the return value of ``set_session_vars``, but the + actual clearing uses ``var.set("")`` rather than ``var.reset(token)`` + to ensure the "explicitly cleared" state is distinguishable from + "never set" (which holds the ``_UNSET`` sentinel). + """ + for var in ( _SESSION_PLATFORM, _SESSION_CHAT_ID, _SESSION_CHAT_NAME, @@ -102,9 +124,8 @@ def clear_session_vars(tokens: list) -> None: _SESSION_USER_ID, _SESSION_USER_NAME, _SESSION_KEY, - ] - for var, token in zip(vars_in_order, tokens): - var.reset(token) + ): + var.set("") def get_session_env(name: str, default: str = "") -> str: @@ -113,8 +134,13 @@ def get_session_env(name: str, default: str = "") -> str: Drop-in replacement for ``os.getenv("HERMES_SESSION_*", default)``. Resolution order: - 1. Context variable (set by the gateway for concurrency-safe access) - 2. ``os.environ`` (used by CLI, cron scheduler, and tests) + 1. Context variable (set by the gateway for concurrency-safe access). + If the variable was explicitly set (even to ``""``) via + ``set_session_vars`` or ``clear_session_vars``, that value is + returned — **no fallback to os.environ**. + 2. ``os.environ`` (only when the context variable was never set in + this context — i.e. CLI, cron scheduler, and test processes that + don't use ``set_session_vars`` at all). 3. *default* """ import os @@ -122,7 +148,7 @@ def get_session_env(name: str, default: str = "") -> str: var = _VAR_MAP.get(name) if var is not None: value = var.get() - if value: + if value is not _UNSET: return value # Fall back to os.environ for CLI, cron, and test compatibility return os.getenv(name, default) diff --git a/gateway/status.py b/gateway/status.py index a801cfe5b811..7f7df182f57e 100644 --- a/gateway/status.py +++ b/gateway/status.py @@ -22,11 +22,18 @@ from hermes_constants import get_hermes_home from typing import Any, Optional +if sys.platform == "win32": + import msvcrt +else: + import fcntl + _GATEWAY_KIND = "hermes-gateway" _RUNTIME_STATUS_FILE = "gateway_state.json" _LOCKS_DIRNAME = "gateway-locks" _IS_WINDOWS = sys.platform == "win32" _UNSET = object() +_GATEWAY_LOCK_FILENAME = "gateway.lock" +_gateway_lock_handle = None def _get_pid_path() -> Path: @@ -35,6 +42,14 @@ def _get_pid_path() -> Path: return home / "gateway.pid" +def _get_gateway_lock_path(pid_path: Optional[Path] = None) -> Path: + """Return the path to the runtime gateway lock file.""" + if pid_path is not None: + return pid_path.with_name(_GATEWAY_LOCK_FILENAME) + home = get_hermes_home() + return home / _GATEWAY_LOCK_FILENAME + + def _get_runtime_status_path() -> Path: """Return the persisted runtime health/status file path.""" return _get_pid_path().with_name(_RUNTIME_STATUS_FILE) @@ -98,6 +113,11 @@ def _get_process_start_time(pid: int) -> Optional[int]: return None +def get_process_start_time(pid: int) -> Optional[int]: + """Public wrapper for retrieving a process start time when available.""" + return _get_process_start_time(pid) + + def _read_process_cmdline(pid: int) -> Optional[str]: """Return the process command line as a space-separated string.""" cmdline_path = Path(f"/proc/{pid}/cmdline") @@ -121,6 +141,7 @@ def _looks_like_gateway_process(pid: int) -> bool: "hermes_cli.main gateway", "hermes_cli/main.py gateway", "hermes gateway", + "hermes-gateway", "gateway/run.py", ) return any(pattern in cmdline for pattern in patterns) @@ -188,8 +209,8 @@ def _write_json_file(path: Path, payload: dict[str, Any]) -> None: path.write_text(json.dumps(payload)) -def _read_pid_record() -> Optional[dict]: - pid_path = _get_pid_path() +def _read_pid_record(pid_path: Optional[Path] = None) -> Optional[dict]: + pid_path = pid_path or _get_pid_path() if not pid_path.exists(): return None @@ -212,9 +233,160 @@ def _read_pid_record() -> Optional[dict]: return None +def _read_gateway_lock_record(lock_path: Optional[Path] = None) -> Optional[dict[str, Any]]: + return _read_pid_record(lock_path or _get_gateway_lock_path()) + + +def _pid_from_record(record: Optional[dict[str, Any]]) -> Optional[int]: + if not record: + return None + try: + return int(record["pid"]) + except (KeyError, TypeError, ValueError): + return None + + +def _cleanup_invalid_pid_path(pid_path: Path, *, cleanup_stale: bool) -> None: + """Delete a stale gateway PID file (and its sibling lock metadata). + + Called from ``get_running_pid()`` after the runtime lock has already been + confirmed inactive, so the on-disk metadata is known to belong to a dead + process. Unlike ``remove_pid_file()`` (which defensively refuses to delete + a PID file whose ``pid`` field differs from ``os.getpid()`` to protect + ``--replace`` handoffs), this path force-unlinks both files so the next + startup sees a clean slate. + """ + if not cleanup_stale: + return + try: + pid_path.unlink(missing_ok=True) + except Exception: + pass + try: + _get_gateway_lock_path(pid_path).unlink(missing_ok=True) + except Exception: + pass + + +def _write_gateway_lock_record(handle) -> None: + handle.seek(0) + handle.truncate() + json.dump(_build_pid_record(), handle) + handle.flush() + try: + os.fsync(handle.fileno()) + except OSError: + pass + + +def _try_acquire_file_lock(handle) -> bool: + try: + if _IS_WINDOWS: + handle.seek(0, os.SEEK_END) + if handle.tell() == 0: + handle.write("\n") + handle.flush() + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) + else: + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + return True + except (BlockingIOError, OSError): + return False + + +def _release_file_lock(handle) -> None: + try: + if _IS_WINDOWS: + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1) + else: + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + except OSError: + pass + + +def acquire_gateway_runtime_lock() -> bool: + """Claim the cross-process runtime lock for the gateway. + + Unlike the PID file, the lock is owned by the live process itself. If the + process dies abruptly, the OS releases the lock automatically. + """ + global _gateway_lock_handle + if _gateway_lock_handle is not None: + return True + + path = _get_gateway_lock_path() + path.parent.mkdir(parents=True, exist_ok=True) + handle = open(path, "a+", encoding="utf-8") + if not _try_acquire_file_lock(handle): + handle.close() + return False + _write_gateway_lock_record(handle) + _gateway_lock_handle = handle + return True + + +def release_gateway_runtime_lock() -> None: + """Release the gateway runtime lock when owned by this process.""" + global _gateway_lock_handle + handle = _gateway_lock_handle + if handle is None: + return + _gateway_lock_handle = None + _release_file_lock(handle) + try: + handle.close() + except OSError: + pass + + +def is_gateway_runtime_lock_active(lock_path: Optional[Path] = None) -> bool: + """Return True when some process currently owns the gateway runtime lock.""" + global _gateway_lock_handle + resolved_lock_path = lock_path or _get_gateway_lock_path() + if _gateway_lock_handle is not None and resolved_lock_path == _get_gateway_lock_path(): + return True + + if not resolved_lock_path.exists(): + return False + + handle = open(resolved_lock_path, "a+", encoding="utf-8") + try: + if _try_acquire_file_lock(handle): + _release_file_lock(handle) + return False + return True + finally: + try: + handle.close() + except OSError: + pass + + def write_pid_file() -> None: - """Write the current process PID and metadata to the gateway PID file.""" - _write_json_file(_get_pid_path(), _build_pid_record()) + """Write the current process PID and metadata to the gateway PID file. + + Uses atomic O_CREAT | O_EXCL creation so that concurrent --replace + invocations race: exactly one process wins and the rest get + FileExistsError. + """ + path = _get_pid_path() + path.parent.mkdir(parents=True, exist_ok=True) + record = json.dumps(_build_pid_record()) + try: + fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY) + except FileExistsError: + raise # Let caller decide: another gateway is racing us + try: + with os.fdopen(fd, "w", encoding="utf-8") as f: + f.write(record) + except Exception: + try: + path.unlink(missing_ok=True) + except OSError: + pass + raise def write_runtime_status( @@ -266,9 +438,25 @@ def read_runtime_status() -> Optional[dict[str, Any]]: def remove_pid_file() -> None: - """Remove the gateway PID file if it exists.""" + """Remove the gateway PID file, but only if it belongs to this process. + + During --replace handoffs, the old process's atexit handler can fire AFTER + the new process has written its own PID file. Blindly removing the file + would delete the new process's record, leaving the gateway running with no + PID file (invisible to ``get_running_pid()``). + """ try: - _get_pid_path().unlink(missing_ok=True) + path = _get_pid_path() + record = _read_json_file(path) + if record is not None: + try: + file_pid = int(record["pid"]) + except (KeyError, TypeError, ValueError): + file_pid = None + if file_pid is not None and file_pid != os.getpid(): + # PID file belongs to a different process — leave it alone. + return + path.unlink(missing_ok=True) except Exception: pass @@ -313,7 +501,8 @@ def acquire_scoped_lock(scope: str, identity: str, metadata: Optional[dict[str, if not stale: try: os.kill(existing_pid, 0) - except (ProcessLookupError, PermissionError): + except (ProcessLookupError, PermissionError, OSError): + # Windows raises OSError with WinError 87 for invalid pid check stale = True else: current_start = _get_process_start_time(existing_pid) @@ -378,17 +567,43 @@ def release_scoped_lock(scope: str, identity: str) -> None: pass -def release_all_scoped_locks() -> int: - """Remove all scoped lock files in the lock directory. +def release_all_scoped_locks( + *, + owner_pid: Optional[int] = None, + owner_start_time: Optional[int] = None, +) -> int: + """Remove scoped lock files in the lock directory. Called during --replace to clean up stale locks left by stopped/killed - gateway processes that did not release their locks gracefully. + gateway processes that did not release their locks gracefully. When an + ``owner_pid`` is provided, only lock records belonging to that gateway + process are removed. ``owner_start_time`` further narrows the match to + protect against PID reuse. + + When no owner is provided, preserves the legacy behavior and removes every + scoped lock file in the directory. + Returns the number of lock files removed. """ lock_dir = _get_lock_dir() removed = 0 if lock_dir.exists(): for lock_file in lock_dir.glob("*.lock"): + if owner_pid is not None: + record = _read_json_file(lock_file) + if not isinstance(record, dict): + continue + try: + record_pid = int(record.get("pid")) + except (TypeError, ValueError): + continue + if record_pid != owner_pid: + continue + if ( + owner_start_time is not None + and record.get("start_time") != owner_start_time + ): + continue try: lock_file.unlink(missing_ok=True) removed += 1 @@ -397,43 +612,190 @@ def release_all_scoped_locks() -> int: return removed -def get_running_pid() -> Optional[int]: - """Return the PID of a running gateway instance, or ``None``. +# ── --replace takeover marker ───────────────────────────────────────── +# +# When a new gateway starts with ``--replace``, it SIGTERMs the existing +# gateway so it can take over the bot token. PR #5646 made SIGTERM exit +# the gateway with code 1 so ``Restart=on-failure`` can revive it after +# unexpected kills — but that also means a --replace takeover target +# exits 1, which tricks systemd into reviving it 30 seconds later, +# starting a flap loop against the replacer when both services are +# enabled in the user's systemd (e.g. ``hermes.service`` + ``hermes- +# gateway.service``). +# +# The takeover marker breaks the loop: the replacer writes a short-lived +# file naming the target PID + start_time BEFORE sending SIGTERM. +# The target's shutdown handler reads the marker and, if it names +# this process, treats the SIGTERM as a planned takeover and exits 0. +# The marker is unlinked after the target has consumed it, so a stale +# marker left by a crashed replacer can grief at most one future +# shutdown on the same PID — and only within _TAKEOVER_MARKER_TTL_S. + +_TAKEOVER_MARKER_FILENAME = ".gateway-takeover.json" +_TAKEOVER_MARKER_TTL_S = 60 # Marker older than this is treated as stale + + +def _get_takeover_marker_path() -> Path: + """Return the path to the --replace takeover marker file.""" + home = get_hermes_home() + return home / _TAKEOVER_MARKER_FILENAME - Checks the PID file and verifies the process is actually alive. - Cleans up stale PID files automatically. + +def write_takeover_marker(target_pid: int) -> bool: + """Record that ``target_pid`` is being replaced by the current process. + + Captures the target's ``start_time`` so that PID reuse after the + target exits cannot later match the marker. Also records the + replacer's PID and a UTC timestamp for TTL-based staleness checks. + + Returns True on successful write, False on any failure. The caller + should proceed with the SIGTERM even if the write fails (the marker + is a best-effort signal, not a correctness requirement). """ - record = _read_pid_record() + try: + target_start_time = _get_process_start_time(target_pid) + record = { + "target_pid": target_pid, + "target_start_time": target_start_time, + "replacer_pid": os.getpid(), + "written_at": _utc_now_iso(), + } + _write_json_file(_get_takeover_marker_path(), record) + return True + except (OSError, PermissionError): + return False + + +def consume_takeover_marker_for_self() -> bool: + """Check & unlink the takeover marker if it names the current process. + + Returns True only when a valid (non-stale) marker names this PID + + start_time. A returning True indicates the current SIGTERM is a + planned --replace takeover; the caller should exit 0 instead of + signalling ``_signal_initiated_shutdown``. + + Always unlinks the marker on match (and on detected staleness) so + subsequent unrelated signals don't re-trigger. + """ + path = _get_takeover_marker_path() + record = _read_json_file(path) if not record: - remove_pid_file() - return None + return False + # Any malformed or stale marker → drop it and return False try: - pid = int(record["pid"]) + target_pid = int(record["target_pid"]) + target_start_time = record.get("target_start_time") + written_at = record.get("written_at") or "" except (KeyError, TypeError, ValueError): - remove_pid_file() - return None + try: + path.unlink(missing_ok=True) + except OSError: + pass + return False + # TTL guard: a stale marker older than _TAKEOVER_MARKER_TTL_S is ignored. + stale = False try: - os.kill(pid, 0) # signal 0 = existence check, no actual signal sent - except (ProcessLookupError, PermissionError): - remove_pid_file() - return None + written_dt = datetime.fromisoformat(written_at) + age = (datetime.now(timezone.utc) - written_dt).total_seconds() + if age > _TAKEOVER_MARKER_TTL_S: + stale = True + except (TypeError, ValueError): + stale = True # Unparseable timestamp — treat as stale + + if stale: + try: + path.unlink(missing_ok=True) + except OSError: + pass + return False + + # Does the marker name THIS process? + our_pid = os.getpid() + our_start_time = _get_process_start_time(our_pid) + matches = ( + target_pid == our_pid + and target_start_time is not None + and our_start_time is not None + and target_start_time == our_start_time + ) + + # Consume the marker whether it matched or not — a marker that doesn't + # match our identity is stale-for-us anyway. + try: + path.unlink(missing_ok=True) + except OSError: + pass + + return matches + + +def clear_takeover_marker() -> None: + """Remove the takeover marker unconditionally. Safe to call repeatedly.""" + try: + _get_takeover_marker_path().unlink(missing_ok=True) + except OSError: + pass + + +def get_running_pid( + pid_path: Optional[Path] = None, + *, + cleanup_stale: bool = True, +) -> Optional[int]: + """Return the PID of a running gateway instance, or ``None``. - recorded_start = record.get("start_time") - current_start = _get_process_start_time(pid) - if recorded_start is not None and current_start is not None and current_start != recorded_start: - remove_pid_file() + Checks the PID file and verifies the process is actually alive. + Cleans up stale PID files automatically. + """ + resolved_pid_path = pid_path or _get_pid_path() + resolved_lock_path = _get_gateway_lock_path(resolved_pid_path) + lock_active = is_gateway_runtime_lock_active(resolved_lock_path) + if not lock_active: + _cleanup_invalid_pid_path(resolved_pid_path, cleanup_stale=cleanup_stale) return None - if not _looks_like_gateway_process(pid): - if not _record_looks_like_gateway(record): - remove_pid_file() - return None + primary_record = _read_pid_record(resolved_pid_path) + fallback_record = _read_gateway_lock_record(resolved_lock_path) - return pid + for record in (primary_record, fallback_record): + pid = _pid_from_record(record) + if pid is None: + continue + try: + os.kill(pid, 0) # signal 0 = existence check, no actual signal sent + except ProcessLookupError: + continue + except PermissionError: + # The process exists but belongs to another user/service scope. + # With the runtime lock still held, prefer keeping it visible + # rather than deleting the PID file as "stale". + if _record_looks_like_gateway(record): + return pid + continue + except OSError: + # Windows raises OSError with WinError 87 for an invalid pid + # (process is definitely gone). Treat as "process doesn't exist". + continue + + recorded_start = record.get("start_time") + current_start = _get_process_start_time(pid) + if recorded_start is not None and current_start is not None and current_start != recorded_start: + continue + + if _looks_like_gateway_process(pid) or _record_looks_like_gateway(record): + return pid -def is_gateway_running() -> bool: + _cleanup_invalid_pid_path(resolved_pid_path, cleanup_stale=cleanup_stale) + return None + + +def is_gateway_running( + pid_path: Optional[Path] = None, + *, + cleanup_stale: bool = True, +) -> bool: """Check if the gateway daemon is currently running.""" - return get_running_pid() is not None + return get_running_pid(pid_path, cleanup_stale=cleanup_stale) is not None diff --git a/gateway/stream_consumer.py b/gateway/stream_consumer.py index 486d179de924..78e365712d99 100644 --- a/gateway/stream_consumer.py +++ b/gateway/stream_consumer.py @@ -43,6 +43,7 @@ class StreamConsumerConfig: edit_interval: float = 1.0 buffer_threshold: int = 40 cursor: str = " ▉" + buffer_only: bool = False class GatewayStreamConsumer: @@ -64,6 +65,18 @@ class GatewayStreamConsumer: # progressive edits for the remainder of the stream. _MAX_FLOOD_STRIKES = 3 + # Reasoning/thinking tags that models emit inline in content. + # Must stay in sync with cli.py _OPEN_TAGS/_CLOSE_TAGS and + # run_agent.py _strip_think_blocks() tag variants. + _OPEN_THINK_TAGS = ( + "", "", "", + "", "", "", + ) + _CLOSE_THINK_TAGS = ( + "", "", "", + "", "", "", + ) + def __init__( self, adapter: Any, @@ -87,6 +100,18 @@ def __init__( self._flood_strikes = 0 # Consecutive flood-control edit failures self._current_edit_interval = self.cfg.edit_interval # Adaptive backoff self._final_response_sent = False + # Cache adapter lifecycle capability: only platforms that need an + # explicit finalize call (e.g. DingTalk AI Cards) force us to make + # a redundant final edit. Everyone else keeps the fast path. + # Use ``is True`` (not ``bool(...)``) so MagicMock attribute access + # in tests doesn't incorrectly enable this path. + self._adapter_requires_finalize: bool = ( + getattr(adapter, "REQUIRES_EDIT_FINALIZE", False) is True + ) + + # Think-block filter state (mirrors CLI's _stream_delta tag suppression) + self._in_think_block = False + self._think_buffer = "" @property def already_sent(self) -> bool: @@ -132,6 +157,112 @@ def finish(self) -> None: """Signal that the stream is complete.""" self._queue.put(_DONE) + # ── Think-block filtering ──────────────────────────────────────── + # Models like MiniMax emit inline ... blocks in their + # content. The CLI's _stream_delta suppresses these via a state + # machine; we do the same here so gateway users never see raw + # reasoning tags. The agent also strips them from the final + # response (run_agent.py _strip_think_blocks), but the stream + # consumer sends intermediate edits before that stripping happens. + + def _filter_and_accumulate(self, text: str) -> None: + """Add a text delta to the accumulated buffer, suppressing think blocks. + + Uses a state machine that tracks whether we are inside a + reasoning/thinking block. Text inside such blocks is silently + discarded. Partial tags at buffer boundaries are held back in + ``_think_buffer`` until enough characters arrive to decide. + """ + buf = self._think_buffer + text + self._think_buffer = "" + + while buf: + if self._in_think_block: + # Look for the earliest closing tag + best_idx = -1 + best_len = 0 + for tag in self._CLOSE_THINK_TAGS: + idx = buf.find(tag) + if idx != -1 and (best_idx == -1 or idx < best_idx): + best_idx = idx + best_len = len(tag) + + if best_len: + # Found closing tag — discard block, process remainder + self._in_think_block = False + buf = buf[best_idx + best_len:] + else: + # No closing tag yet — hold tail that could be a + # partial closing tag prefix, discard the rest. + max_tag = max(len(t) for t in self._CLOSE_THINK_TAGS) + self._think_buffer = buf[-max_tag:] if len(buf) > max_tag else buf + return + else: + # Look for earliest opening tag at a block boundary + # (start of text / preceded by newline + optional whitespace). + # This prevents false positives when models *mention* tags + # in prose (e.g. "the tag is used for…"). + best_idx = -1 + best_len = 0 + for tag in self._OPEN_THINK_TAGS: + search_start = 0 + while True: + idx = buf.find(tag, search_start) + if idx == -1: + break + # Block-boundary check (mirrors cli.py logic) + if idx == 0: + is_boundary = ( + not self._accumulated + or self._accumulated.endswith("\n") + ) + else: + preceding = buf[:idx] + last_nl = preceding.rfind("\n") + if last_nl == -1: + is_boundary = ( + (not self._accumulated + or self._accumulated.endswith("\n")) + and preceding.strip() == "" + ) + else: + is_boundary = preceding[last_nl + 1:].strip() == "" + + if is_boundary and (best_idx == -1 or idx < best_idx): + best_idx = idx + best_len = len(tag) + break # first boundary hit for this tag is enough + search_start = idx + 1 + + if best_len: + # Emit text before the tag, enter think block + self._accumulated += buf[:best_idx] + self._in_think_block = True + buf = buf[best_idx + best_len:] + else: + # No opening tag — check for a partial tag at the tail + held_back = 0 + for tag in self._OPEN_THINK_TAGS: + for i in range(1, len(tag)): + if buf.endswith(tag[:i]) and i > held_back: + held_back = i + if held_back: + self._accumulated += buf[:-held_back] + self._think_buffer = buf[-held_back:] + else: + self._accumulated += buf + return + + def _flush_think_buffer(self) -> None: + """Flush any held-back partial-tag buffer into accumulated text. + + Called when the stream ends (got_done) so that partial text that + was held back waiting for a possible opening tag is not lost. + """ + if self._think_buffer and not self._in_think_block: + self._accumulated += self._think_buffer + self._think_buffer = "" + async def run(self) -> None: """Async task that drains the queue and edits the platform message.""" # Platform message length limit — leave room for cursor + formatting @@ -156,10 +287,16 @@ async def run(self) -> None: if isinstance(item, tuple) and len(item) == 2 and item[0] is _COMMENTARY: commentary_text = item[1] break - self._accumulated += item + self._filter_and_accumulate(item) except queue.Empty: break + # Flush any held-back partial-tag buffer on stream end + # so trailing text that was waiting for a potential open + # tag is not lost. + if got_done: + self._flush_think_buffer() + # Decide whether to flush an edit now = time.monotonic() elapsed = now - self._last_edit_time @@ -167,10 +304,13 @@ async def run(self) -> None: got_done or got_segment_break or commentary_text is not None - or (elapsed >= self._current_edit_interval - and self._accumulated) - or len(self._accumulated) >= self.cfg.buffer_threshold ) + if not self.cfg.buffer_only: + should_edit = should_edit or ( + (elapsed >= self._current_edit_interval + and self._accumulated) + or len(self._accumulated) >= self.cfg.buffer_threshold + ) current_update_visible = False if should_edit and self._accumulated: @@ -229,7 +369,16 @@ async def run(self) -> None: if not got_done and not got_segment_break and commentary_text is None: display_text += self.cfg.cursor - current_update_visible = await self._send_or_edit(display_text) + # Segment break: finalize the current message so platforms + # that need explicit closure (e.g. DingTalk AI Cards) don't + # leave the previous segment stuck in a loading state when + # the next segment (tool progress, next chunk) creates a + # new message below it. got_done has its own finalize + # path below so we don't finalize here for it. + current_update_visible = await self._send_or_edit( + display_text, + finalize=got_segment_break, + ) self._last_edit_time = time.monotonic() if got_done: @@ -240,10 +389,22 @@ async def run(self) -> None: if self._accumulated: if self._fallback_final_send: await self._send_fallback_final(self._accumulated) - elif current_update_visible: + elif ( + current_update_visible + and not self._adapter_requires_finalize + ): + # Mid-stream edit above already delivered the + # final accumulated content. Skip the redundant + # final edit — but only for adapters that don't + # need an explicit finalize signal. self._final_response_sent = True elif self._message_id: - self._final_response_sent = await self._send_or_edit(self._accumulated) + # Either the mid-stream edit didn't run (no + # visible update this tick) OR the adapter needs + # explicit finalize=True to close the stream. + self._final_response_sent = await self._send_or_edit( + self._accumulated, finalize=True, + ) elif not self._already_sent: self._final_response_sent = await self._send_or_edit(self._accumulated) return @@ -269,17 +430,42 @@ async def run(self) -> None: # a real string like "msg_1", not "__no_edit__", so that case # still resets and creates a fresh segment as intended.) if got_segment_break: + # If the segment-break edit failed to deliver the + # accumulated content (flood control that has not yet + # promoted to fallback mode, or fallback mode itself), + # _accumulated still holds pre-boundary text the user + # never saw. Flush that tail as a continuation message + # before the reset below wipes _accumulated — otherwise + # text generated before the tool boundary is silently + # dropped (issue #8124). + if ( + self._accumulated + and not current_update_visible + and self._message_id + and self._message_id != "__no_edit__" + ): + await self._flush_segment_tail_on_edit_failure() self._reset_segment_state(preserve_no_edit=True) await asyncio.sleep(0.05) # Small yield to not busy-loop except asyncio.CancelledError: # Best-effort final edit on cancellation + _best_effort_ok = False if self._accumulated and self._message_id: try: - await self._send_or_edit(self._accumulated) + _best_effort_ok = bool(await self._send_or_edit(self._accumulated)) except Exception: pass + # Only confirm final delivery if the best-effort send above + # actually succeeded OR if the final response was already + # confirmed before we were cancelled. Previously this + # promoted any partial send (already_sent=True) to + # final_response_sent — which suppressed the gateway's + # fallback send even when only intermediate text (e.g. + # "Let me search…") had been delivered, not the real answer. + if _best_effort_ok and not self._final_response_sent: + self._final_response_sent = True except Exception as e: logger.error("Stream consumer error: %s", e) @@ -377,9 +563,41 @@ async def _send_fallback_final(self, text: str) -> None: self._fallback_final_send = False if not continuation.strip(): # Nothing new to send — the visible partial already matches final text. - self._already_sent = True - self._final_response_sent = True - return + # BUT: if final_text itself has meaningful content (e.g. a timeout + # message after a long tool call), the prefix-based continuation + # calculation may wrongly conclude "already shown" because the + # streamed prefix was from a *previous* segment (before the tool + # boundary). In that case, send the full final_text as-is (#10807). + if final_text.strip() and final_text != self._visible_prefix(): + continuation = final_text + else: + # Defence-in-depth for #7183: the last edit may still show the + # cursor character because fallback mode was entered after an + # edit failure left it stuck. Try one final edit to strip it + # so the message doesn't freeze with a visible ▉. Best-effort + # — if this edit also fails (flood control still active), + # _try_strip_cursor has already been called on fallback entry + # and the adaptive-backoff retries will have had their shot. + if ( + self._message_id + and self._last_sent_text + and self.cfg.cursor + and self._last_sent_text.endswith(self.cfg.cursor) + ): + clean_text = self._last_sent_text[:-len(self.cfg.cursor)] + try: + result = await self.adapter.edit_message( + chat_id=self.chat_id, + message_id=self._message_id, + content=clean_text, + ) + if result.success: + self._last_sent_text = clean_text + except Exception: + pass + self._already_sent = True + self._final_response_sent = True + return raw_limit = getattr(self.adapter, "MAX_MESSAGE_LENGTH", 4096) safe_limit = max(500, raw_limit - 100) @@ -441,6 +659,39 @@ def _is_flood_error(self, result) -> bool: err_lower = err.lower() return "flood" in err_lower or "retry after" in err_lower or "rate" in err_lower + async def _flush_segment_tail_on_edit_failure(self) -> None: + """Deliver un-sent tail content before a segment-break reset. + + When an edit fails (flood control, transport error) and a tool + boundary arrives before the next retry, ``_accumulated`` holds text + that was generated but never shown to the user. Without this flush, + the segment reset would discard that tail and leave a frozen cursor + in the partial message. + + Sends the tail that sits after the last successfully-delivered + prefix as a new message, and best-effort strips the stuck cursor + from the previous partial message. + """ + if not self._fallback_final_send: + await self._try_strip_cursor() + visible = self._fallback_prefix or self._visible_prefix() + tail = self._accumulated + if visible and tail.startswith(visible): + tail = tail[len(visible):].lstrip() + tail = self._clean_for_display(tail) + if not tail.strip(): + return + try: + result = await self.adapter.send( + chat_id=self.chat_id, + content=tail, + metadata=self.metadata, + ) + if result.success: + self._already_sent = True + except Exception as e: + logger.error("Segment-break tail flush error: %s", e) + async def _try_strip_cursor(self) -> None: """Best-effort edit to remove the cursor from the last visible message. @@ -473,37 +724,74 @@ async def _send_commentary(self, text: str) -> bool: content=text, metadata=self.metadata, ) - if result.success: - self._already_sent = True - return True + # Note: do NOT set _already_sent = True here. + # Commentary messages are interim status updates (e.g. "Using browser + # tool..."), not the final response. Setting already_sent would cause + # the final response to be incorrectly suppressed when there are + # multiple tool calls. See: https://github.com/NousResearch/hermes-agent/issues/10454 + return result.success except Exception as e: logger.error("Commentary send error: %s", e) - return False + return False - async def _send_or_edit(self, text: str) -> bool: + async def _send_or_edit(self, text: str, *, finalize: bool = False) -> bool: """Send or edit the streaming message. Returns True if the text was successfully delivered (sent or edited), False otherwise. Callers like the overflow split loop use this to decide whether to advance past the delivered chunk. + + ``finalize`` is True when this is the last edit in a streaming + sequence. """ # Strip MEDIA: directives so they don't appear as visible text. # Media files are delivered as native attachments after the stream # finishes (via _deliver_media_from_response in gateway/run.py). text = self._clean_for_display(text) + # A bare streaming cursor is not meaningful user-visible content and + # can render as a stray tofu/white-box message on some clients. + visible_without_cursor = text + if self.cfg.cursor: + visible_without_cursor = visible_without_cursor.replace(self.cfg.cursor, "") + _visible_stripped = visible_without_cursor.strip() + if not _visible_stripped: + return True # cursor-only / whitespace-only update if not text.strip(): return True # nothing to send is "success" + # Guard: do not create a brand-new standalone message when the only + # visible content is a handful of characters alongside the streaming + # cursor. During rapid tool-calling the model often emits 1-2 tokens + # before switching to tool calls; the resulting "X ▉" message risks + # leaving the cursor permanently visible if the follow-up edit (to + # strip the cursor on segment break) is rate-limited by the platform. + # This was reported on Telegram, Matrix, and other clients where the + # ▉ block character renders as a visible white box ("tofu"). + # Existing messages (edits) are unaffected — only first sends gated. + _MIN_NEW_MSG_CHARS = 4 + if (self._message_id is None + and self.cfg.cursor + and self.cfg.cursor in text + and len(_visible_stripped) < _MIN_NEW_MSG_CHARS): + return True # too short for a standalone message — accumulate more try: if self._message_id is not None: if self._edit_supported: - # Skip if text is identical to what we last sent - if text == self._last_sent_text: + # Skip if text is identical to what we last sent. + # Exception: adapters that require an explicit finalize + # call (REQUIRES_EDIT_FINALIZE) must still receive the + # finalize=True edit even when content is unchanged, so + # their streaming UI can transition out of the in- + # progress state. Everyone else short-circuits. + if text == self._last_sent_text and not ( + finalize and self._adapter_requires_finalize + ): return True # Edit existing message result = await self.adapter.edit_message( chat_id=self.chat_id, message_id=self._message_id, content=text, + finalize=finalize, ) if result.success: self._already_sent = True diff --git a/hermes-already-has-routines.md b/hermes-already-has-routines.md new file mode 100644 index 000000000000..fd4c04d679b4 --- /dev/null +++ b/hermes-already-has-routines.md @@ -0,0 +1,160 @@ +# Hermes Agent Has Had "Routines" Since March + +Anthropic just announced [Claude Code Routines](https://claude.com/blog/introducing-routines-in-claude-code) — scheduled tasks, GitHub event triggers, and API-triggered agent runs. Bundled prompt + repo + connectors, running on their infrastructure. + +It's a good feature. We shipped it two months ago. + +--- + +## The Three Trigger Types — Side by Side + +Claude Code Routines offers three ways to trigger an automation: + +**1. Scheduled (cron)** +> "Every night at 2am: pull the top bug from Linear, attempt a fix, and open a draft PR." + +Hermes equivalent — works today: +```bash +hermes cron create "0 2 * * *" \ + "Pull the top bug from the issue tracker, attempt a fix, and open a draft PR." \ + --name "Nightly bug fix" \ + --deliver telegram +``` + +**2. GitHub Events (webhook)** +> "Flag PRs that touch the /auth-provider module and post to #auth-changes." + +Hermes equivalent — works today: +```bash +hermes webhook subscribe auth-watch \ + --events "pull_request" \ + --prompt "PR #{pull_request.number}: {pull_request.title} by {pull_request.user.login}. Check if it touches the auth-provider module. If yes, summarize the changes." \ + --deliver slack +``` + +**3. API Triggers** +> "Read the alert payload, find the owning service, post a triage summary to #oncall." + +Hermes equivalent — works today: +```bash +hermes webhook subscribe alert-triage \ + --prompt "Alert: {alert.name} — Severity: {alert.severity}. Find the owning service, investigate, and post a triage summary with proposed first steps." \ + --deliver slack +``` + +Every use case in their blog post — backlog triage, docs drift, deploy verification, alert correlation, library porting, bespoke PR review — has a working Hermes implementation. No new features needed. It's been shipping since March 2026. + +--- + +## What's Different + +| | Claude Code Routines | Hermes Agent | +|---|---|---| +| **Scheduled tasks** | ✅ Schedule-based | ✅ Any cron expression + human-readable intervals | +| **GitHub triggers** | ✅ PR, issue, push events | ✅ Any GitHub event via webhook subscriptions | +| **API triggers** | ✅ POST to unique endpoint | ✅ POST to webhook routes with HMAC auth | +| **MCP connectors** | ✅ Native connectors | ✅ Full MCP client support | +| **Script pre-processing** | ❌ | ✅ Python scripts run before agent, inject context | +| **Skill chaining** | ❌ | ✅ Load multiple skills per automation | +| **Daily limit** | 5-25 runs/day | **Unlimited** | +| **Model choice** | Claude only | **Any model** — Claude, GPT, Gemini, DeepSeek, Qwen, local | +| **Delivery targets** | GitHub comments | Telegram, Discord, Slack, SMS, email, GitHub comments, webhooks, local files | +| **Infrastructure** | Anthropic's servers | **Your infrastructure** — VPS, home server, laptop | +| **Data residency** | Anthropic's cloud | **Your machines** | +| **Cost** | Pro/Max/Team/Enterprise subscription | Your API key, your rates | +| **Open source** | No | **Yes** — MIT license | + +--- + +## Things Hermes Does That Routines Can't + +### Script Injection + +Run a Python script *before* the agent. The script's stdout becomes context. The script handles mechanical work (fetching, diffing, computing); the agent handles reasoning. + +```bash +hermes cron create "every 1h" \ + "If CHANGE DETECTED, summarize what changed. If NO_CHANGE, respond with [SILENT]." \ + --script ~/.hermes/scripts/watch-site.py \ + --name "Pricing monitor" \ + --deliver telegram +``` + +The `[SILENT]` pattern means you only get notified when something actually happens. No spam. + +### Multi-Skill Workflows + +Chain specialized skills together. Each skill teaches the agent a specific capability, and the prompt ties them together. + +```bash +hermes cron create "0 8 * * *" \ + "Search arXiv for papers on language model reasoning. Save the top 3 as Obsidian notes." \ + --skills "arxiv,obsidian" \ + --name "Paper digest" +``` + +### Deliver Anywhere + +One automation, any destination: + +```bash +--deliver telegram # Telegram home channel +--deliver discord # Discord home channel +--deliver slack # Slack channel +--deliver sms:+15551234567 # Text message +--deliver telegram:-1001234567890:42 # Specific Telegram forum topic +--deliver local # Save to file, no notification +``` + +### Model-Agnostic + +Your nightly triage can run on Claude. Your deploy verification can run on GPT. Your cost-sensitive monitors can run on DeepSeek or a local model. Same automation system, any backend. + +--- + +## The Limits Tell the Story + +Claude Code Routines: **5 routines per day** on Pro. **25 on Enterprise.** That's their ceiling. + +Hermes has no daily limit. Run 500 automations a day if you want. The only constraint is your API budget, and you choose which models to use for which tasks. + +A nightly backlog triage on Sonnet costs roughly $0.02-0.05. A monitoring check on DeepSeek costs fractions of a cent. You control the economics. + +--- + +## Get Started + +Hermes Agent is open source and free. The automation infrastructure — cron scheduler, webhook platform, skill system, multi-platform delivery — is built in. + +```bash +pip install hermes-agent +hermes setup +``` + +Set up a scheduled task in 30 seconds: +```bash +hermes cron create "0 9 * * 1" \ + "Generate a weekly AI news digest. Search the web for major announcements, trending repos, and notable papers. Keep it under 500 words with links." \ + --name "Weekly digest" \ + --deliver telegram +``` + +Set up a GitHub webhook in 60 seconds: +```bash +hermes gateway setup # enable webhooks +hermes webhook subscribe pr-review \ + --events "pull_request" \ + --prompt "Review PR #{pull_request.number}: {pull_request.title}" \ + --skills "github-code-review" \ + --deliver github_comment +``` + +Full automation templates gallery: [hermes-agent.nousresearch.com/docs/guides/automation-templates](https://hermes-agent.nousresearch.com/docs/guides/automation-templates) + +Documentation: [hermes-agent.nousresearch.com](https://hermes-agent.nousresearch.com) + +GitHub: [github.com/NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent) + +--- + +*Hermes Agent is built by [Nous Research](https://nousresearch.com). Open source, model-agnostic, runs on your infrastructure.* diff --git a/hermes_cli/__init__.py b/hermes_cli/__init__.py index 632aa5bae0f4..2bf9acb4001e 100644 --- a/hermes_cli/__init__.py +++ b/hermes_cli/__init__.py @@ -11,5 +11,5 @@ - hermes cron - Manage cron jobs """ -__version__ = "0.9.0" -__release_date__ = "2026.4.13" +__version__ = "0.11.0" +__release_date__ = "2026.4.23" diff --git a/hermes_cli/auth.py b/hermes_cli/auth.py index 795e5ea09f0d..28c5bd9a617c 100644 --- a/hermes_cli/auth.py +++ b/hermes_cli/auth.py @@ -20,6 +20,7 @@ import os import shutil import shlex +import ssl import stat import base64 import hashlib @@ -70,6 +71,9 @@ DEFAULT_QWEN_BASE_URL = "https://portal.qwen.ai/v1" DEFAULT_GITHUB_MODELS_BASE_URL = "https://api.githubcopilot.com" DEFAULT_COPILOT_ACP_BASE_URL = "acp://copilot" +DEFAULT_OLLAMA_CLOUD_BASE_URL = "https://ollama.com/v1" +STEPFUN_STEP_PLAN_INTL_BASE_URL = "https://api.stepfun.ai/step_plan/v1" +STEPFUN_STEP_PLAN_CN_BASE_URL = "https://api.stepfun.com/step_plan/v1" CODEX_OAUTH_CLIENT_ID = "app_EMoamEEZ73f0CkXaXp7hrann" CODEX_OAUTH_TOKEN_URL = "https://auth.openai.com/oauth/token" CODEX_ACCESS_TOKEN_REFRESH_SKEW_SECONDS = 120 @@ -77,6 +81,10 @@ QWEN_OAUTH_TOKEN_URL = "https://chat.qwen.ai/api/v1/oauth2/token" QWEN_ACCESS_TOKEN_REFRESH_SKEW_SECONDS = 120 +# Google Gemini OAuth (google-gemini-cli provider, Cloud Code Assist backend) +DEFAULT_GEMINI_CLOUDCODE_BASE_URL = "cloudcode-pa://google" +GEMINI_OAUTH_ACCESS_TOKEN_REFRESH_SKEW_SECONDS = 60 # refresh 60s before expiry + # ============================================================================= # Provider Registry @@ -121,6 +129,12 @@ class ProviderConfig: auth_type="oauth_external", inference_base_url=DEFAULT_QWEN_BASE_URL, ), + "google-gemini-cli": ProviderConfig( + id="google-gemini-cli", + name="Google Gemini (OAuth)", + auth_type="oauth_external", + inference_base_url=DEFAULT_GEMINI_CLOUDCODE_BASE_URL, + ), "copilot": ProviderConfig( id="copilot", name="GitHub Copilot", @@ -140,7 +154,7 @@ class ProviderConfig: id="gemini", name="Google AI Studio", auth_type="api_key", - inference_base_url="https://generativelanguage.googleapis.com/v1beta/openai", + inference_base_url="https://generativelanguage.googleapis.com/v1beta", api_key_env_vars=("GOOGLE_API_KEY", "GEMINI_API_KEY"), base_url_env_var="GEMINI_BASE_URL", ), @@ -156,8 +170,11 @@ class ProviderConfig: id="kimi-coding", name="Kimi / Moonshot", auth_type="api_key", + # Legacy platform.moonshot.ai keys use this endpoint (OpenAI-compat). + # sk-kimi- (Kimi Code) keys are auto-redirected to api.kimi.com/coding + # by _resolve_kimi_base_url() below. inference_base_url="https://api.moonshot.ai/v1", - api_key_env_vars=("KIMI_API_KEY",), + api_key_env_vars=("KIMI_API_KEY", "KIMI_CODING_API_KEY"), base_url_env_var="KIMI_BASE_URL", ), "kimi-coding-cn": ProviderConfig( @@ -167,6 +184,22 @@ class ProviderConfig: inference_base_url="https://api.moonshot.cn/v1", api_key_env_vars=("KIMI_CN_API_KEY",), ), + "stepfun": ProviderConfig( + id="stepfun", + name="StepFun Step Plan", + auth_type="api_key", + inference_base_url=STEPFUN_STEP_PLAN_INTL_BASE_URL, + api_key_env_vars=("STEPFUN_API_KEY",), + base_url_env_var="STEPFUN_BASE_URL", + ), + "arcee": ProviderConfig( + id="arcee", + name="Arcee AI", + auth_type="api_key", + inference_base_url="https://api.arcee.ai/api/v1", + api_key_env_vars=("ARCEEAI_API_KEY",), + base_url_env_var="ARCEE_BASE_URL", + ), "minimax": ProviderConfig( id="minimax", name="MiniMax", @@ -181,6 +214,7 @@ class ProviderConfig: auth_type="api_key", inference_base_url="https://api.anthropic.com", api_key_env_vars=("ANTHROPIC_API_KEY", "ANTHROPIC_TOKEN", "CLAUDE_CODE_OAUTH_TOKEN"), + base_url_env_var="ANTHROPIC_BASE_URL", ), "alibaba": ProviderConfig( id="alibaba", @@ -214,9 +248,17 @@ class ProviderConfig: api_key_env_vars=("XAI_API_KEY",), base_url_env_var="XAI_BASE_URL", ), + "nvidia": ProviderConfig( + id="nvidia", + name="NVIDIA NIM", + auth_type="api_key", + inference_base_url="https://integrate.api.nvidia.com/v1", + api_key_env_vars=("NVIDIA_API_KEY",), + base_url_env_var="NVIDIA_BASE_URL", + ), "ai-gateway": ProviderConfig( id="ai-gateway", - name="AI Gateway", + name="Vercel AI Gateway", auth_type="api_key", inference_base_url="https://ai-gateway.vercel.sh/v1", api_key_env_vars=("AI_GATEWAY_API_KEY",), @@ -266,6 +308,22 @@ class ProviderConfig: api_key_env_vars=("XIAOMI_API_KEY",), base_url_env_var="XIAOMI_BASE_URL", ), + "ollama-cloud": ProviderConfig( + id="ollama-cloud", + name="Ollama Cloud", + auth_type="api_key", + inference_base_url=DEFAULT_OLLAMA_CLOUD_BASE_URL, + api_key_env_vars=("OLLAMA_API_KEY",), + base_url_env_var="OLLAMA_BASE_URL", + ), + "bedrock": ProviderConfig( + id="bedrock", + name="AWS Bedrock", + auth_type="aws_sdk", + inference_base_url="https://bedrock-runtime.us-east-1.amazonaws.com", + api_key_env_vars=(), + base_url_env_var="BEDROCK_BASE_URL", + ), } @@ -296,10 +354,16 @@ def get_anthropic_key() -> str: # ============================================================================= # Kimi Code (kimi.com/code) issues keys prefixed "sk-kimi-" that only work -# on api.kimi.com/coding/v1. Legacy keys from platform.moonshot.ai work on -# api.moonshot.ai/v1 (the default). Auto-detect when user hasn't set +# on api.kimi.com/coding. Legacy keys from platform.moonshot.ai work on +# api.moonshot.ai/v1 (the old default). Auto-detect when user hasn't set # KIMI_BASE_URL explicitly. -KIMI_CODE_BASE_URL = "https://api.kimi.com/coding/v1" +# +# Note: the base URL intentionally has NO /v1 suffix. The /coding endpoint +# speaks the Anthropic Messages protocol, and the anthropic SDK appends +# "/v1/messages" internally — so "/coding" + SDK suffix → "/coding/v1/messages" +# (the correct target). Using "/coding/v1" here would produce +# "/coding/v1/v1/messages" (a 404). +KIMI_CODE_BASE_URL = "https://api.kimi.com/coding" def _resolve_kimi_base_url(api_key: str, default_url: str, env_override: str) -> str: @@ -310,6 +374,9 @@ def _resolve_kimi_base_url(api_key: str, default_url: str, env_override: str) -> """ if env_override: return env_override + # No key → nothing to infer from. Return default without inspecting. + if not api_key: + return default_url if api_key.startswith("sk-kimi-"): return KIMI_CODE_BASE_URL return default_url @@ -375,13 +442,16 @@ def _resolve_api_key_provider_secret( # Z.AI has separate billing for general vs coding plans, and global vs China # endpoints. A key that works on one may return "Insufficient balance" on # another. We probe at setup time and store the working endpoint. +# Each entry lists candidate models to try in order — newer coding plan accounts +# may only have access to recent models (glm-5.1, glm-5v-turbo) while older +# ones still use glm-4.7. ZAI_ENDPOINTS = [ - # (id, base_url, default_model, label) - ("global", "https://api.z.ai/api/paas/v4", "glm-5", "Global"), - ("cn", "https://open.bigmodel.cn/api/paas/v4", "glm-5", "China"), - ("coding-global", "https://api.z.ai/api/coding/paas/v4", "glm-4.7", "Global (Coding Plan)"), - ("coding-cn", "https://open.bigmodel.cn/api/coding/paas/v4", "glm-4.7", "China (Coding Plan)"), + # (id, base_url, probe_models, label) + ("global", "https://api.z.ai/api/paas/v4", ["glm-5"], "Global"), + ("cn", "https://open.bigmodel.cn/api/paas/v4", ["glm-5"], "China"), + ("coding-global", "https://api.z.ai/api/coding/paas/v4", ["glm-5.1", "glm-5v-turbo", "glm-4.7"], "Global (Coding Plan)"), + ("coding-cn", "https://open.bigmodel.cn/api/coding/paas/v4", ["glm-5.1", "glm-5v-turbo", "glm-4.7"], "China (Coding Plan)"), ] @@ -389,35 +459,37 @@ def detect_zai_endpoint(api_key: str, timeout: float = 8.0) -> Optional[Dict[str """Probe z.ai endpoints to find one that accepts this API key. Returns {"id": ..., "base_url": ..., "model": ..., "label": ...} for the - first working endpoint, or None if all fail. + first working endpoint, or None if all fail. For endpoints with multiple + candidate models, tries each in order and returns the first that succeeds. """ - for ep_id, base_url, model, label in ZAI_ENDPOINTS: - try: - resp = httpx.post( - f"{base_url}/chat/completions", - headers={ - "Authorization": f"Bearer {api_key}", - "Content-Type": "application/json", - }, - json={ - "model": model, - "stream": False, - "max_tokens": 1, - "messages": [{"role": "user", "content": "ping"}], - }, - timeout=timeout, - ) - if resp.status_code == 200: - logger.debug("Z.AI endpoint probe: %s (%s) OK", ep_id, base_url) - return { - "id": ep_id, - "base_url": base_url, - "model": model, - "label": label, - } - logger.debug("Z.AI endpoint probe: %s returned %s", ep_id, resp.status_code) - except Exception as exc: - logger.debug("Z.AI endpoint probe: %s failed: %s", ep_id, exc) + for ep_id, base_url, probe_models, label in ZAI_ENDPOINTS: + for model in probe_models: + try: + resp = httpx.post( + f"{base_url}/chat/completions", + headers={ + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + }, + json={ + "model": model, + "stream": False, + "max_tokens": 1, + "messages": [{"role": "user", "content": "ping"}], + }, + timeout=timeout, + ) + if resp.status_code == 200: + logger.debug("Z.AI endpoint probe: %s (%s) model=%s OK", ep_id, base_url, model) + return { + "id": ep_id, + "base_url": base_url, + "model": model, + "label": label, + } + logger.debug("Z.AI endpoint probe: %s model=%s returned %s", ep_id, model, resp.status_code) + except Exception as exc: + logger.debug("Z.AI endpoint probe: %s model=%s failed: %s", ep_id, model, exc) return None @@ -432,6 +504,14 @@ def _resolve_zai_base_url(api_key: str, default_url: str, env_override: str) -> if env_override: return env_override + # No API key set → don't probe (would fire N×M HTTPS requests with an + # empty Bearer token, all returning 401). This path is hit during + # auxiliary-client auto-detection when the user has no Z.AI credentials + # at all — the caller discards the result immediately, so the probe is + # pure latency for every AIAgent construction. + if not api_key: + return default_url + # Check provider-state cache for a previously-detected endpoint. auth_store = _load_auth_store() state = _load_provider_state(auth_store, "zai") or {} @@ -539,7 +619,25 @@ def _oauth_trace(event: str, *, sequence_id: Optional[str] = None, **fields: Any # ============================================================================= def _auth_file_path() -> Path: - return get_hermes_home() / "auth.json" + path = get_hermes_home() / "auth.json" + # Seat belt: if pytest is running and HERMES_HOME resolves to the real + # user's auth store, refuse rather than silently corrupt it. This catches + # tests that forgot to monkeypatch HERMES_HOME, tests invoked without the + # hermetic conftest, or sandbox escapes via threads/subprocesses. In + # production (no PYTEST_CURRENT_TEST) this is a single dict lookup. + if os.environ.get("PYTEST_CURRENT_TEST"): + real_home_auth = (Path.home() / ".hermes" / "auth.json").resolve(strict=False) + try: + resolved = path.resolve(strict=False) + except Exception: + resolved = path + if resolved == real_home_auth: + raise RuntimeError( + f"Refusing to touch real user auth store during test run: {path}. " + "Set HERMES_HOME to a tmp_path in your test fixture, or run " + "via scripts/run_tests.sh for hermetic CI-parity env." + ) + return path def _auth_lock_path() -> Path: @@ -733,6 +831,28 @@ def is_source_suppressed(provider_id: str, source: str) -> bool: return False +def unsuppress_credential_source(provider_id: str, source: str) -> bool: + """Clear a suppression marker so the source will be re-seeded on the next load. + + Returns True if a marker was cleared, False if no marker existed. + """ + with _auth_store_lock(): + auth_store = _load_auth_store() + suppressed = auth_store.get("suppressed_sources") + if not isinstance(suppressed, dict): + return False + provider_list = suppressed.get(provider_id) + if not isinstance(provider_list, list) or source not in provider_list: + return False + provider_list.remove(source) + if not provider_list: + suppressed.pop(provider_id, None) + if not suppressed: + auth_store.pop("suppressed_sources", None) + _save_auth_store(auth_store) + return True + + def get_provider_auth_state(provider_id: str) -> Optional[Dict[str, Any]]: """Return persisted auth state for a provider, or None.""" auth_store = _load_auth_store() @@ -898,8 +1018,11 @@ def resolve_provider( _PROVIDER_ALIASES = { "glm": "zai", "z-ai": "zai", "z.ai": "zai", "zhipu": "zai", "google": "gemini", "google-gemini": "gemini", "google-ai-studio": "gemini", + "x-ai": "xai", "x.ai": "xai", "grok": "xai", "kimi": "kimi-coding", "kimi-for-coding": "kimi-coding", "moonshot": "kimi-coding", "kimi-cn": "kimi-coding-cn", "moonshot-cn": "kimi-coding-cn", + "step": "stepfun", "stepfun-coding-plan": "stepfun", + "arcee-ai": "arcee", "arceeai": "arcee", "minimax-china": "minimax-cn", "minimax_cn": "minimax-cn", "claude": "anthropic", "claude-code": "anthropic", "github": "copilot", "github-copilot": "copilot", @@ -907,14 +1030,16 @@ def resolve_provider( "github-copilot-acp": "copilot-acp", "copilot-acp-agent": "copilot-acp", "aigateway": "ai-gateway", "vercel": "ai-gateway", "vercel-ai-gateway": "ai-gateway", "opencode": "opencode-zen", "zen": "opencode-zen", - "qwen-portal": "qwen-oauth", "qwen-cli": "qwen-oauth", "qwen-oauth": "qwen-oauth", + "qwen-portal": "qwen-oauth", "qwen-cli": "qwen-oauth", "qwen-oauth": "qwen-oauth", "google-gemini-cli": "google-gemini-cli", "gemini-cli": "google-gemini-cli", "gemini-oauth": "google-gemini-cli", "hf": "huggingface", "hugging-face": "huggingface", "huggingface-hub": "huggingface", "mimo": "xiaomi", "xiaomi-mimo": "xiaomi", + "aws": "bedrock", "aws-bedrock": "bedrock", "amazon-bedrock": "bedrock", "amazon": "bedrock", "go": "opencode-go", "opencode-go-sub": "opencode-go", "kilo": "kilocode", "kilo-code": "kilocode", "kilo-gateway": "kilocode", # Local server aliases — route through the generic custom provider "lmstudio": "custom", "lm-studio": "custom", "lm_studio": "custom", - "ollama": "custom", "vllm": "custom", "llamacpp": "custom", + "ollama": "custom", "ollama_cloud": "ollama-cloud", + "vllm": "custom", "llamacpp": "custom", "llama.cpp": "custom", "llama-cpp": "custom", } normalized = _PROVIDER_ALIASES.get(normalized, normalized) @@ -966,6 +1091,15 @@ def resolve_provider( if has_usable_secret(os.getenv(env_var, "")): return pid + # AWS Bedrock — detect via boto3 credential chain (IAM roles, SSO, env vars). + # This runs after API-key providers so explicit keys always win. + try: + from agent.bedrock_adapter import has_aws_credentials + if has_aws_credentials(): + return "bedrock" + except ImportError: + pass # boto3 not installed — skip Bedrock auto-detection + raise AuthError( "No inference provider configured. Run 'hermes model' to choose a " "provider and model, or set an API key (OPENROUTER_API_KEY, " @@ -1208,6 +1342,83 @@ def get_qwen_auth_status() -> Dict[str, Any]: } +# ============================================================================= +# Google Gemini OAuth (google-gemini-cli) — PKCE flow + Cloud Code Assist. +# +# Tokens live in ~/.hermes/auth/google_oauth.json (managed by agent.google_oauth). +# The `base_url` here is the marker "cloudcode-pa://google" that run_agent.py +# uses to construct a GeminiCloudCodeClient instead of the default OpenAI SDK. +# Actual HTTP traffic goes to https://cloudcode-pa.googleapis.com/v1internal:*. +# ============================================================================= + +def resolve_gemini_oauth_runtime_credentials( + *, + force_refresh: bool = False, +) -> Dict[str, Any]: + """Resolve runtime OAuth creds for google-gemini-cli.""" + try: + from agent.google_oauth import ( + GoogleOAuthError, + _credentials_path, + get_valid_access_token, + load_credentials, + ) + except ImportError as exc: + raise AuthError( + f"agent.google_oauth is not importable: {exc}", + provider="google-gemini-cli", + code="google_oauth_module_missing", + ) from exc + + try: + access_token = get_valid_access_token(force_refresh=force_refresh) + except GoogleOAuthError as exc: + raise AuthError( + str(exc), + provider="google-gemini-cli", + code=exc.code, + ) from exc + + creds = load_credentials() + base_url = DEFAULT_GEMINI_CLOUDCODE_BASE_URL + return { + "provider": "google-gemini-cli", + "base_url": base_url, + "api_key": access_token, + "source": "google-oauth", + "expires_at_ms": (creds.expires_ms if creds else None), + "auth_file": str(_credentials_path()), + "email": (creds.email if creds else "") or "", + "project_id": (creds.project_id if creds else "") or "", + } + + +def get_gemini_oauth_auth_status() -> Dict[str, Any]: + """Return a status dict for `hermes auth list` / `hermes status`.""" + try: + from agent.google_oauth import _credentials_path, load_credentials + except ImportError: + return {"logged_in": False, "error": "agent.google_oauth unavailable"} + auth_path = _credentials_path() + creds = load_credentials() + if creds is None or not creds.access_token: + return { + "logged_in": False, + "auth_file": str(auth_path), + "error": "not logged in", + } + return { + "logged_in": True, + "auth_file": str(auth_path), + "source": "google-oauth", + "api_key": creds.access_token, + "expires_at_ms": creds.expires_ms, + "email": creds.email, + "project_id": creds.project_id, + } + + + # ============================================================================= # SSH / remote session detection # ============================================================================= @@ -1274,49 +1485,6 @@ def _read_codex_tokens(*, _lock: bool = True) -> Dict[str, Any]: } -def _write_codex_cli_tokens( - access_token: str, - refresh_token: str, - *, - last_refresh: Optional[str] = None, -) -> None: - """Write refreshed tokens back to ~/.codex/auth.json. - - OpenAI OAuth refresh tokens are single-use and rotate on every refresh. - When Hermes refreshes a token it consumes the old refresh_token; if we - don't write the new pair back, the Codex CLI (or VS Code extension) will - fail with ``refresh_token_reused`` on its next refresh attempt. - - This mirrors the Anthropic write-back to ~/.claude/.credentials.json - via ``_write_claude_code_credentials()``. - """ - codex_home = os.getenv("CODEX_HOME", "").strip() - if not codex_home: - codex_home = str(Path.home() / ".codex") - auth_path = Path(codex_home).expanduser() / "auth.json" - try: - existing: Dict[str, Any] = {} - if auth_path.is_file(): - existing = json.loads(auth_path.read_text(encoding="utf-8")) - if not isinstance(existing, dict): - existing = {} - - tokens_dict = existing.get("tokens") - if not isinstance(tokens_dict, dict): - tokens_dict = {} - tokens_dict["access_token"] = access_token - tokens_dict["refresh_token"] = refresh_token - existing["tokens"] = tokens_dict - if last_refresh is not None: - existing["last_refresh"] = last_refresh - - auth_path.parent.mkdir(parents=True, exist_ok=True) - auth_path.write_text(json.dumps(existing, indent=2), encoding="utf-8") - auth_path.chmod(0o600) - except (OSError, IOError) as exc: - logger.debug("Failed to write refreshed tokens to %s: %s", auth_path, exc) - - def _save_codex_tokens(tokens: Dict[str, str], last_refresh: str = None) -> None: """Save Codex OAuth tokens to Hermes auth store (~/.hermes/auth.json).""" if last_refresh is None: @@ -1384,6 +1552,11 @@ def refresh_codex_oauth_pure( "then run `hermes auth` to re-authenticate." ) relogin_required = True + # A 401/403 from the token endpoint always means the refresh token + # is invalid/expired — force relogin even if the body error code + # wasn't one of the known strings above. + if response.status_code in (401, 403) and not relogin_required: + relogin_required = True raise AuthError( message, provider="openai-codex", @@ -1439,12 +1612,6 @@ def _refresh_codex_auth_tokens( updated_tokens["refresh_token"] = refreshed["refresh_token"] _save_codex_tokens(updated_tokens) - # Write back to ~/.codex/auth.json so Codex CLI / VS Code stay in sync. - _write_codex_cli_tokens( - refreshed["access_token"], - refreshed["refresh_token"], - last_refresh=refreshed.get("last_refresh"), - ) return updated_tokens @@ -1489,25 +1656,7 @@ def resolve_codex_runtime_credentials( refresh_skew_seconds: int = CODEX_ACCESS_TOKEN_REFRESH_SKEW_SECONDS, ) -> Dict[str, Any]: """Resolve runtime credentials from Hermes's own Codex token store.""" - try: - data = _read_codex_tokens() - except AuthError as orig_err: - # Only attempt migration when there are NO tokens stored at all - # (code == "codex_auth_missing"), not when tokens exist but are invalid. - if orig_err.code != "codex_auth_missing": - raise - - # Migration: user had Codex as active provider with old storage (~/.codex/). - cli_tokens = _import_codex_cli_tokens() - if cli_tokens: - logger.info("Migrating Codex credentials from ~/.codex/ to Hermes auth store") - print("⚠️ Migrating Codex credentials to Hermes's own auth store.") - print(" This avoids conflicts with Codex CLI and VS Code.") - print(" Run `hermes auth` to create a fully independent session.\n") - _save_codex_tokens(cli_tokens) - data = _read_codex_tokens() - else: - raise + data = _read_codex_tokens() tokens = dict(data["tokens"]) access_token = str(tokens.get("access_token", "") or "").strip() refresh_timeout_seconds = float(os.getenv("HERMES_CODEX_REFRESH_TIMEOUT_SECONDS", "20")) @@ -1554,7 +1703,7 @@ def _resolve_verify( insecure: Optional[bool] = None, ca_bundle: Optional[str] = None, auth_state: Optional[Dict[str, Any]] = None, -) -> bool | str: +) -> bool | ssl.SSLContext: tls_state = auth_state.get("tls") if isinstance(auth_state, dict) else {} tls_state = tls_state if isinstance(tls_state, dict) else {} @@ -1574,13 +1723,12 @@ def _resolve_verify( if effective_ca: ca_path = str(effective_ca) if not os.path.isfile(ca_path): - import logging - logging.getLogger("hermes.auth").warning( + logger.warning( "CA bundle path does not exist: %s — falling back to default certificates", ca_path, ) return True - return ca_path + return ssl.create_default_context(cafile=ca_path) return True @@ -1999,6 +2147,62 @@ def refresh_nous_oauth_from_state( ) +NOUS_DEVICE_CODE_SOURCE = "device_code" + + +def persist_nous_credentials( + creds: Dict[str, Any], + *, + label: Optional[str] = None, +): + """Persist minted Nous OAuth credentials as the singleton provider state + and ensure the credential pool is in sync. + + Nous credentials are read at runtime from two independent locations: + + - ``providers.nous``: singleton state read by + ``resolve_nous_runtime_credentials()`` during 401 recovery and by + ``_seed_from_singletons()`` during pool load. + - ``credential_pool.nous``: used by the runtime ``pool.select()`` path. + + Historically ``hermes auth add nous`` wrote a ``manual:device_code`` pool + entry only, skipping ``providers.nous``. When the 24h agent_key TTL + expired, the recovery path read the empty singleton state and raised + ``AuthError`` silently (``logger.debug`` at INFO level). + + This helper writes ``providers.nous`` then calls ``load_pool("nous")`` so + ``_seed_from_singletons`` materialises the canonical ``device_code`` pool + entry from the singleton. Re-running login upserts the same entry in + place; the pool never accumulates duplicate device_code rows. + + ``label`` is an optional user-chosen display name (from + ``hermes auth add nous --label ``). It gets embedded in the + singleton state so that ``_seed_from_singletons`` uses it as the pool + entry's label on every subsequent ``load_pool("nous")`` instead of the + auto-derived token fingerprint. When ``None``, the auto-derived label + via ``label_from_token`` is used (unchanged default behaviour). + + Returns the upserted :class:`PooledCredential` entry (or ``None`` if + seeding somehow produced no match — shouldn't happen). + """ + from agent.credential_pool import load_pool + + state = dict(creds) + if label and str(label).strip(): + state["label"] = str(label).strip() + + with _auth_store_lock(): + auth_store = _load_auth_store() + _save_provider_state(auth_store, "nous", state) + _save_auth_store(auth_store) + + pool = load_pool("nous") + return next( + (e for e in pool.entries() if e.source == NOUS_DEVICE_CODE_SOURCE), + None, + ) + + def resolve_nous_runtime_credentials( *, min_key_ttl_seconds: int = DEFAULT_AGENT_KEY_MIN_TTL_SECONDS, @@ -2253,7 +2457,40 @@ def _persist_state(reason: str) -> None: # ============================================================================= def get_nous_auth_status() -> Dict[str, Any]: - """Status snapshot for `hermes status` output.""" + """Status snapshot for `hermes status` output. + + Checks the credential pool first (where the dashboard device-code flow + and ``hermes auth`` store credentials), then falls back to the legacy + auth-store provider state. + """ + # Check credential pool first — the dashboard device-code flow saves + # here but may not have written to the auth store yet. + try: + from agent.credential_pool import load_pool + pool = load_pool("nous") + if pool and pool.has_credentials(): + entry = pool.select() + if entry is not None: + access_token = ( + getattr(entry, "access_token", None) + or getattr(entry, "runtime_api_key", "") + ) + if access_token: + return { + "logged_in": True, + "portal_base_url": getattr(entry, "portal_base_url", None) + or getattr(entry, "base_url", None), + "inference_base_url": getattr(entry, "inference_base_url", None) + or getattr(entry, "base_url", None), + "access_token": access_token, + "access_expires_at": getattr(entry, "expires_at", None), + "agent_key_expires_at": getattr(entry, "agent_key_expires_at", None), + "has_refresh_token": bool(getattr(entry, "refresh_token", None)), + } + except Exception: + pass + + # Fall back to auth-store provider state state = get_provider_auth_state("nous") if not state: return { @@ -2337,7 +2574,7 @@ def get_api_key_provider_status(provider_id: str) -> Dict[str, Any]: if pconfig.base_url_env_var: env_url = os.getenv(pconfig.base_url_env_var, "").strip() - if provider_id == "kimi-coding": + if provider_id in ("kimi-coding", "kimi-coding-cn"): base_url = _resolve_kimi_base_url(api_key, pconfig.inference_base_url, env_url) elif env_url: base_url = env_url @@ -2393,12 +2630,21 @@ def get_auth_status(provider_id: Optional[str] = None) -> Dict[str, Any]: return get_codex_auth_status() if target == "qwen-oauth": return get_qwen_auth_status() + if target == "google-gemini-cli": + return get_gemini_oauth_auth_status() if target == "copilot-acp": return get_external_process_provider_status(target) # API-key providers pconfig = PROVIDER_REGISTRY.get(target) if pconfig and pconfig.auth_type == "api_key": return get_api_key_provider_status(target) + # AWS SDK providers (Bedrock) — check via boto3 credential chain + if pconfig and pconfig.auth_type == "aws_sdk": + try: + from agent.bedrock_adapter import has_aws_credentials + return {"logged_in": has_aws_credentials(), "provider": target} + except ImportError: + return {"logged_in": False, "provider": target, "error": "boto3 not installed"} return {"logged_in": False} @@ -2423,7 +2669,7 @@ def resolve_api_key_provider_credentials(provider_id: str) -> Dict[str, Any]: if pconfig.base_url_env_var: env_url = os.getenv(pconfig.base_url_env_var, "").strip() - if provider_id == "kimi-coding": + if provider_id in ("kimi-coding", "kimi-coding-cn"): base_url = _resolve_kimi_base_url(api_key, pconfig.inference_base_url, env_url) elif provider_id == "zai": base_url = _resolve_zai_base_url(api_key, pconfig.inference_base_url, env_url) @@ -2525,6 +2771,17 @@ def _update_config_for_provider( # Clear stale base_url to prevent contamination when switching providers model_cfg.pop("base_url", None) + # Clear stale api_key/api_mode left over from a previous custom provider. + # When the user switches from e.g. a MiniMax custom endpoint + # (api_mode=anthropic_messages, api_key=mxp-...) to a built-in provider + # (e.g. OpenRouter), the stale api_key/api_mode would override the new + # provider's credentials and transport choice. Built-in providers that + # need a specific api_mode (copilot, xai) set it at request-resolution + # time via `_copilot_runtime_api_mode` / `_detect_api_mode_for_url`, so + # removing the persisted value here is safe. + model_cfg.pop("api_key", None) + model_cfg.pop("api_mode", None) + # When switching to a non-OpenRouter provider, ensure model.default is # valid for the new provider. An OpenRouter-formatted name like # "anthropic/claude-opus-4.6" will fail on direct-API providers. @@ -3125,6 +3382,14 @@ def _login_nous(args, pconfig: ProviderConfig) -> None: inference_base_url = auth_state["inference_base_url"] + # Snapshot the prior active_provider BEFORE _save_provider_state + # overwrites it to "nous". If the user picks "Skip (keep current)" + # during model selection below, we restore this so the user's previous + # provider (e.g. openrouter) is preserved. + with _auth_store_lock(): + _prior_store = _load_auth_store() + prior_active_provider = _prior_store.get("active_provider") + with _auth_store_lock(): auth_store = _load_auth_store() _save_provider_state(auth_store, "nous", auth_state) @@ -3149,7 +3414,7 @@ def _login_nous(args, pconfig: ProviderConfig) -> None: ) from hermes_cli.models import ( - _PROVIDER_MODELS, get_pricing_for_provider, filter_nous_free_models, + _PROVIDER_MODELS, get_pricing_for_provider, check_nous_free_tier, partition_nous_models_by_tier, ) model_ids = _PROVIDER_MODELS.get("nous", []) @@ -3158,7 +3423,6 @@ def _login_nous(args, pconfig: ProviderConfig) -> None: unavailable_models: list = [] if model_ids: pricing = get_pricing_for_provider("nous") - model_ids = filter_nous_free_models(model_ids, pricing) free_tier = check_nous_free_tier() if free_tier: model_ids, unavailable_models = partition_nous_models_by_tier( @@ -3184,6 +3448,27 @@ def _login_nous(args, pconfig: ProviderConfig) -> None: print(f"Login succeeded, but could not fetch available models. Reason: {message}") # Write provider + model atomically so config is never mismatched. + # If no model was selected (user picked "Skip (keep current)", + # model list fetch failed, or no curated models were available), + # preserve the user's previous provider — don't silently switch + # them to Nous with a mismatched model. The Nous OAuth tokens + # stay saved for future use. + if not selected_model: + # Restore the prior active_provider that _save_provider_state + # overwrote to "nous". config.yaml model.provider is left + # untouched, so the user's previous provider is fully preserved. + with _auth_store_lock(): + auth_store = _load_auth_store() + if prior_active_provider: + auth_store["active_provider"] = prior_active_provider + else: + auth_store.pop("active_provider", None) + _save_auth_store(auth_store) + print() + print("No provider change. Nous credentials saved for future use.") + print(" Run `hermes model` again to switch to Nous Portal.") + return + config_path = _update_config_for_provider( "nous", inference_base_url, default_model=selected_model, ) diff --git a/hermes_cli/auth_commands.py b/hermes_cli/auth_commands.py index c1cf0ff6182b..9c33200107c9 100644 --- a/hermes_cli/auth_commands.py +++ b/hermes_cli/auth_commands.py @@ -4,6 +4,7 @@ from getpass import getpass import math +import sys import time from types import SimpleNamespace import uuid @@ -32,7 +33,7 @@ # Providers that support OAuth login in addition to API keys. -_OAUTH_CAPABLE_PROVIDERS = {"anthropic", "nous", "openai-codex", "qwen-oauth"} +_OAUTH_CAPABLE_PROVIDERS = {"anthropic", "nous", "openai-codex", "qwen-oauth", "google-gemini-cli"} def _get_custom_provider_names() -> list: @@ -147,10 +148,27 @@ def auth_add_command(args) -> None: if provider.startswith(CUSTOM_POOL_PREFIX): requested_type = AUTH_TYPE_API_KEY else: - requested_type = AUTH_TYPE_OAUTH if provider in {"anthropic", "nous", "openai-codex", "qwen-oauth"} else AUTH_TYPE_API_KEY + requested_type = AUTH_TYPE_OAUTH if provider in {"anthropic", "nous", "openai-codex", "qwen-oauth", "google-gemini-cli"} else AUTH_TYPE_API_KEY pool = load_pool(provider) + # Clear ALL suppressions for this provider — re-adding a credential is + # a strong signal the user wants auth re-enabled. This covers env:* + # (shell-exported vars), gh_cli (copilot), claude_code, qwen-cli, + # device_code (codex), etc. One consistent re-engagement pattern. + # Matches the Codex device_code re-link pattern that predates this. + if not provider.startswith(CUSTOM_POOL_PREFIX): + try: + from hermes_cli.auth import ( + _load_auth_store, + unsuppress_credential_source, + ) + suppressed = _load_auth_store().get("suppressed_sources", {}) + for src in list(suppressed.get(provider, []) or []): + unsuppress_credential_source(provider, src) + except Exception: + pass + if requested_type == AUTH_TYPE_API_KEY: token = (getattr(args, "api_key", None) or "").strip() if not token: @@ -160,7 +178,10 @@ def auth_add_command(args) -> None: default_label = _api_key_default_label(len(pool.entries()) + 1) label = (getattr(args, "label", None) or "").strip() if not label: - label = input(f"Label (optional, default: {default_label}): ").strip() or default_label + if sys.stdin.isatty(): + label = input(f"Label (optional, default: {default_label}): ").strip() or default_label + else: + label = default_label entry = PooledCredential( provider=provider, id=uuid.uuid4().hex[:6], @@ -213,22 +234,21 @@ def auth_add_command(args) -> None: ca_bundle=getattr(args, "ca_bundle", None), min_key_ttl_seconds=max(60, int(getattr(args, "min_key_ttl_seconds", 5 * 60))), ) - label = (getattr(args, "label", None) or "").strip() or label_from_token( - creds.get("access_token", ""), - _oauth_default_label(provider, len(pool.entries()) + 1), + # Honor `--label ` so nous matches other providers' UX. The + # helper embeds this into providers.nous so that label_from_token + # doesn't overwrite it on every subsequent load_pool("nous"). + custom_label = (getattr(args, "label", None) or "").strip() or None + entry = auth_mod.persist_nous_credentials(creds, label=custom_label) + shown_label = entry.label if entry is not None else label_from_token( + creds.get("access_token", ""), _oauth_default_label(provider, 1), ) - entry = PooledCredential.from_dict(provider, { - **creds, - "label": label, - "auth_type": AUTH_TYPE_OAUTH, - "source": f"{SOURCE_MANUAL}:device_code", - "base_url": creds.get("inference_base_url"), - }) - pool.add_entry(entry) - print(f'Added {provider} OAuth credential #{len(pool.entries())}: "{entry.label}"') + print(f'Saved {provider} OAuth device-code credentials: "{shown_label}"') return if provider == "openai-codex": + # Clear any existing suppression marker so a re-link after `hermes auth + # remove openai-codex` works without the new tokens being skipped. + auth_mod.unsuppress_credential_source(provider, "device_code") creds = auth_mod._codex_device_code_login() label = (getattr(args, "label", None) or "").strip() or label_from_token( creds["tokens"]["access_token"], @@ -250,6 +270,27 @@ def auth_add_command(args) -> None: print(f'Added {provider} OAuth credential #{len(pool.entries())}: "{entry.label}"') return + if provider == "google-gemini-cli": + from agent.google_oauth import run_gemini_oauth_login_pure + + creds = run_gemini_oauth_login_pure() + label = (getattr(args, "label", None) or "").strip() or ( + creds.get("email") or _oauth_default_label(provider, len(pool.entries()) + 1) + ) + entry = PooledCredential( + provider=provider, + id=uuid.uuid4().hex[:6], + label=label, + auth_type=AUTH_TYPE_OAUTH, + priority=0, + source=f"{SOURCE_MANUAL}:google_pkce", + access_token=creds["access_token"], + refresh_token=creds.get("refresh_token"), + ) + pool.add_entry(entry) + print(f'Added {provider} OAuth credential #{len(pool.entries())}: "{entry.label}"') + return + if provider == "qwen-oauth": creds = auth_mod.resolve_qwen_runtime_credentials(refresh_if_expiring=False) label = (getattr(args, "label", None) or "").strip() or label_from_token( @@ -314,44 +355,28 @@ def auth_remove_command(args) -> None: raise SystemExit(f'No credential matching "{target}" for provider {provider}.') print(f"Removed {provider} credential #{index} ({removed.label})") - # If this was an env-seeded credential, also clear the env var from .env - # so it doesn't get re-seeded on the next load_pool() call. - if removed.source.startswith("env:"): - env_var = removed.source[len("env:"):] - if env_var: - from hermes_cli.config import remove_env_value - cleared = remove_env_value(env_var) - if cleared: - print(f"Cleared {env_var} from .env") - - # If this was a singleton-seeded credential (OAuth device_code, hermes_pkce), - # clear the underlying auth store / credential file so it doesn't get - # re-seeded on the next load_pool() call. - elif removed.source == "device_code" and provider in ("openai-codex", "nous"): - from hermes_cli.auth import ( - _load_auth_store, _save_auth_store, _auth_store_lock, - ) - with _auth_store_lock(): - auth_store = _load_auth_store() - providers_dict = auth_store.get("providers") - if isinstance(providers_dict, dict) and provider in providers_dict: - del providers_dict[provider] - _save_auth_store(auth_store) - print(f"Cleared {provider} OAuth tokens from auth store") - - elif removed.source == "hermes_pkce" and provider == "anthropic": - from hermes_constants import get_hermes_home - oauth_file = get_hermes_home() / ".anthropic_oauth.json" - if oauth_file.exists(): - oauth_file.unlink() - print("Cleared Hermes Anthropic OAuth credentials") - - elif removed.source == "claude_code" and provider == "anthropic": - from hermes_cli.auth import suppress_credential_source - suppress_credential_source(provider, "claude_code") - print("Suppressed claude_code credential — it will not be re-seeded.") - print("Note: Claude Code credentials still live in ~/.claude/.credentials.json") - print("Run `hermes auth add anthropic` to re-enable if needed.") + # Unified removal dispatch. Every credential source Hermes reads from + # (env vars, external OAuth files, auth.json blocks, custom config) + # has a RemovalStep registered in agent.credential_sources. The step + # handles its source-specific cleanup and we centralise suppression + + # user-facing output here so every source behaves identically from + # the user's perspective. + from agent.credential_sources import find_removal_step + from hermes_cli.auth import suppress_credential_source + + step = find_removal_step(provider, removed.source) + if step is None: + # Unregistered source — e.g. "manual", which has nothing external + # to clean up. The pool entry is already gone; we're done. + return + + result = step.remove_fn(provider, removed) + for line in result.cleaned: + print(line) + if result.suppress: + suppress_credential_source(provider, removed.source) + for line in result.hints: + print(line) def auth_reset_command(args) -> None: @@ -368,6 +393,27 @@ def _interactive_auth() -> None: print("=" * 50) auth_list_command(SimpleNamespace(provider=None)) + + # Show AWS Bedrock credential status (not in the pool — uses boto3 chain) + try: + from agent.bedrock_adapter import has_aws_credentials, resolve_aws_auth_env_var, resolve_bedrock_region + if has_aws_credentials(): + auth_source = resolve_aws_auth_env_var() or "unknown" + region = resolve_bedrock_region() + print(f"bedrock (AWS SDK credential chain):") + print(f" Auth: {auth_source}") + print(f" Region: {region}") + try: + import boto3 + sts = boto3.client("sts", region_name=region) + identity = sts.get_caller_identity() + arn = identity.get("Arn", "unknown") + print(f" Identity: {arn}") + except Exception: + print(f" Identity: (could not resolve — boto3 STS call failed)") + print() + except ImportError: + pass # boto3 or bedrock_adapter not available print() # Main menu diff --git a/hermes_cli/backup.py b/hermes_cli/backup.py index 667b8915afd0..8b5b90ef1f93 100644 --- a/hermes_cli/backup.py +++ b/hermes_cli/backup.py @@ -201,7 +201,7 @@ def run_backup(args) -> None: else: zf.write(abs_path, arcname=str(rel_path)) total_bytes += abs_path.stat().st_size - except (PermissionError, OSError) as exc: + except (PermissionError, OSError, ValueError) as exc: errors.append(f" {rel_path}: {exc}") continue diff --git a/hermes_cli/banner.py b/hermes_cli/banner.py index b41ff5578904..fb6068a81b39 100644 --- a/hermes_cli/banner.py +++ b/hermes_cli/banner.py @@ -5,7 +5,6 @@ import json import logging -import os import shutil import subprocess import threading diff --git a/hermes_cli/callbacks.py b/hermes_cli/callbacks.py index 724e6e4c86d6..fa40eced5ede 100644 --- a/hermes_cli/callbacks.py +++ b/hermes_cli/callbacks.py @@ -75,12 +75,12 @@ def prompt_for_secret(cli, var_name: str, prompt: str, metadata=None) -> dict: if not hasattr(cli, "_secret_deadline"): cli._secret_deadline = 0 try: - value = getpass.getpass(f"{prompt} (hidden, Enter to skip): ") + value = getpass.getpass(f"{prompt} (hidden, ESC or empty Enter to skip): ") except (EOFError, KeyboardInterrupt): value = "" if not value: - cprint(f"\n{_DIM} ⏭ Secret entry cancelled{_RST}") + cprint(f"\n{_DIM} ⏭ Secret entry skipped{_RST}") return { "success": True, "reason": "cancelled", @@ -133,7 +133,7 @@ def prompt_for_secret(cli, var_name: str, prompt: str, metadata=None) -> dict: cli._app.invalidate() if not value: - cprint(f"\n{_DIM} ⏭ Secret entry cancelled{_RST}") + cprint(f"\n{_DIM} ⏭ Secret entry skipped{_RST}") return { "success": True, "reason": "cancelled", diff --git a/hermes_cli/claw.py b/hermes_cli/claw.py index e62efe47ea38..aa0c288280c4 100644 --- a/hermes_cli/claw.py +++ b/hermes_cli/claw.py @@ -249,7 +249,7 @@ def _scan_workspace_state(source_dir: Path) -> list[tuple[Path, str]]: state_path = child / state_name if state_path.exists(): kind = "directory" if state_path.is_dir() else "file" - rel = state_path.relative_to(source_dir) + rel = state_path.relative_to(source_dir).as_posix() findings.append((state_path, f"Workspace {kind}: {rel}")) return findings diff --git a/hermes_cli/cli_output.py b/hermes_cli/cli_output.py index 3d454eb30854..2f07129704e8 100644 --- a/hermes_cli/cli_output.py +++ b/hermes_cli/cli_output.py @@ -6,7 +6,6 @@ """ import getpass -import sys from hermes_cli.colors import Colors, color diff --git a/hermes_cli/clipboard.py b/hermes_cli/clipboard.py index fd81ed4c8b97..facc8f3c50ad 100644 --- a/hermes_cli/clipboard.py +++ b/hermes_cli/clipboard.py @@ -7,8 +7,8 @@ Platform support: macOS — osascript (always available), pngpaste (if installed) - Windows — PowerShell via .NET System.Windows.Forms.Clipboard - WSL2 — powershell.exe via .NET System.Windows.Forms.Clipboard + Windows — PowerShell via WinForms, Get-Clipboard, file-drop fallback + WSL2 — powershell.exe via WinForms, Get-Clipboard, file-drop fallback Linux — wl-paste (Wayland), xclip (X11) """ @@ -46,10 +46,11 @@ def has_clipboard_image() -> bool: return _macos_has_image() if sys.platform == "win32": return _windows_has_image() - if _is_wsl(): - return _wsl_has_image() - if os.environ.get("WAYLAND_DISPLAY"): - return _wayland_has_image() + # Match _linux_save fallthrough order: WSL → Wayland → X11 + if _is_wsl() and _wsl_has_image(): + return True + if os.environ.get("WAYLAND_DISPLAY") and _wayland_has_image(): + return True return _xclip_has_image() @@ -135,6 +136,114 @@ def _macos_osascript(dest: Path) -> bool: "[System.Convert]::ToBase64String($ms.ToArray())" ) +_PS_CHECK_IMAGE_GET_CLIPBOARD = ( + "try { " + "$img = Get-Clipboard -Format Image -ErrorAction Stop;" + "if ($null -ne $img) { 'True' } else { 'False' }" + "} catch { 'False' }" +) + +_PS_EXTRACT_IMAGE_GET_CLIPBOARD = ( + "try { " + "Add-Type -AssemblyName System.Drawing;" + "Add-Type -AssemblyName PresentationCore;" + "Add-Type -AssemblyName WindowsBase;" + "$img = Get-Clipboard -Format Image -ErrorAction Stop;" + "if ($null -eq $img) { exit 1 }" + "$ms = New-Object System.IO.MemoryStream;" + "if ($img -is [System.Drawing.Image]) {" + "$img.Save($ms, [System.Drawing.Imaging.ImageFormat]::Png)" + "} elseif ($img -is [System.Windows.Media.Imaging.BitmapSource]) {" + "$enc = New-Object System.Windows.Media.Imaging.PngBitmapEncoder;" + "$enc.Frames.Add([System.Windows.Media.Imaging.BitmapFrame]::Create($img));" + "$enc.Save($ms)" + "} else { exit 2 }" + "[System.Convert]::ToBase64String($ms.ToArray())" + "} catch { exit 1 }" +) + +_FILEDROP_IMAGE_EXTS = "'.png','.jpg','.jpeg','.gif','.webp','.bmp','.tiff','.tif'" + +_PS_CHECK_FILEDROP_IMAGE = ( + "try { " + "$files = Get-Clipboard -Format FileDropList -ErrorAction Stop;" + f"$exts = @({_FILEDROP_IMAGE_EXTS});" + "$hit = $files | Where-Object { $exts -contains ([System.IO.Path]::GetExtension($_).ToLowerInvariant()) } | Select-Object -First 1;" + "if ($null -ne $hit) { 'True' } else { 'False' }" + "} catch { 'False' }" +) + +_PS_EXTRACT_FILEDROP_IMAGE = ( + "try { " + "$files = Get-Clipboard -Format FileDropList -ErrorAction Stop;" + f"$exts = @({_FILEDROP_IMAGE_EXTS});" + "$hit = $files | Where-Object { $exts -contains ([System.IO.Path]::GetExtension($_).ToLowerInvariant()) } | Select-Object -First 1;" + "if ($null -eq $hit) { exit 1 }" + "[System.Convert]::ToBase64String([System.IO.File]::ReadAllBytes($hit))" + "} catch { exit 1 }" +) + +_POWERSHELL_HAS_IMAGE_SCRIPTS = ( + _PS_CHECK_IMAGE, + _PS_CHECK_IMAGE_GET_CLIPBOARD, + _PS_CHECK_FILEDROP_IMAGE, +) + +_POWERSHELL_EXTRACT_IMAGE_SCRIPTS = ( + _PS_EXTRACT_IMAGE, + _PS_EXTRACT_IMAGE_GET_CLIPBOARD, + _PS_EXTRACT_FILEDROP_IMAGE, +) + + +def _run_powershell(exe: str, script: str, timeout: int) -> subprocess.CompletedProcess: + return subprocess.run( + [exe, "-NoProfile", "-NonInteractive", "-Command", script], + capture_output=True, text=True, timeout=timeout, + ) + + +def _write_base64_image(dest: Path, b64_data: str) -> bool: + image_bytes = base64.b64decode(b64_data, validate=True) + dest.write_bytes(image_bytes) + return dest.exists() and dest.stat().st_size > 0 + + +def _powershell_has_image(exe: str, *, timeout: int, label: str) -> bool: + for script in _POWERSHELL_HAS_IMAGE_SCRIPTS: + try: + r = _run_powershell(exe, script, timeout=timeout) + if r.returncode == 0 and "True" in r.stdout: + return True + except FileNotFoundError: + logger.debug("%s not found — clipboard unavailable", exe) + return False + except Exception as e: + logger.debug("%s clipboard image check failed: %s", label, e) + return False + + +def _powershell_save_image(exe: str, dest: Path, *, timeout: int, label: str) -> bool: + for script in _POWERSHELL_EXTRACT_IMAGE_SCRIPTS: + try: + r = _run_powershell(exe, script, timeout=timeout) + if r.returncode != 0: + continue + + b64_data = r.stdout.strip() + if not b64_data: + continue + + if _write_base64_image(dest, b64_data): + return True + except FileNotFoundError: + logger.debug("%s not found — clipboard unavailable", exe) + return False + except Exception as e: + logger.debug("%s clipboard image extraction failed: %s", label, e) + dest.unlink(missing_ok=True) + return False + # ── Native Windows ──────────────────────────────────────────────────────── @@ -175,15 +284,7 @@ def _windows_has_image() -> bool: ps = _get_ps_exe() if ps is None: return False - try: - r = subprocess.run( - [ps, "-NoProfile", "-NonInteractive", "-Command", _PS_CHECK_IMAGE], - capture_output=True, text=True, timeout=5, - ) - return r.returncode == 0 and "True" in r.stdout - except Exception as e: - logger.debug("Windows clipboard image check failed: %s", e) - return False + return _powershell_has_image(ps, timeout=5, label="Windows") def _windows_save(dest: Path) -> bool: @@ -192,26 +293,7 @@ def _windows_save(dest: Path) -> bool: if ps is None: logger.debug("No PowerShell found — Windows clipboard image paste unavailable") return False - try: - r = subprocess.run( - [ps, "-NoProfile", "-NonInteractive", "-Command", _PS_EXTRACT_IMAGE], - capture_output=True, text=True, timeout=15, - ) - if r.returncode != 0: - return False - - b64_data = r.stdout.strip() - if not b64_data: - return False - - png_bytes = base64.b64decode(b64_data) - dest.write_bytes(png_bytes) - return dest.exists() and dest.stat().st_size > 0 - - except Exception as e: - logger.debug("Windows clipboard image extraction failed: %s", e) - dest.unlink(missing_ok=True) - return False + return _powershell_save_image(ps, dest, timeout=15, label="Windows") # ── Linux ──────────────────────────────────────────────────────────────── @@ -235,45 +317,12 @@ def _linux_save(dest: Path) -> bool: def _wsl_has_image() -> bool: """Check if Windows clipboard has an image (via powershell.exe).""" - try: - r = subprocess.run( - ["powershell.exe", "-NoProfile", "-NonInteractive", "-Command", - _PS_CHECK_IMAGE], - capture_output=True, text=True, timeout=8, - ) - return r.returncode == 0 and "True" in r.stdout - except FileNotFoundError: - logger.debug("powershell.exe not found — WSL clipboard unavailable") - except Exception as e: - logger.debug("WSL clipboard check failed: %s", e) - return False + return _powershell_has_image("powershell.exe", timeout=8, label="WSL") def _wsl_save(dest: Path) -> bool: """Extract clipboard image via powershell.exe → base64 → decode to PNG.""" - try: - r = subprocess.run( - ["powershell.exe", "-NoProfile", "-NonInteractive", "-Command", - _PS_EXTRACT_IMAGE], - capture_output=True, text=True, timeout=15, - ) - if r.returncode != 0: - return False - - b64_data = r.stdout.strip() - if not b64_data: - return False - - png_bytes = base64.b64decode(b64_data) - dest.write_bytes(png_bytes) - return dest.exists() and dest.stat().st_size > 0 - - except FileNotFoundError: - logger.debug("powershell.exe not found — WSL clipboard unavailable") - except Exception as e: - logger.debug("WSL clipboard extraction failed: %s", e) - dest.unlink(missing_ok=True) - return False + return _powershell_save_image("powershell.exe", dest, timeout=15, label="WSL") # ── Wayland (wl-paste) ────────────────────────────────────────────────── diff --git a/hermes_cli/codex_models.py b/hermes_cli/codex_models.py index f5616b68d680..e39b2c5943b7 100644 --- a/hermes_cli/codex_models.py +++ b/hermes_cli/codex_models.py @@ -12,6 +12,7 @@ logger = logging.getLogger(__name__) DEFAULT_CODEX_MODELS: List[str] = [ + "gpt-5.5", "gpt-5.4-mini", "gpt-5.4", "gpt-5.3-codex", @@ -21,10 +22,10 @@ ] _FORWARD_COMPAT_TEMPLATE_MODELS: List[tuple[str, tuple[str, ...]]] = [ + ("gpt-5.5", ("gpt-5.4", "gpt-5.4-mini", "gpt-5.3-codex")), ("gpt-5.4-mini", ("gpt-5.3-codex", "gpt-5.2-codex")), ("gpt-5.4", ("gpt-5.3-codex", "gpt-5.2-codex")), ("gpt-5.3-codex", ("gpt-5.2-codex",)), - ("gpt-5.3-codex-spark", ("gpt-5.3-codex", "gpt-5.2-codex")), ] diff --git a/hermes_cli/commands.py b/hermes_cli/commands.py index fedeef294492..87d73af58eca 100644 --- a/hermes_cli/commands.py +++ b/hermes_cli/commands.py @@ -12,6 +12,9 @@ import os import re +import shutil +import subprocess +import time from collections.abc import Callable, Mapping from dataclasses import dataclass from typing import Any @@ -84,8 +87,12 @@ class CommandDef: aliases=("bg",), args_hint=""), CommandDef("btw", "Ephemeral side question using session context (no tools, not persisted)", "Session", args_hint=""), + CommandDef("agents", "Show active agents and running tasks", "Session", + aliases=("tasks",)), CommandDef("queue", "Queue a prompt for the next turn (doesn't interrupt)", "Session", aliases=("q",), args_hint=""), + CommandDef("steer", "Inject a message after the next tool call without interrupting", "Session", + args_hint=""), CommandDef("status", "Show session info", "Session"), CommandDef("profile", "Show active profile name and home directory", "Info"), CommandDef("sethome", "Set this chat as the home channel", "Session", @@ -96,9 +103,10 @@ class CommandDef: # Configuration CommandDef("config", "Show current configuration", "Configuration", cli_only=True), - CommandDef("model", "Switch model for this session", "Configuration", args_hint="[model] [--global]"), + CommandDef("model", "Switch model for this session", "Configuration", args_hint="[model] [--provider name] [--global]"), CommandDef("provider", "Show available providers and current provider", "Configuration"), + CommandDef("gquota", "Show Google Gemini Code Assist quota usage", "Info"), CommandDef("personality", "Set a predefined personality", "Configuration", args_hint="[name]"), @@ -116,7 +124,7 @@ class CommandDef: args_hint="[normal|fast|status]", subcommands=("normal", "fast", "status", "on", "off")), CommandDef("skin", "Show or change the display skin/theme", "Configuration", - cli_only=True, args_hint="[name]"), + args_hint="[name]"), CommandDef("voice", "Toggle voice mode", "Configuration", args_hint="[on|off|tts|status]", subcommands=("on", "off", "tts", "status")), @@ -151,7 +159,9 @@ class CommandDef: args_hint="[days]"), CommandDef("platforms", "Show gateway/messaging platform status", "Info", cli_only=True, aliases=("gateway",)), - CommandDef("paste", "Check clipboard for an image and attach it", "Info", + CommandDef("copy", "Copy the last assistant response to clipboard", "Info", + cli_only=True, args_hint="[number]"), + CommandDef("paste", "Attach clipboard image from your clipboard", "Info", cli_only=True), CommandDef("image", "Attach a local image file for your next prompt", "Info", cli_only=True, args_hint=""), @@ -161,7 +171,7 @@ class CommandDef: # Exit CommandDef("quit", "Exit the CLI", "Exit", - cli_only=True, aliases=("exit", "q")), + cli_only=True, aliases=("exit",)), ] @@ -190,52 +200,6 @@ def resolve_command(name: str) -> CommandDef | None: return _COMMAND_LOOKUP.get(name.lower().lstrip("/")) -def rebuild_lookups() -> None: - """Rebuild all derived lookup dicts from the current COMMAND_REGISTRY. - - Called after plugin commands are registered so they appear in help, - autocomplete, gateway dispatch, Telegram menu, and Slack mapping. - """ - global GATEWAY_KNOWN_COMMANDS - - _COMMAND_LOOKUP.clear() - _COMMAND_LOOKUP.update(_build_command_lookup()) - - COMMANDS.clear() - for cmd in COMMAND_REGISTRY: - if not cmd.gateway_only: - COMMANDS[f"/{cmd.name}"] = _build_description(cmd) - for alias in cmd.aliases: - COMMANDS[f"/{alias}"] = f"{cmd.description} (alias for /{cmd.name})" - - COMMANDS_BY_CATEGORY.clear() - for cmd in COMMAND_REGISTRY: - if not cmd.gateway_only: - cat = COMMANDS_BY_CATEGORY.setdefault(cmd.category, {}) - cat[f"/{cmd.name}"] = COMMANDS[f"/{cmd.name}"] - for alias in cmd.aliases: - cat[f"/{alias}"] = COMMANDS[f"/{alias}"] - - SUBCOMMANDS.clear() - for cmd in COMMAND_REGISTRY: - if cmd.subcommands: - SUBCOMMANDS[f"/{cmd.name}"] = list(cmd.subcommands) - for cmd in COMMAND_REGISTRY: - key = f"/{cmd.name}" - if key in SUBCOMMANDS or not cmd.args_hint: - continue - m = _PIPE_SUBS_RE.search(cmd.args_hint) - if m: - SUBCOMMANDS[key] = m.group(0).split("|") - - GATEWAY_KNOWN_COMMANDS = frozenset( - name - for cmd in COMMAND_REGISTRY - if not cmd.cli_only or cmd.gateway_config_gate - for name in (cmd.name, *cmd.aliases) - ) - - def _build_description(cmd: CommandDef) -> str: """Build a CLI-facing description string including usage hint.""" if cmd.args_hint: @@ -296,6 +260,73 @@ def _build_description(cmd: CommandDef) -> str: ) +def is_gateway_known_command(name: str | None) -> bool: + """Return True if ``name`` resolves to a gateway-dispatchable slash command. + + This covers both built-in commands (``GATEWAY_KNOWN_COMMANDS`` derived + from ``COMMAND_REGISTRY``) and plugin-registered commands, which are + looked up lazily so importing this module never forces plugin + discovery. Gateway code uses this to decide whether to emit + ``command:`` hooks — plugin commands get the same lifecycle + events as built-ins. + """ + if not name: + return False + if name in GATEWAY_KNOWN_COMMANDS: + return True + for plugin_name, _description, _args_hint in _iter_plugin_command_entries(): + if plugin_name == name: + return True + return False + + +# Commands with explicit Level-2 running-agent handlers in gateway/run.py. +# Listed here for introspection / tests; semantically a subset of +# "all resolvable commands" — which is the real bypass set (see +# should_bypass_active_session below). +ACTIVE_SESSION_BYPASS_COMMANDS: frozenset[str] = frozenset( + { + "agents", + "approve", + "background", + "commands", + "deny", + "help", + "new", + "profile", + "queue", + "restart", + "status", + "steer", + "stop", + "update", + } +) + + +def should_bypass_active_session(command_name: str | None) -> bool: + """Return True for any resolvable slash command. + + Rationale: every gateway-registered slash command either has a + specific Level-2 handler in gateway/run.py (/stop, /new, /model, + /approve, etc.) or reaches the running-agent catch-all that returns + a "busy — wait or /stop first" response. In both paths the command + is dispatched, not queued. + + Queueing is always wrong for a recognized slash command because the + safety net in gateway.run discards any command text that reaches + the pending queue — which meant a mid-run /model (or /reasoning, + /voice, /insights, /title, /resume, /retry, /undo, /compress, + /usage, /provider, /reload-mcp, /sethome, /reset) would silently + interrupt the agent AND get discarded, producing a zero-char + response. See issue #5057 / PRs #6252, #10370, #4665. + + ACTIVE_SESSION_BYPASS_COMMANDS remains the subset of commands with + explicit Level-2 handlers; the rest fall through to the catch-all. + """ + return resolve_command(command_name) is not None if command_name else False + + def _resolve_config_gates() -> set[str]: """Return canonical names of commands whose ``gateway_config_gate`` is truthy. @@ -360,12 +391,47 @@ def gateway_help_lines() -> list[str]: return lines +def _iter_plugin_command_entries() -> list[tuple[str, str, str]]: + """Yield (name, description, args_hint) tuples for all plugin slash commands. + + Plugin commands are registered via + :func:`hermes_cli.plugins.PluginContext.register_command`. They behave + like ``CommandDef`` entries for gateway surfacing: they appear in the + Telegram command menu, in Slack's ``/hermes`` subcommand mapping, and + (via :func:`gateway.platforms.discord._register_slash_commands`) in + Discord's native slash command picker. + + Lookup is lazy so importing this module never forces plugin discovery + (which can trigger filesystem scans and environment-dependent + behavior). + """ + try: + from hermes_cli.plugins import get_plugin_commands + except Exception: + return [] + try: + commands = get_plugin_commands() or {} + except Exception: + return [] + entries: list[tuple[str, str, str]] = [] + for name, meta in commands.items(): + if not isinstance(name, str) or not isinstance(meta, dict): + continue + description = str(meta.get("description") or f"Run /{name}") + args_hint = str(meta.get("args_hint") or "").strip() + entries.append((name, description, args_hint)) + return entries + + def telegram_bot_commands() -> list[tuple[str, str]]: """Return (command_name, description) pairs for Telegram setMyCommands. Telegram command names cannot contain hyphens, so they are replaced with underscores. Aliases are skipped -- Telegram shows one menu entry per canonical command. + + Plugin-registered slash commands are included so plugins get native + autocomplete in Telegram without touching core code. """ overrides = _resolve_config_gates() result: list[tuple[str, str]] = [] @@ -375,6 +441,10 @@ def telegram_bot_commands() -> list[tuple[str, str]]: tg_name = _sanitize_telegram_name(cmd.name) if tg_name: result.append((tg_name, cmd.description)) + for name, description, _args_hint in _iter_plugin_command_entries(): + tg_name = _sanitize_telegram_name(name) + if tg_name: + result.append((tg_name, description)) return result @@ -486,14 +556,13 @@ def _collect_gateway_skill_entries( # --- Tier 1: Plugin slash commands (never trimmed) --------------------- plugin_pairs: list[tuple[str, str]] = [] try: - from hermes_cli.plugins import get_plugin_manager - pm = get_plugin_manager() - plugin_cmds = getattr(pm, "_plugin_commands", {}) + from hermes_cli.plugins import get_plugin_commands + plugin_cmds = get_plugin_commands() for cmd_name in sorted(plugin_cmds): name = sanitize_name(cmd_name) if sanitize_name else cmd_name if not name: continue - desc = "Plugin command" + desc = plugin_cmds[cmd_name].get("description", "Plugin command") if len(desc) > desc_limit: desc = desc[:desc_limit - 3] + "..." plugin_pairs.append((name, desc)) @@ -625,11 +694,124 @@ def discord_skill_commands( ) +def discord_skill_commands_by_category( + reserved_names: set[str], +) -> tuple[dict[str, list[tuple[str, str, str]]], list[tuple[str, str, str]], int]: + """Return skill entries organized by category for Discord ``/skill`` subcommand groups. + + Skills whose directory is nested at least 2 levels under ``SKILLS_DIR`` + (e.g. ``creative/ascii-art/SKILL.md``) are grouped by their top-level + category. Root-level skills (e.g. ``dogfood/SKILL.md``) are returned as + *uncategorized* — the caller should register them as direct subcommands + of the ``/skill`` group. + + The same filtering as :func:`discord_skill_commands` is applied: hub + skills excluded, per-platform disabled excluded, names clamped. + + Returns: + ``(categories, uncategorized, hidden_count)`` + + - *categories*: ``{category_name: [(name, description, cmd_key), ...]}`` + - *uncategorized*: ``[(name, description, cmd_key), ...]`` + - *hidden_count*: skills dropped due to Discord group limits + (25 subcommand groups, 25 subcommands per group) + """ + from pathlib import Path as _P + + _platform_disabled: set[str] = set() + try: + from agent.skill_utils import get_disabled_skill_names + _platform_disabled = get_disabled_skill_names(platform="discord") + except Exception: + pass + + # Collect raw skill data -------------------------------------------------- + categories: dict[str, list[tuple[str, str, str]]] = {} + uncategorized: list[tuple[str, str, str]] = [] + _names_used: set[str] = set(reserved_names) + hidden = 0 + + try: + from agent.skill_commands import get_skill_commands + from tools.skills_tool import SKILLS_DIR + _skills_dir = SKILLS_DIR.resolve() + _hub_dir = (SKILLS_DIR / ".hub").resolve() + skill_cmds = get_skill_commands() + + for cmd_key in sorted(skill_cmds): + info = skill_cmds[cmd_key] + skill_path = info.get("skill_md_path", "") + if not skill_path: + continue + sp = _P(skill_path).resolve() + # Skip skills outside SKILLS_DIR or from the hub + if not str(sp).startswith(str(_skills_dir)): + continue + if str(sp).startswith(str(_hub_dir)): + continue + + skill_name = info.get("name", "") + if skill_name in _platform_disabled: + continue + + raw_name = cmd_key.lstrip("/") + # Clamp to 32 chars (Discord limit) + discord_name = raw_name[:32] + if discord_name in _names_used: + continue + _names_used.add(discord_name) + + desc = info.get("description", "") + if len(desc) > 100: + desc = desc[:97] + "..." + + # Determine category from the relative path within SKILLS_DIR. + # e.g. creative/ascii-art/SKILL.md → parts = ("creative", "ascii-art") + try: + rel = sp.parent.relative_to(_skills_dir) + except ValueError: + continue + parts = rel.parts + if len(parts) >= 2: + cat = parts[0] + categories.setdefault(cat, []).append((discord_name, desc, cmd_key)) + else: + uncategorized.append((discord_name, desc, cmd_key)) + except Exception: + pass + + # Enforce Discord limits: 25 subcommand groups, 25 subcommands each ------ + _MAX_GROUPS = 25 + _MAX_PER_GROUP = 25 + + trimmed_categories: dict[str, list[tuple[str, str, str]]] = {} + group_count = 0 + for cat in sorted(categories): + if group_count >= _MAX_GROUPS: + hidden += len(categories[cat]) + continue + entries = categories[cat][:_MAX_PER_GROUP] + hidden += max(0, len(categories[cat]) - _MAX_PER_GROUP) + trimmed_categories[cat] = entries + group_count += 1 + + # Uncategorized skills also count against the 25 top-level limit + remaining_slots = _MAX_GROUPS - group_count + if len(uncategorized) > remaining_slots: + hidden += len(uncategorized) - remaining_slots + uncategorized = uncategorized[:remaining_slots] + + return trimmed_categories, uncategorized, hidden + + def slack_subcommand_map() -> dict[str, str]: """Return subcommand -> /command mapping for Slack /hermes handler. Maps both canonical names and aliases so /hermes bg do stuff works the same as /hermes background do stuff. + + Plugin-registered slash commands are included so ``/hermes `` + routes through the plugin handler. """ overrides = _resolve_config_gates() mapping: dict[str, str] = {} @@ -639,6 +821,9 @@ def slack_subcommand_map() -> dict[str, str]: mapping[cmd.name] = f"/{cmd.name}" for alias in cmd.aliases: mapping[alias] = f"/{alias}" + for name, _description, _args_hint in _iter_plugin_command_entries(): + if name not in mapping: + mapping[name] = f"/{name}" return mapping @@ -656,6 +841,10 @@ def __init__( ) -> None: self._skill_commands_provider = skill_commands_provider self._command_filter = command_filter + # Cached project file list for fuzzy @ completions + self._file_cache: list[str] = [] + self._file_cache_time: float = 0.0 + self._file_cache_cwd: str = "" def _command_allowed(self, slash_command: str) -> bool: if self._command_filter is None: @@ -773,8 +962,7 @@ def _extract_context_word(text: str) -> str | None: return None return word - @staticmethod - def _context_completions(word: str, limit: int = 30): + def _context_completions(self, word: str, limit: int = 30): """Yield Claude Code-style @ context completions. Bare ``@`` or ``@partial`` shows static references and matching @@ -801,12 +989,22 @@ def _context_completions(word: str, limit: int = 30): display_meta=meta, ) - # If the user typed @file: or @folder:, delegate to path completions + # If the user typed @file: / @folder: (or just @file / @folder with + # no colon yet), delegate to path completions. Accepting the bare + # form lets the picker surface directories as soon as the user has + # typed `@folder`, without requiring them to first accept the static + # `@folder:` hint and re-trigger completion. for prefix in ("@file:", "@folder:"): - if word.startswith(prefix): - path_part = word[len(prefix):] or "." + bare = prefix[:-1] + + if word == bare or word.startswith(prefix): + want_dir = prefix == "@folder:" + path_part = '' if word == bare else word[len(prefix):] expanded = os.path.expanduser(path_part) - if expanded.endswith("/"): + + if not expanded or expanded == ".": + search_dir, match_prefix = ".", "" + elif expanded.endswith("/"): search_dir, match_prefix = expanded, "" else: search_dir = os.path.dirname(expanded) or "." @@ -822,15 +1020,21 @@ def _context_completions(word: str, limit: int = 30): for entry in sorted(entries): if match_prefix and not entry.lower().startswith(prefix_lower): continue - if count >= limit: - break full_path = os.path.join(search_dir, entry) is_dir = os.path.isdir(full_path) + # `@folder:` must only surface directories; `@file:` only + # regular files. Without this filter `@folder:` listed + # every .env / .gitignore in the cwd, defeating the + # explicit prefix and confusing users expecting a + # directory picker. + if want_dir != is_dir: + continue + if count >= limit: + break display_path = os.path.relpath(full_path) suffix = "/" if is_dir else "" - kind = "folder" if is_dir else "file" meta = "dir" if is_dir else _file_size_label(full_path) - completion = f"@{kind}:{display_path}{suffix}" + completion = f"{prefix}{display_path}{suffix}" yield Completion( completion, start_position=-len(word), @@ -840,46 +1044,183 @@ def _context_completions(word: str, limit: int = 30): count += 1 return - # Bare @ or @partial — show matching files/folders from cwd + # Bare @ or @partial — fuzzy project-wide file search query = word[1:] # strip the @ + yield from self._fuzzy_file_completions(word, query, limit) + + def _get_project_files(self) -> list[str]: + """Return cached list of project files (refreshed every 5s).""" + cwd = os.getcwd() + now = time.monotonic() + if ( + self._file_cache + and self._file_cache_cwd == cwd + and now - self._file_cache_time < 5.0 + ): + return self._file_cache + + files: list[str] = [] + # Try rg first (fast, respects .gitignore), then fd, then find. + for cmd in [ + ["rg", "--files", "--sortr=modified", cwd], + ["rg", "--files", cwd], + ["fd", "--type", "f", "--base-directory", cwd], + ]: + tool = cmd[0] + if not shutil.which(tool): + continue + try: + proc = subprocess.run( + cmd, capture_output=True, text=True, timeout=2, + cwd=cwd, + ) + if proc.returncode == 0 and proc.stdout.strip(): + raw = proc.stdout.strip().split("\n") + # Store relative paths + for p in raw[:5000]: + rel = os.path.relpath(p, cwd) if os.path.isabs(p) else p + files.append(rel) + break + except (subprocess.TimeoutExpired, OSError): + continue + + self._file_cache = files + self._file_cache_time = now + self._file_cache_cwd = cwd + return files + + @staticmethod + def _score_path(filepath: str, query: str) -> int: + """Score a file path against a fuzzy query. Higher = better match.""" if not query: - search_dir, match_prefix = ".", "" - else: - expanded = os.path.expanduser(query) - if expanded.endswith("/"): - search_dir, match_prefix = expanded, "" - else: - search_dir = os.path.dirname(expanded) or "." - match_prefix = os.path.basename(expanded) + return 1 # show everything when query is empty + + filename = os.path.basename(filepath) + lower_file = filename.lower() + lower_path = filepath.lower() + lower_q = query.lower() + + # Exact filename match + if lower_file == lower_q: + return 100 + # Filename starts with query + if lower_file.startswith(lower_q): + return 80 + # Filename contains query as substring + if lower_q in lower_file: + return 60 + # Full path contains query + if lower_q in lower_path: + return 40 + # Initials / abbreviation match: e.g. "fo" matches "file_operations" + # Check if query chars appear in order in filename + qi = 0 + for c in lower_file: + if qi < len(lower_q) and c == lower_q[qi]: + qi += 1 + if qi == len(lower_q): + # Bonus if matches land on word boundaries (after _, -, /, .) + boundary_hits = 0 + qi = 0 + prev = "_" # treat start as boundary + for c in lower_file: + if qi < len(lower_q) and c == lower_q[qi]: + if prev in "_-./": + boundary_hits += 1 + qi += 1 + prev = c + if boundary_hits >= len(lower_q) * 0.5: + return 35 + return 25 + return 0 + + def _fuzzy_file_completions(self, word: str, query: str, limit: int = 20): + """Yield fuzzy file completions for bare @query.""" + files = self._get_project_files() - try: - entries = os.listdir(search_dir) - except OSError: + if not query: + # No query — show recently modified files (already sorted by mtime) + for fp in files[:limit]: + is_dir = fp.endswith("/") + filename = os.path.basename(fp) + kind = "folder" if is_dir else "file" + meta = "dir" if is_dir else _file_size_label( + os.path.join(os.getcwd(), fp) + ) + yield Completion( + f"@{kind}:{fp}", + start_position=-len(word), + display=filename, + display_meta=meta, + ) return - count = 0 - prefix_lower = match_prefix.lower() - for entry in sorted(entries): - if match_prefix and not entry.lower().startswith(prefix_lower): - continue - if entry.startswith("."): - continue # skip hidden files in bare @ mode - if count >= limit: - break - full_path = os.path.join(search_dir, entry) - is_dir = os.path.isdir(full_path) - display_path = os.path.relpath(full_path) - suffix = "/" if is_dir else "" + # Score and rank + scored = [] + for fp in files: + s = self._score_path(fp, query) + if s > 0: + scored.append((s, fp)) + scored.sort(key=lambda x: (-x[0], x[1])) + + for _, fp in scored[:limit]: + is_dir = fp.endswith("/") + filename = os.path.basename(fp) kind = "folder" if is_dir else "file" - meta = "dir" if is_dir else _file_size_label(full_path) - completion = f"@{kind}:{display_path}{suffix}" + meta = "dir" if is_dir else _file_size_label( + os.path.join(os.getcwd(), fp) + ) yield Completion( - completion, + f"@{kind}:{fp}", start_position=-len(word), - display=entry + suffix, - display_meta=meta, + display=filename, + display_meta=f"{fp} {meta}" if meta else fp, ) - count += 1 + + @staticmethod + def _skin_completions(sub_text: str, sub_lower: str): + """Yield completions for /skin from available skins.""" + try: + from hermes_cli.skin_engine import list_skins + for s in list_skins(): + name = s["name"] + if name.startswith(sub_lower) and name != sub_lower: + yield Completion( + name, + start_position=-len(sub_text), + display=name, + display_meta=s.get("description", "") or s.get("source", ""), + ) + except Exception: + pass + + @staticmethod + def _personality_completions(sub_text: str, sub_lower: str): + """Yield completions for /personality from configured personalities.""" + try: + from hermes_cli.config import load_config + personalities = load_config().get("agent", {}).get("personalities", {}) + if "none".startswith(sub_lower) and "none" != sub_lower: + yield Completion( + "none", + start_position=-len(sub_text), + display="none", + display_meta="clear personality overlay", + ) + for name, prompt in personalities.items(): + if name.startswith(sub_lower) and name != sub_lower: + if isinstance(prompt, dict): + meta = prompt.get("description") or prompt.get("system_prompt", "")[:50] + else: + meta = str(prompt)[:50] + yield Completion( + name, + start_position=-len(sub_text), + display=name, + display_meta=meta, + ) + except Exception: + pass def _model_completions(self, sub_text: str, sub_lower: str): """Yield completions for /model from config aliases + built-in aliases.""" @@ -935,10 +1276,17 @@ def get_completions(self, document, complete_event): sub_text = parts[1] if len(parts) > 1 else "" sub_lower = sub_text.lower() - # Dynamic model alias completions for /model - if " " not in sub_text and base_cmd == "/model": - yield from self._model_completions(sub_text, sub_lower) - return + # Dynamic completions for commands with runtime lists + if " " not in sub_text: + if base_cmd == "/model": + yield from self._model_completions(sub_text, sub_lower) + return + if base_cmd == "/skin": + yield from self._skin_completions(sub_text, sub_lower) + return + if base_cmd == "/personality": + yield from self._personality_completions(sub_text, sub_lower) + return # Static subcommand completions if " " not in sub_text and base_cmd in SUBCOMMANDS and self._command_allowed(base_cmd): @@ -977,6 +1325,22 @@ def get_completions(self, document, complete_event): display_meta=f"⚡ {short_desc}", ) + # Plugin-registered slash commands + try: + from hermes_cli.plugins import get_plugin_commands + for cmd_name, cmd_info in get_plugin_commands().items(): + if cmd_name.startswith(word): + desc = str(cmd_info.get("description", "Plugin command")) + short_desc = desc[:50] + ("..." if len(desc) > 50 else "") + yield Completion( + self._completion_text(cmd_name, word), + start_position=-len(word), + display=f"/{cmd_name}", + display_meta=f"🔌 {short_desc}", + ) + except Exception: + pass + # --------------------------------------------------------------------------- # Inline auto-suggest (ghost text) for slash commands diff --git a/hermes_cli/completion.py b/hermes_cli/completion.py new file mode 100644 index 000000000000..18de08cc9012 --- /dev/null +++ b/hermes_cli/completion.py @@ -0,0 +1,315 @@ +"""Shell completion script generation for hermes CLI. + +Walks the live argparse parser tree to generate accurate, always-up-to-date +completion scripts — no hardcoded subcommand lists, no extra dependencies. + +Supports bash, zsh, and fish. +""" + +from __future__ import annotations + +import argparse +from typing import Any + + +def _walk(parser: argparse.ArgumentParser) -> dict[str, Any]: + """Recursively extract subcommands and flags from a parser. + + Uses _SubParsersAction._choices_actions to get canonical names (no aliases) + along with their help text. + """ + flags: list[str] = [] + subcommands: dict[str, Any] = {} + + for action in parser._actions: + if isinstance(action, argparse._SubParsersAction): + # _choices_actions has one entry per canonical name; aliases are + # omitted, which keeps completion lists clean. + seen: set[str] = set() + for pseudo in action._choices_actions: + name = pseudo.dest + if name in seen: + continue + seen.add(name) + subparser = action.choices.get(name) + if subparser is None: + continue + info = _walk(subparser) + info["help"] = _clean(pseudo.help or "") + subcommands[name] = info + elif action.option_strings: + flags.extend(o for o in action.option_strings if o.startswith("-")) + + return {"flags": flags, "subcommands": subcommands} + + +def _clean(text: str, maxlen: int = 60) -> str: + """Strip shell-unsafe characters and truncate.""" + return text.replace("'", "").replace('"', "").replace("\\", "")[:maxlen] + + +# --------------------------------------------------------------------------- +# Bash +# --------------------------------------------------------------------------- + +def generate_bash(parser: argparse.ArgumentParser) -> str: + tree = _walk(parser) + top_cmds = " ".join(sorted(tree["subcommands"])) + + cases: list[str] = [] + for cmd in sorted(tree["subcommands"]): + info = tree["subcommands"][cmd] + if cmd == "profile" and info["subcommands"]: + # Profile subcommand: complete actions, then profile names for + # actions that accept a profile argument. + subcmds = " ".join(sorted(info["subcommands"])) + profile_actions = "use delete show alias rename export" + cases.append( + f" profile)\n" + f" case \"$prev\" in\n" + f" profile)\n" + f" COMPREPLY=($(compgen -W \"{subcmds}\" -- \"$cur\"))\n" + f" return\n" + f" ;;\n" + f" {profile_actions.replace(' ', '|')})\n" + f" COMPREPLY=($(compgen -W \"$(_hermes_profiles)\" -- \"$cur\"))\n" + f" return\n" + f" ;;\n" + f" esac\n" + f" ;;" + ) + elif info["subcommands"]: + subcmds = " ".join(sorted(info["subcommands"])) + cases.append( + f" {cmd})\n" + f" COMPREPLY=($(compgen -W \"{subcmds}\" -- \"$cur\"))\n" + f" return\n" + f" ;;" + ) + elif info["flags"]: + flags = " ".join(info["flags"]) + cases.append( + f" {cmd})\n" + f" COMPREPLY=($(compgen -W \"{flags}\" -- \"$cur\"))\n" + f" return\n" + f" ;;" + ) + + cases_str = "\n".join(cases) + + return f"""# Hermes Agent bash completion +# Add to ~/.bashrc: +# eval "$(hermes completion bash)" + +_hermes_profiles() {{ + local profiles_dir="$HOME/.hermes/profiles" + local profiles="default" + if [ -d "$profiles_dir" ]; then + profiles="$profiles $(ls "$profiles_dir" 2>/dev/null)" + fi + echo "$profiles" +}} + +_hermes_completion() {{ + local cur prev + COMPREPLY=() + cur="${{COMP_WORDS[COMP_CWORD]}}" + prev="${{COMP_WORDS[COMP_CWORD-1]}}" + + # Complete profile names after -p / --profile + if [[ "$prev" == "-p" || "$prev" == "--profile" ]]; then + COMPREPLY=($(compgen -W "$(_hermes_profiles)" -- "$cur")) + return + fi + + if [[ $COMP_CWORD -ge 2 ]]; then + case "${{COMP_WORDS[1]}}" in +{cases_str} + esac + fi + + if [[ $COMP_CWORD -eq 1 ]]; then + COMPREPLY=($(compgen -W "{top_cmds}" -- "$cur")) + fi +}} + +complete -F _hermes_completion hermes +""" + + +# --------------------------------------------------------------------------- +# Zsh +# --------------------------------------------------------------------------- + +def generate_zsh(parser: argparse.ArgumentParser) -> str: + tree = _walk(parser) + + top_cmds_lines: list[str] = [] + for cmd in sorted(tree["subcommands"]): + help_text = _clean(tree["subcommands"][cmd].get("help", "")) + top_cmds_lines.append(f" '{cmd}:{help_text}'") + top_cmds_str = "\n".join(top_cmds_lines) + + sub_cases: list[str] = [] + for cmd in sorted(tree["subcommands"]): + info = tree["subcommands"][cmd] + if not info["subcommands"]: + continue + if cmd == "profile": + # Profile subcommand: complete actions, then profile names for + # actions that accept a profile argument. + sub_lines: list[str] = [] + for sc in sorted(info["subcommands"]): + sh = _clean(info["subcommands"][sc].get("help", "")) + sub_lines.append(f" '{sc}:{sh}'") + sub_str = "\n".join(sub_lines) + sub_cases.append( + f" profile)\n" + f" case ${{line[2]}} in\n" + f" use|delete|show|alias|rename|export)\n" + f" _hermes_profiles\n" + f" ;;\n" + f" *)\n" + f" local -a profile_cmds\n" + f" profile_cmds=(\n" + f"{sub_str}\n" + f" )\n" + f" _describe 'profile command' profile_cmds\n" + f" ;;\n" + f" esac\n" + f" ;;" + ) + else: + sub_lines = [] + for sc in sorted(info["subcommands"]): + sh = _clean(info["subcommands"][sc].get("help", "")) + sub_lines.append(f" '{sc}:{sh}'") + sub_str = "\n".join(sub_lines) + safe = cmd.replace("-", "_") + sub_cases.append( + f" {cmd})\n" + f" local -a {safe}_cmds\n" + f" {safe}_cmds=(\n" + f"{sub_str}\n" + f" )\n" + f" _describe '{cmd} command' {safe}_cmds\n" + f" ;;" + ) + sub_cases_str = "\n".join(sub_cases) + + return f"""#compdef hermes +# Hermes Agent zsh completion +# Add to ~/.zshrc: +# eval "$(hermes completion zsh)" + +_hermes_profiles() {{ + local -a profiles + profiles=(default) + if [[ -d "$HOME/.hermes/profiles" ]]; then + profiles+=("${{(@f)$(ls $HOME/.hermes/profiles 2>/dev/null)}}") + fi + _describe 'profile' profiles +}} + +_hermes() {{ + local context state line + typeset -A opt_args + + _arguments -C \\ + '(-h --help){{-h,--help}}[Show help and exit]' \\ + '(-V --version){{-V,--version}}[Show version and exit]' \\ + '(-p --profile){{-p,--profile}}[Profile name]:profile:_hermes_profiles' \\ + '1:command:->commands' \\ + '*::arg:->args' + + case $state in + commands) + local -a subcmds + subcmds=( +{top_cmds_str} + ) + _describe 'hermes command' subcmds + ;; + args) + case ${{line[1]}} in +{sub_cases_str} + esac + ;; + esac +}} + +_hermes "$@" +""" + + +# --------------------------------------------------------------------------- +# Fish +# --------------------------------------------------------------------------- + +def generate_fish(parser: argparse.ArgumentParser) -> str: + tree = _walk(parser) + top_cmds = sorted(tree["subcommands"]) + top_cmds_str = " ".join(top_cmds) + + lines: list[str] = [ + "# Hermes Agent fish completion", + "# Add to your config:", + "# hermes completion fish | source", + "", + "# Helper: list available profiles", + "function __hermes_profiles", + " echo default", + " if test -d $HOME/.hermes/profiles", + " ls $HOME/.hermes/profiles 2>/dev/null", + " end", + "end", + "", + "# Disable file completion by default", + "complete -c hermes -f", + "", + "# Complete profile names after -p / --profile", + "complete -c hermes -f -s p -l profile" + " -d 'Profile name' -xa '(__hermes_profiles)'", + "", + "# Top-level subcommands", + ] + + for cmd in top_cmds: + info = tree["subcommands"][cmd] + help_text = _clean(info.get("help", "")) + lines.append( + f"complete -c hermes -f " + f"-n 'not __fish_seen_subcommand_from {top_cmds_str}' " + f"-a {cmd} -d '{help_text}'" + ) + + lines.append("") + lines.append("# Subcommand completions") + + profile_name_actions = {"use", "delete", "show", "alias", "rename", "export"} + + for cmd in top_cmds: + info = tree["subcommands"][cmd] + if not info["subcommands"]: + continue + lines.append(f"# {cmd}") + for sc in sorted(info["subcommands"]): + sinfo = info["subcommands"][sc] + sh = _clean(sinfo.get("help", "")) + lines.append( + f"complete -c hermes -f " + f"-n '__fish_seen_subcommand_from {cmd}' " + f"-a {sc} -d '{sh}'" + ) + # For profile subcommand, complete profile names for relevant actions + if cmd == "profile": + for action in sorted(profile_name_actions): + lines.append( + f"complete -c hermes -f " + f"-n '__fish_seen_subcommand_from {action}; " + f"and __fish_seen_subcommand_from profile' " + f"-a '(__hermes_profiles)' -d 'Profile name'" + ) + + lines.append("") + return "\n".join(lines) diff --git a/hermes_cli/config.py b/hermes_cli/config.py index 738960bb4708..cfcc7ff28f73 100644 --- a/hermes_cli/config.py +++ b/hermes_cli/config.py @@ -12,6 +12,8 @@ - hermes config wizard - Re-run setup wizard """ +import copy +import logging import os import platform import re @@ -23,10 +25,11 @@ from pathlib import Path from typing import Dict, Any, Optional, List, Tuple -from tools.tool_backend_helpers import managed_nous_tools_enabled as _managed_nous_tools_enabled +logger = logging.getLogger(__name__) _IS_WINDOWS = platform.system() == "Windows" _ENV_VAR_NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +_LAST_EXPANDED_CONFIG_BY_PATH: Dict[str, Any] = {} # Env var names written to .env that aren't in OPTIONAL_ENV_VARS # (managed by setup/provider flows directly). _EXTRA_ENV_KEYS = frozenset({ @@ -45,6 +48,10 @@ "WEIXIN_HOME_CHANNEL", "WEIXIN_HOME_CHANNEL_NAME", "WEIXIN_DM_POLICY", "WEIXIN_GROUP_POLICY", "WEIXIN_ALLOWED_USERS", "WEIXIN_GROUP_ALLOWED_USERS", "WEIXIN_ALLOW_ALL_USERS", "BLUEBUBBLES_SERVER_URL", "BLUEBUBBLES_PASSWORD", + "QQ_APP_ID", "QQ_CLIENT_SECRET", "QQBOT_HOME_CHANNEL", "QQBOT_HOME_CHANNEL_NAME", + "QQ_HOME_CHANNEL", "QQ_HOME_CHANNEL_NAME", # legacy aliases (pre-rename, still read for back-compat) + "QQ_ALLOWED_USERS", "QQ_GROUP_ALLOWED_USERS", "QQ_ALLOW_ALL_USERS", "QQ_MARKDOWN_SUPPORT", + "QQ_STT_API_KEY", "QQ_STT_BASE_URL", "QQ_STT_MODEL", "TERMINAL_ENV", "TERMINAL_SSH_KEY", "TERMINAL_SSH_PORT", "WHATSAPP_MODE", "WHATSAPP_ENABLED", "MATTERMOST_HOME_CHANNEL", "MATTERMOST_REPLY_MODE", @@ -238,13 +245,41 @@ def _secure_dir(path): pass +def _is_container() -> bool: + """Detect if we're running inside a Docker/Podman/LXC container. + + When Hermes runs in a container with volume-mounted config files, forcing + 0o600 permissions breaks multi-process setups where the gateway and + dashboard run as different UIDs or the volume mount requires broader + permissions. + """ + # Explicit opt-out + if os.environ.get("HERMES_CONTAINER") or os.environ.get("HERMES_SKIP_CHMOD"): + return True + # Docker / Podman marker file + if os.path.exists("/.dockerenv"): + return True + # LXC / cgroup-based detection + try: + with open("/proc/1/cgroup", "r") as f: + cgroup_content = f.read() + if "docker" in cgroup_content or "lxc" in cgroup_content or "kubepods" in cgroup_content: + return True + except (OSError, IOError): + pass + return False + + def _secure_file(path): """Set file to owner-only read/write (0600). No-op on Windows. Skipped in managed mode — the NixOS activation script sets group-readable permissions (0640) on config files. + + Skipped in containers — Docker/Podman volume mounts often need broader + permissions. Set HERMES_SKIP_CHMOD=1 to force-skip on other systems. """ - if is_managed(): + if is_managed() or _is_container(): return try: if os.path.exists(str(path)): @@ -326,6 +361,15 @@ def _ensure_hermes_home_managed(home: Path): # to finish, then interrupts any remaining runs after the timeout. # 0 = no drain, interrupt immediately. "restart_drain_timeout": 60, + # Max app-level retry attempts for API errors (connection drops, + # provider timeouts, 5xx, etc.) before the agent surfaces the + # failure. The OpenAI SDK already does its own low-level retries + # (max_retries=2 default) for transient network errors; this is + # the Hermes-level retry loop that wraps the whole call. Lower + # this to 1 if you use fallback providers and want fast failover + # on flaky primaries; raise it if you prefer to tolerate longer + # provider hiccups on a single provider. + "api_max_retries": 3, "service_tier": "", # Tool-use enforcement: injects system prompt guidance that tells the # model to actually call tools instead of describing intended actions. @@ -340,7 +384,11 @@ def _ensure_hermes_home_managed(home: Path): # Periodic "still working" notification interval (seconds). # Sends a status message every N seconds so the user knows the # agent hasn't died during long tasks. 0 = disable notifications. - "gateway_notify_interval": 600, + # Lower values mean faster feedback on slow tasks but more chat + # noise; 180s is a compromise that catches spinning weak-model runs + # (60+ tool iterations with tiny output) before users assume the + # bot is dead and /restart. + "gateway_notify_interval": 180, }, "terminal": { @@ -352,6 +400,32 @@ def _ensure_hermes_home_managed(home: Path): # (terminal and execute_code). Skill-declared required_environment_variables # are passed through automatically; this list is for non-skill use cases. "env_passthrough": [], + # Extra files to source in the login shell when building the + # per-session environment snapshot. Use this when tools like nvm, + # pyenv, asdf, or custom PATH entries are registered by files that + # a bash login shell would skip — most commonly ``~/.bashrc`` + # (bash doesn't source bashrc in non-interactive login mode) or + # zsh-specific files like ``~/.zshrc`` / ``~/.zprofile``. + # Paths support ``~`` / ``${VAR}``. Missing files are silently + # skipped. When empty, Hermes auto-sources ``~/.profile``, + # ``~/.bash_profile``, and ``~/.bashrc`` (in that order) if the + # snapshot shell is bash (this is the ``auto_source_bashrc`` + # behaviour — disable with that key if you want strict login-only + # semantics). + "shell_init_files": [], + # When true (default), Hermes sources the user's shell rc files + # (``~/.profile``, ``~/.bash_profile``, ``~/.bashrc``) in the + # login shell used to build the environment snapshot. This + # captures PATH additions, shell functions, and aliases — which a + # plain ``bash -l -c`` would otherwise miss because bash skips + # bashrc in non-interactive login mode, and because a default + # Debian/Ubuntu ``~/.bashrc`` short-circuits on non-interactive + # sources. ``~/.profile`` and ``~/.bash_profile`` are tried first + # because ``n`` / ``nvm`` / ``asdf`` installers typically write + # their PATH exports there without an interactivity guard. Turn + # this off if your rc files misbehave when sourced + # non-interactively (e.g. one that hard-exits on TTY checks). + "auto_source_bashrc": True, "docker_image": "nikolaik/python-nodejs:python3.11-nodejs20", "docker_forward_env": [], # Explicit environment variables to set inside Docker containers. @@ -370,7 +444,11 @@ def _ensure_hermes_home_managed(home: Path): "container_persistent": True, # Persist filesystem across sessions # Docker volume mounts — share host directories with the container. # Each entry is "host_path:container_path" (standard Docker -v syntax). - # Example: ["/home/user/projects:/workspace/projects", "/data:/data"] + # Example: + # ["/home/user/projects:/workspace/projects", + # "/home/user/.hermes/cache/documents:/output"] + # For gateway MEDIA delivery, write inside Docker to /output/... and emit + # the host-visible path in MEDIA:, not the container path. "docker_volumes": [], # Explicit opt-in: mount the host cwd into /workspace for Docker sessions. # Default off because passing host directories into a sandbox weakens isolation. @@ -387,10 +465,10 @@ def _ensure_hermes_home_managed(home: Path): "command_timeout": 30, # Timeout for browser commands in seconds (screenshot, navigate, etc.) "record_sessions": False, # Auto-record browser sessions as WebM videos "allow_private_urls": False, # Allow navigating to private/internal IPs (localhost, 192.168.x.x, etc.) + "cdp_url": "", # Optional persistent CDP endpoint for attaching to an existing Chromium/Chrome "camofox": { # When true, Hermes sends a stable profile-scoped userId to Camofox - # so the server can map it to a persistent browser profile directory. - # Requires Camofox server to be configured with CAMOFOX_PROFILE_DIR. + # so the server maps it to a persistent Firefox profile automatically. # When false (default), each session gets a random userId (ephemeral). "managed_persistence": False, }, @@ -416,13 +494,27 @@ def _ensure_hermes_home_managed(home: Path): "protect_last_n": 20, # minimum recent messages to keep uncompressed }, - "smart_model_routing": { - "enabled": False, - "max_simple_chars": 160, - "max_simple_words": 28, - "cheap_model": {}, + + # AWS Bedrock provider configuration. + # Only used when model.provider is "bedrock". + "bedrock": { + "region": "", # AWS region for Bedrock API calls (empty = AWS_REGION env var → us-east-1) + "discovery": { + "enabled": True, # Auto-discover models via ListFoundationModels + "provider_filter": [], # Only show models from these providers (e.g. ["anthropic", "amazon"]) + "refresh_interval": 3600, # Cache discovery results for this many seconds + }, + "guardrail": { + # Amazon Bedrock Guardrails — content filtering and safety policies. + # Create a guardrail in the Bedrock console, then set the ID and version here. + # See: https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html + "guardrail_identifier": "", # e.g. "abc123def456" + "guardrail_version": "", # e.g. "1" or "DRAFT" + "stream_processing_mode": "async", # "sync" or "async" + "trace": "disabled", # "enabled", "disabled", or "enabled_full" + }, }, - + # Auxiliary model config — provider:model for each side task. # Format: provider is the provider name, model is the model slug. # "auto" for provider = auto-detect best available provider. @@ -436,6 +528,7 @@ def _ensure_hermes_home_managed(home: Path): "base_url": "", # direct OpenAI-compatible endpoint (takes precedence over provider) "api_key": "", # API key for base_url (falls back to OPENAI_API_KEY) "timeout": 120, # seconds — LLM API call timeout; vision payloads need generous timeout + "extra_body": {}, # OpenAI-compatible provider-specific request fields "download_timeout": 30, # seconds — image HTTP download timeout; increase for slow connections }, "web_extract": { @@ -444,6 +537,7 @@ def _ensure_hermes_home_managed(home: Path): "base_url": "", "api_key": "", "timeout": 360, # seconds (6min) — per-attempt LLM summarization timeout; increase for slow local models + "extra_body": {}, }, "compression": { "provider": "auto", @@ -451,6 +545,7 @@ def _ensure_hermes_home_managed(home: Path): "base_url": "", "api_key": "", "timeout": 120, # seconds — compression summarises large contexts; increase for local models + "extra_body": {}, }, "session_search": { "provider": "auto", @@ -458,6 +553,8 @@ def _ensure_hermes_home_managed(home: Path): "base_url": "", "api_key": "", "timeout": 30, + "extra_body": {}, + "max_concurrency": 3, # Clamp parallel summaries to avoid request-burst 429s on small providers }, "skills_hub": { "provider": "auto", @@ -465,6 +562,7 @@ def _ensure_hermes_home_managed(home: Path): "base_url": "", "api_key": "", "timeout": 30, + "extra_body": {}, }, "approval": { "provider": "auto", @@ -472,6 +570,7 @@ def _ensure_hermes_home_managed(home: Path): "base_url": "", "api_key": "", "timeout": 30, + "extra_body": {}, }, "mcp": { "provider": "auto", @@ -479,6 +578,7 @@ def _ensure_hermes_home_managed(home: Path): "base_url": "", "api_key": "", "timeout": 30, + "extra_body": {}, }, "flush_memories": { "provider": "auto", @@ -486,6 +586,15 @@ def _ensure_hermes_home_managed(home: Path): "base_url": "", "api_key": "", "timeout": 30, + "extra_body": {}, + }, + "title_generation": { + "provider": "auto", + "model": "", + "base_url": "", + "api_key": "", + "timeout": 30, + "extra_body": {}, }, }, @@ -497,9 +606,14 @@ def _ensure_hermes_home_managed(home: Path): "bell_on_complete": False, "show_reasoning": False, "streaming": False, + "final_response_markdown": "strip", # render | strip | raw "inline_diffs": True, # Show inline diff previews for write actions (write_file, patch, skill_manage) "show_cost": False, # Show $ cost in the status bar (off by default) "skin": "default", + "user_message_preview": { # CLI: how many submitted user-message lines to echo back in scrollback + "first_lines": 2, + "last_lines": 2, + }, "interim_assistant_messages": True, # Gateway: show natural mid-turn assistant status messages "tool_progress_command": False, # Enable /verbose command in messaging gateway "tool_progress_overrides": {}, # DEPRECATED — use display.platforms instead @@ -507,14 +621,23 @@ def _ensure_hermes_home_managed(home: Path): "platforms": {}, # Per-platform display overrides: {"telegram": {"tool_progress": "all"}, "slack": {"tool_progress": "off"}} }, + # Web dashboard settings + "dashboard": { + "theme": "default", # Dashboard visual theme: "default", "midnight", "ember", "mono", "cyberpunk", "rose" + }, + # Privacy settings "privacy": { "redact_pii": False, # When True, hash user IDs and strip phone numbers from LLM context }, # Text-to-speech configuration + # Each provider supports an optional `max_text_length:` override for the + # per-request input-character cap. Omit it to use the provider's documented + # limit (OpenAI 4096, xAI 15000, MiniMax 10000, ElevenLabs 5k-40k model-aware, + # Gemini 5000, Edge 5000, Mistral 4000, NeuTTS/KittenTTS 2000). "tts": { - "provider": "edge", # "edge" (free) | "elevenlabs" (premium) | "openai" | "minimax" | "mistral" | "neutts" (local) + "provider": "edge", # "edge" (free) | "elevenlabs" (premium) | "openai" | "xai" | "minimax" | "mistral" | "neutts" (local) "edge": { "voice": "en-US-AriaNeural", # Popular: AriaNeural, JennyNeural, AndrewNeural, BrianNeural, SoniaNeural @@ -528,6 +651,12 @@ def _ensure_hermes_home_managed(home: Path): "voice": "alloy", # Voices: alloy, echo, fable, onyx, nova, shimmer }, + "xai": { + "voice_id": "eve", + "language": "en", + "sample_rate": 24000, + "bit_rate": 128000, + }, "mistral": { "model": "voxtral-mini-tts-2603", "voice_id": "c69964a6-ab8b-4f8a-9465-ec0925096ec8", # Paul - Neutral @@ -559,6 +688,7 @@ def _ensure_hermes_home_managed(home: Path): "record_key": "ctrl+b", "max_recording_seconds": 120, "auto_tts": False, + "beep_enabled": True, # Play record start/stop beeps in CLI voice mode "silence_threshold": 200, # RMS below this = silence (0-32767) "silence_duration": 3.0, # Seconds of silence before auto-stop }, @@ -601,10 +731,22 @@ def _ensure_hermes_home_managed(home: Path): "provider": "", # e.g. "openrouter" (empty = inherit parent provider + credentials) "base_url": "", # direct OpenAI-compatible endpoint for subagents "api_key": "", # API key for delegation.base_url (falls back to OPENAI_API_KEY) + # When delegate_task narrows child toolsets explicitly, preserve any + # MCP toolsets the parent already has enabled. On by default so + # narrowing (e.g. toolsets=["web","browser"]) expresses "I want these + # extras" without silently stripping MCP tools the parent already has. + # Set to false for strict intersection. + "inherit_mcp_toolsets": True, "max_iterations": 50, # per-subagent iteration cap (each subagent gets its own budget, # independent of the parent's max_iterations) "reasoning_effort": "", # reasoning effort for subagents: "xhigh", "high", "medium", # "low", "minimal", "none" (empty = inherit parent's level) + "max_concurrent_children": 3, # max parallel children per batch; floor of 1 enforced, no ceiling + # Orchestrator role controls (see tools/delegate_tool.py:_get_max_spawn_depth + # and _get_orchestrator_enabled). Values are clamped to [1, 3] with a + # warning log if out of range. + "max_spawn_depth": 1, # depth cap (1 = flat [default], 2 = orchestrator→leaf, 3 = three-level) + "orchestrator_enabled": True, # kill switch for role="orchestrator" }, # Ephemeral prefill messages file — JSON list of {role, content} dicts @@ -617,6 +759,31 @@ def _ensure_hermes_home_managed(home: Path): # always goes to ~/.hermes/skills/. "skills": { "external_dirs": [], # e.g. ["~/.agents/skills", "/shared/team-skills"] + # Substitute ${HERMES_SKILL_DIR} and ${HERMES_SESSION_ID} in SKILL.md + # content with the absolute skill directory and the active session id + # before the agent sees it. Lets skill authors reference bundled + # scripts without the agent having to join paths. + "template_vars": True, + # Pre-execute inline shell snippets written as !`cmd` in SKILL.md + # body. Their stdout is inlined into the skill message before the + # agent reads it, so skills can inject dynamic context (dates, git + # state, detected tool versions, …). Off by default because any + # content from the skill author runs on the host without approval; + # only enable for skill sources you trust. + "inline_shell": False, + # Timeout (seconds) for each !`cmd` snippet when inline_shell is on. + "inline_shell_timeout": 10, + # Run the keyword/pattern security scanner on skills the agent + # writes via skill_manage (create/edit/patch). Off by default + # because the agent can already execute the same code paths via + # terminal() with no gate, so the scan adds friction (blocks + # skills that mention risky keywords in prose) without meaningful + # security. Turn on if you want the belt-and-suspenders — a + # dangerous verdict will then surface as a tool error to the + # agent, which can retry with the flagged content removed. + # External hub installs (trusted/community sources) are always + # scanned regardless of this setting. + "guard_agent_created": False, }, # Honcho AI-native memory -- reads ~/.honcho/config.json as single source of truth. @@ -635,6 +802,15 @@ def _ensure_hermes_home_managed(home: Path): "allowed_channels": "", # If set, bot ONLY responds in these channel IDs (whitelist) "auto_thread": True, # Auto-create threads on @mention in channels (like Slack) "reactions": True, # Add 👀/✅/❌ reactions to messages during processing + "channel_prompts": {}, # Per-channel ephemeral system prompts (forum parents apply to child threads) + # discord_server tool: restrict which actions the agent may call. + # Default (empty) = all actions allowed (subject to bot privileged intents). + # Accepts comma-separated string ("list_guilds,list_channels,fetch_messages") + # or YAML list. Unknown names are dropped with a warning at load time. + # Actions: list_guilds, server_info, list_channels, channel_info, + # list_roles, member_info, search_members, fetch_messages, list_pins, + # pin_message, unpin_message, create_thread, add_role, remove_role. + "server_actions": "", }, # WhatsApp platform settings (gateway mode) @@ -645,19 +821,54 @@ def _ensure_hermes_home_managed(home: Path): # Supports \n for newlines, e.g. "🤖 *My Bot*\n──────\n" }, + # Telegram platform settings (gateway mode) + "telegram": { + "channel_prompts": {}, # Per-chat/topic ephemeral system prompts (topics inherit from parent group) + }, + + # Slack platform settings (gateway mode) + "slack": { + "channel_prompts": {}, # Per-channel ephemeral system prompts + }, + + # Mattermost platform settings (gateway mode) + "mattermost": { + "channel_prompts": {}, # Per-channel ephemeral system prompts + }, + # Approval mode for dangerous commands: # manual — always prompt the user (default) # smart — use auxiliary LLM to auto-approve low-risk commands, prompt for high-risk # off — skip all approval prompts (equivalent to --yolo) + # + # cron_mode — what to do when a cron job hits a dangerous command: + # deny — block the command and let the agent find another way (default, safe) + # approve — auto-approve all dangerous commands in cron jobs "approvals": { "mode": "manual", "timeout": 60, + "cron_mode": "deny", }, # Permanently allowed dangerous command patterns (added via "always" approval) "command_allowlist": [], # User-defined quick commands that bypass the agent loop (type: exec only) "quick_commands": {}, + + # Shell-script hooks — declarative bridge that invokes shell scripts + # on plugin-hook events (pre_tool_call, post_tool_call, pre_llm_call, + # subagent_stop, etc.). Each entry maps an event name to a list of + # {matcher, command, timeout} dicts. First registration of a new + # command prompts the user for consent; subsequent runs reuse the + # stored approval from ~/.hermes/shell-hooks-allowlist.json. + # See `website/docs/user-guide/features/hooks.md` for schema + examples. + "hooks": {}, + + # Auto-accept shell-hook registrations without a TTY prompt. Also + # toggleable per-invocation via --accept-hooks or HERMES_ACCEPT_HOOKS=1. + # Gateway / cron / non-interactive runs need this (or one of the other + # channels) to pick up newly-added hooks. + "hooks_auto_accept": False, # Custom personalities — add your own entries here # Supports string format: {"name": "system prompt"} # Or dict format: {"name": {"description": "...", "system_prompt": "...", "tone": "...", "style": "..."}} @@ -665,6 +876,7 @@ def _ensure_hermes_home_managed(home: Path): # Pre-exec security scanning via tirith "security": { + "allow_private_urls": False, # Allow requests to private/internal IPs (for OpenWrt, proxies, VPNs) "redact_secrets": True, "tirith_enabled": True, "tirith_path": "tirith", @@ -681,6 +893,25 @@ def _ensure_hermes_home_managed(home: Path): # Wrap delivered cron responses with a header (task name) and footer # ("The agent cannot see this message"). Set to false for clean output. "wrap_response": True, + # Maximum number of due jobs to run in parallel per tick. + # null/0 = unbounded (limited only by thread count). + # 1 = serial (pre-v0.9 behaviour). + # Also overridable via HERMES_CRON_MAX_PARALLEL env var. + "max_parallel_jobs": None, + }, + + # execute_code settings — controls the tool used for programmatic tool calls. + "code_execution": { + # Execution mode: + # project (default) — scripts run in the session's working directory + # with the active virtualenv/conda env's python, so project deps + # (pandas, torch, project packages) and relative paths resolve. + # strict — scripts run in an isolated temp directory with + # hermes-agent's own python (sys.executable). Maximum isolation + # and reproducibility; project deps and relative paths won't work. + # Env scrubbing (strips *_API_KEY, *_TOKEN, *_SECRET, ...) and the + # tool whitelist apply identically in both modes. + "mode": "project", }, # Logging — controls file logging to ~/.hermes/logs/. @@ -699,8 +930,36 @@ def _ensure_hermes_home_managed(home: Path): "force_ipv4": False, }, + # Session storage — controls automatic cleanup of ~/.hermes/state.db. + # state.db accumulates every session, message, tool call, and FTS5 index + # entry forever. Without auto-pruning, a heavy user (gateway + cron) + # reports 384MB+ databases with 68K+ messages, which slows down FTS5 + # inserts, /resume listing, and insights queries. + "sessions": { + # When true, prune ended sessions older than retention_days once + # per (roughly) min_interval_hours at CLI/gateway/cron startup. + # Only touches ended sessions — active sessions are always preserved. + # Default false: session history is valuable for search recall, and + # silently deleting it could surprise users. Opt in explicitly. + "auto_prune": False, + # How many days of ended-session history to keep. Matches the + # default of ``hermes sessions prune``. + "retention_days": 90, + # VACUUM after a prune that actually deleted rows. SQLite does not + # reclaim disk space on DELETE — freed pages are just reused on + # subsequent INSERTs — so without VACUUM the file stays bloated + # even after pruning. VACUUM blocks writes for a few seconds per + # 100MB, so it only runs at startup, and only when prune deleted + # ≥1 session. + "vacuum_after_prune": True, + # Minimum hours between auto-maintenance runs (avoids repeating + # the sweep on every CLI invocation). Tracked via state_meta in + # state.db itself, so it's shared across all processes. + "min_interval_hours": 24, + }, + # Config schema version - bump this when adding new required fields - "_config_version": 17, + "_config_version": 22, } # ============================================================================= @@ -768,6 +1027,38 @@ def _ensure_hermes_home_managed(home: Path): "category": "provider", "advanced": True, }, + "XAI_API_KEY": { + "description": "xAI API key", + "prompt": "xAI API key", + "url": "https://console.x.ai/", + "password": True, + "category": "provider", + "advanced": True, + }, + "XAI_BASE_URL": { + "description": "xAI base URL override", + "prompt": "xAI base URL (leave empty for default)", + "url": None, + "password": False, + "category": "provider", + "advanced": True, + }, + "NVIDIA_API_KEY": { + "description": "NVIDIA NIM API key (build.nvidia.com or local NIM endpoint)", + "prompt": "NVIDIA NIM API key", + "url": "https://build.nvidia.com/", + "password": True, + "category": "provider", + "advanced": True, + }, + "NVIDIA_BASE_URL": { + "description": "NVIDIA NIM base URL override (e.g. http://localhost:8000/v1 for local NIM)", + "prompt": "NVIDIA NIM base URL (leave empty for default)", + "url": None, + "password": False, + "category": "provider", + "advanced": True, + }, "GLM_API_KEY": { "description": "Z.AI / GLM API key (also recognized as ZAI_API_KEY / Z_AI_API_KEY)", "prompt": "Z.AI / GLM API key", @@ -824,6 +1115,38 @@ def _ensure_hermes_home_managed(home: Path): "category": "provider", "advanced": True, }, + "STEPFUN_API_KEY": { + "description": "StepFun Step Plan API key", + "prompt": "StepFun Step Plan API key", + "url": "https://platform.stepfun.com/", + "password": True, + "category": "provider", + "advanced": True, + }, + "STEPFUN_BASE_URL": { + "description": "StepFun Step Plan base URL override", + "prompt": "StepFun Step Plan base URL (leave empty for default)", + "url": None, + "password": False, + "category": "provider", + "advanced": True, + }, + "ARCEEAI_API_KEY": { + "description": "Arcee AI API key", + "prompt": "Arcee AI API key", + "url": "https://chat.arcee.ai/", + "password": True, + "category": "provider", + "advanced": True, + }, + "ARCEE_BASE_URL": { + "description": "Arcee AI base URL override", + "prompt": "Arcee base URL (leave empty for default)", + "url": None, + "password": False, + "category": "provider", + "advanced": True, + }, "MINIMAX_API_KEY": { "description": "MiniMax API key (international)", "prompt": "MiniMax API key", @@ -893,6 +1216,30 @@ def _ensure_hermes_home_managed(home: Path): "category": "provider", "advanced": True, }, + "HERMES_GEMINI_CLIENT_ID": { + "description": "Google OAuth client ID for google-gemini-cli (optional; defaults to Google's public gemini-cli client)", + "prompt": "Google OAuth client ID (optional — leave empty to use the public default)", + "url": "https://console.cloud.google.com/apis/credentials", + "password": False, + "category": "provider", + "advanced": True, + }, + "HERMES_GEMINI_CLIENT_SECRET": { + "description": "Google OAuth client secret for google-gemini-cli (optional)", + "prompt": "Google OAuth client secret (optional)", + "url": "https://console.cloud.google.com/apis/credentials", + "password": True, + "category": "provider", + "advanced": True, + }, + "HERMES_GEMINI_PROJECT_ID": { + "description": "GCP project ID for paid Gemini tiers (free tier auto-provisions)", + "prompt": "GCP project ID for Gemini OAuth (leave empty for free tier)", + "url": None, + "password": False, + "category": "provider", + "advanced": True, + }, "OPENCODE_ZEN_API_KEY": { "description": "OpenCode Zen API key (pay-as-you-go access to curated models)", "prompt": "OpenCode Zen API key", @@ -940,8 +1287,24 @@ def _ensure_hermes_home_managed(home: Path): "category": "provider", "advanced": True, }, + "OLLAMA_API_KEY": { + "description": "Ollama Cloud API key (ollama.com — cloud-hosted open models)", + "prompt": "Ollama Cloud API key", + "url": "https://ollama.com/settings", + "password": True, + "category": "provider", + "advanced": True, + }, + "OLLAMA_BASE_URL": { + "description": "Ollama Cloud base URL override (default: https://ollama.com/v1)", + "prompt": "Ollama base URL (leave empty for default)", + "url": None, + "password": False, + "category": "provider", + "advanced": True, + }, "XIAOMI_API_KEY": { - "description": "Xiaomi MiMo API key for MiMo models (mimo-v2-pro, mimo-v2-omni, mimo-v2-flash)", + "description": "Xiaomi MiMo API key for MiMo models (mimo-v2.5-pro, mimo-v2.5, mimo-v2-pro, mimo-v2-omni, mimo-v2-flash)", "prompt": "Xiaomi MiMo API Key", "url": "https://platform.xiaomimimo.com", "password": True, @@ -955,6 +1318,22 @@ def _ensure_hermes_home_managed(home: Path): "category": "provider", "advanced": True, }, + "AWS_REGION": { + "description": "AWS region for Bedrock API calls (e.g. us-east-1, eu-central-1)", + "prompt": "AWS Region", + "url": "https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-regions.html", + "password": False, + "category": "provider", + "advanced": True, + }, + "AWS_PROFILE": { + "description": "AWS named profile for Bedrock authentication (from ~/.aws/credentials)", + "prompt": "AWS Profile", + "url": None, + "password": False, + "category": "provider", + "advanced": True, + }, # ── Tool API keys ── "EXA_API_KEY": { @@ -1152,6 +1531,12 @@ def _ensure_hermes_home_managed(home: Path): "password": False, "category": "messaging", }, + "TELEGRAM_PROXY": { + "description": "Proxy URL for Telegram connections (overrides HTTPS_PROXY). Supports http://, https://, socks5://", + "prompt": "Telegram proxy URL (optional)", + "password": False, + "category": "messaging", + }, "DISCORD_BOT_TOKEN": { "description": "Discord bot token from Developer Portal", "prompt": "Discord bot token", @@ -1176,7 +1561,7 @@ def _ensure_hermes_home_managed(home: Path): "SLACK_BOT_TOKEN": { "description": "Slack bot token (xoxb-). Get from OAuth & Permissions after installing your app. " "Required scopes: chat:write, app_mentions:read, channels:history, groups:history, " - "im:history, im:read, im:write, users:read, files:write", + "im:history, im:read, im:write, users:read, files:read, files:write", "prompt": "Slack Bot Token (xoxb-...)", "url": "https://api.slack.com/apps", "password": True, @@ -1315,6 +1700,53 @@ def _ensure_hermes_home_managed(home: Path): "password": False, "category": "messaging", }, + "BLUEBUBBLES_ALLOW_ALL_USERS": { + "description": "Allow all BlueBubbles users without allowlist", + "prompt": "Allow All BlueBubbles Users", + "category": "messaging", + }, + "QQ_APP_ID": { + "description": "QQ Bot App ID from QQ Open Platform (q.qq.com)", + "prompt": "QQ App ID", + "url": "https://q.qq.com", + "category": "messaging", + }, + "QQ_CLIENT_SECRET": { + "description": "QQ Bot Client Secret from QQ Open Platform", + "prompt": "QQ Client Secret", + "password": True, + "category": "messaging", + }, + "QQ_ALLOWED_USERS": { + "description": "Comma-separated QQ user IDs allowed to use the bot", + "prompt": "QQ Allowed Users", + "category": "messaging", + }, + "QQ_GROUP_ALLOWED_USERS": { + "description": "Comma-separated QQ group IDs allowed to interact with the bot", + "prompt": "QQ Group Allowed Users", + "category": "messaging", + }, + "QQ_ALLOW_ALL_USERS": { + "description": "Allow all QQ users without an allowlist (true/false)", + "prompt": "Allow All QQ Users", + "category": "messaging", + }, + "QQBOT_HOME_CHANNEL": { + "description": "Default QQ channel/group for cron delivery and notifications", + "prompt": "QQ Home Channel", + "category": "messaging", + }, + "QQBOT_HOME_CHANNEL_NAME": { + "description": "Display name for the QQ home channel", + "prompt": "QQ Home Channel Name", + "category": "messaging", + }, + "QQ_SANDBOX": { + "description": "Enable QQ sandbox mode for development testing (true/false)", + "prompt": "QQ Sandbox Mode", + "category": "messaging", + }, "GATEWAY_ALLOW_ALL_USERS": { "description": "Allow all users to interact with messaging bots (true/false). Default: false.", "prompt": "Allow all users (true/false)", @@ -1363,6 +1795,22 @@ def _ensure_hermes_home_managed(home: Path): "category": "messaging", "advanced": True, }, + "GATEWAY_PROXY_URL": { + "description": "URL of a remote Hermes API server to forward messages to (proxy mode). When set, the gateway handles platform I/O only — all agent work is delegated to the remote server. Use for Docker E2EE containers that relay to a host agent. Also configurable via gateway.proxy_url in config.yaml.", + "prompt": "Remote Hermes API server URL (e.g. http://192.168.1.100:8642)", + "url": None, + "password": False, + "category": "messaging", + "advanced": True, + }, + "GATEWAY_PROXY_KEY": { + "description": "Bearer token for authenticating with the remote Hermes API server (proxy mode). Must match the API_SERVER_KEY on the remote host.", + "prompt": "Remote API server auth key", + "url": None, + "password": True, + "category": "messaging", + "advanced": True, + }, "WEBHOOK_ENABLED": { "description": "Enable the webhook platform adapter for receiving events from GitHub, GitLab, etc.", "prompt": "Enable webhooks (true/false)", @@ -1386,13 +1834,8 @@ def _ensure_hermes_home_managed(home: Path): }, # ── Agent settings ── - "MESSAGING_CWD": { - "description": "Working directory for terminal commands via messaging", - "prompt": "Messaging working directory (default: home)", - "url": None, - "password": False, - "category": "setting", - }, + # NOTE: MESSAGING_CWD was removed here — use terminal.cwd in config.yaml + # instead. The gateway reads TERMINAL_CWD (bridged from terminal.cwd). "SUDO_PASSWORD": { "description": "Sudo password for terminal commands requiring root access; set to an explicit empty string to try empty without prompting", "prompt": "Sudo password", @@ -1440,14 +1883,8 @@ def _ensure_hermes_home_managed(home: Path): }, } -if not _managed_nous_tools_enabled(): - for _hidden_var in ( - "FIRECRAWL_GATEWAY_URL", - "TOOL_GATEWAY_DOMAIN", - "TOOL_GATEWAY_SCHEME", - "TOOL_GATEWAY_USER_TOKEN", - ): - OPTIONAL_ENV_VARS.pop(_hidden_var, None) +# Tool Gateway env vars are always visible — they're useful for +# self-hosted / custom gateway setups regardless of subscription state. def get_missing_env_vars(required_only: bool = False) -> List[Dict[str, Any]]: @@ -1561,12 +1998,53 @@ def _normalize_custom_provider_entry( if not isinstance(entry, dict): return None + # Accept camelCase aliases commonly used in hand-written configs. + _CAMEL_ALIASES: Dict[str, str] = { + "apiKey": "api_key", + "baseUrl": "base_url", + "apiMode": "api_mode", + "keyEnv": "key_env", + "defaultModel": "default_model", + "contextLength": "context_length", + "rateLimitDelay": "rate_limit_delay", + } + _KNOWN_KEYS = { + "name", "api", "url", "base_url", "api_key", "key_env", + "api_mode", "transport", "model", "default_model", "models", + "context_length", "rate_limit_delay", + } + for camel, snake in _CAMEL_ALIASES.items(): + if camel in entry and snake not in entry: + logger.warning( + "providers.%s: camelCase key '%s' auto-mapped to '%s' " + "(use snake_case to avoid this warning)", + provider_key or "?", camel, snake, + ) + entry[snake] = entry[camel] + unknown = set(entry.keys()) - _KNOWN_KEYS - set(_CAMEL_ALIASES.keys()) + if unknown: + logger.warning( + "providers.%s: unknown config keys ignored: %s", + provider_key or "?", ", ".join(sorted(unknown)), + ) + + from urllib.parse import urlparse + base_url = "" - for url_key in ("api", "url", "base_url"): + for url_key in ("base_url", "url", "api"): raw_url = entry.get(url_key) if isinstance(raw_url, str) and raw_url.strip(): - base_url = raw_url.strip() - break + candidate = raw_url.strip() + parsed = urlparse(candidate) + if parsed.scheme and parsed.netloc: + base_url = candidate + break + else: + logger.warning( + "providers.%s: '%s' value '%s' is not a valid URL " + "(no scheme or host) — skipped", + provider_key or "?", url_key, candidate, + ) if not base_url: return None @@ -1607,6 +2085,14 @@ def _normalize_custom_provider_entry( models = entry.get("models") if isinstance(models, dict) and models: normalized["models"] = models + elif isinstance(models, list) and models: + # Hand-edited configs (and older Hermes versions) write ``models`` as + # a plain list of model ids. Preserve them by converting to the dict + # shape downstream code expects; otherwise normalize silently drops + # the list and /model shows the provider with (0) models. + normalized["models"] = { + str(m): {} for m in models if isinstance(m, str) and m.strip() + } context_length = entry.get("context_length") if isinstance(context_length, int) and context_length > 0: @@ -1656,7 +2142,8 @@ def _append_if_new(entry: Optional[Dict[str, Any]]) -> None: provider_key = str(entry.get("provider_key", "") or "").strip().lower() name = str(entry.get("name", "") or "").strip().lower() base_url = str(entry.get("base_url", "") or "").strip().rstrip("/").lower() - pair = (name, base_url) + model = str(entry.get("model", "") or "").strip().lower() + pair = (name, base_url, model) if provider_key and provider_key in seen_provider_keys: return @@ -1704,6 +2191,7 @@ def check_config_version() -> Tuple[int, int]: "fallback_providers", "credential_pool_strategies", "toolsets", "agent", "terminal", "display", "compression", "delegation", "auxiliary", "custom_providers", "context", "memory", "gateway", + "sessions", } # Valid fields inside a custom_providers list entry @@ -1861,7 +2349,6 @@ def print_config_warnings(config: Optional[Dict[str, Any]] = None) -> None: if not issues: return - import sys lines = ["\033[33m⚠ Config issues detected in config.yaml:\033[0m"] for ci in issues: marker = "\033[31m✗\033[0m" if ci.severity == "error" else "\033[33m⚠\033[0m" @@ -1870,6 +2357,51 @@ def print_config_warnings(config: Optional[Dict[str, Any]] = None) -> None: sys.stderr.write("\n".join(lines) + "\n\n") +def warn_deprecated_cwd_env_vars(config: Optional[Dict[str, Any]] = None) -> None: + """Warn if MESSAGING_CWD or TERMINAL_CWD is set in .env instead of config.yaml. + + These env vars are deprecated — the canonical setting is terminal.cwd + in config.yaml. Prints a migration hint to stderr. + """ + messaging_cwd = os.environ.get("MESSAGING_CWD") + terminal_cwd_env = os.environ.get("TERMINAL_CWD") + + if config is None: + try: + config = load_config() + except Exception: + return + + terminal_cfg = config.get("terminal", {}) + config_cwd = terminal_cfg.get("cwd", ".") if isinstance(terminal_cfg, dict) else "." + # Only warn if config.yaml doesn't have an explicit path + config_has_explicit_cwd = config_cwd not in (".", "auto", "cwd", "") + + lines: list[str] = [] + if messaging_cwd: + lines.append( + f" \033[33m⚠\033[0m MESSAGING_CWD={messaging_cwd} found in .env — " + f"this is deprecated." + ) + if terminal_cwd_env and not config_has_explicit_cwd: + # TERMINAL_CWD in env but not from config bridge — likely from .env + lines.append( + f" \033[33m⚠\033[0m TERMINAL_CWD={terminal_cwd_env} found in .env — " + f"this is deprecated." + ) + if lines: + hint_path = os.environ.get("HERMES_HOME", "~/.hermes") + lines.insert(0, "\033[33m⚠ Deprecated .env settings detected:\033[0m") + lines.append( + f" \033[2mMove to config.yaml instead: " + f"terminal:\\n cwd: /your/project/path\033[0m" + ) + lines.append( + f" \033[2mThen remove the old entries from {hint_path}/.env\033[0m" + ) + sys.stderr.write("\n".join(lines) + "\n\n") + + def migrate_config(interactive: bool = True, quiet: bool = False) -> Dict[str, Any]: """ Migrate config to latest version, prompting for new required fields. @@ -2148,6 +2680,71 @@ def migrate_config(interactive: bool = True, quiet: bool = False) -> Dict[str, A else: print(" ✓ Removed unused compression.summary_* keys") + # ── Version 20 → 21: plugins are now opt-in; grandfather existing user plugins ── + # The loader now requires plugins to appear in ``plugins.enabled`` before + # loading. Existing installs had all discovered plugins loading by default + # (minus anything in ``plugins.disabled``). To avoid silently breaking + # those setups on upgrade, populate ``plugins.enabled`` with the set of + # currently-installed user plugins that aren't already disabled. + # + # Bundled plugins (shipped in the repo itself) are NOT grandfathered — + # they ship off for everyone, including existing users, so any user who + # wants one has to opt in explicitly. + if current_ver < 21: + config = read_raw_config() + plugins_cfg = config.get("plugins") + if not isinstance(plugins_cfg, dict): + plugins_cfg = {} + # Only migrate if the enabled allow-list hasn't been set yet. + if "enabled" not in plugins_cfg: + disabled = plugins_cfg.get("disabled", []) or [] + if not isinstance(disabled, list): + disabled = [] + disabled_set = set(disabled) + + # Scan ``$HERMES_HOME/plugins/`` for currently installed user plugins. + grandfathered: List[str] = [] + try: + user_plugins_dir = get_hermes_home() / "plugins" + if user_plugins_dir.is_dir(): + for child in sorted(user_plugins_dir.iterdir()): + if not child.is_dir(): + continue + manifest_file = child / "plugin.yaml" + if not manifest_file.exists(): + manifest_file = child / "plugin.yml" + if not manifest_file.exists(): + continue + try: + with open(manifest_file) as _mf: + manifest = yaml.safe_load(_mf) or {} + except Exception: + manifest = {} + name = manifest.get("name") or child.name + if name in disabled_set: + continue + grandfathered.append(name) + except Exception: + grandfathered = [] + + plugins_cfg["enabled"] = grandfathered + config["plugins"] = plugins_cfg + save_config(config) + results["config_added"].append( + f"plugins.enabled (opt-in allow-list, {len(grandfathered)} grandfathered)" + ) + if not quiet: + if grandfathered: + print( + f" ✓ Plugins now opt-in: grandfathered " + f"{len(grandfathered)} existing plugin(s) into plugins.enabled" + ) + else: + print( + " ✓ Plugins now opt-in: no existing plugins to grandfather. " + "Use `hermes plugins enable ` to activate." + ) + if current_ver < latest_ver and not quiet: print(f"Config version: {current_ver} → {latest_ver}") @@ -2340,6 +2937,85 @@ def _expand_env_vars(obj): return obj +def _items_by_unique_name(items): + """Return a name-indexed dict only when all items have unique string names.""" + if not isinstance(items, list): + return None + indexed = {} + for item in items: + if not isinstance(item, dict) or not isinstance(item.get("name"), str): + return None + name = item["name"] + if name in indexed: + return None + indexed[name] = item + return indexed + + +def _preserve_env_ref_templates(current, raw, loaded_expanded=None): + """Restore raw ``${VAR}`` templates when a value is otherwise unchanged. + + ``load_config()`` expands env refs for runtime use. When a caller later + persists that config after modifying some unrelated setting, keep the + original on-disk template instead of writing the expanded plaintext + secret back to ``config.yaml``. + + Prefer preserving the raw template when ``current`` still matches either + the value previously returned by ``load_config()`` for this config path or + the current environment expansion of ``raw``. This handles env-var + rotation between load and save while still treating mixed literal/template + string edits as caller-owned once their rendered value diverges. + """ + if isinstance(current, str) and isinstance(raw, str) and re.search(r"\${[^}]+}", raw): + if current == raw: + return raw + if isinstance(loaded_expanded, str) and current == loaded_expanded: + return raw + if _expand_env_vars(raw) == current: + return raw + return current + + if isinstance(current, dict) and isinstance(raw, dict): + return { + key: _preserve_env_ref_templates( + value, + raw.get(key), + loaded_expanded.get(key) if isinstance(loaded_expanded, dict) else None, + ) + for key, value in current.items() + } + + if isinstance(current, list) and isinstance(raw, list): + # Prefer matching named config objects (e.g. custom_providers) by name + # so harmless reordering doesn't drop the original template. If names + # are duplicated, fall back to positional matching instead of silently + # shadowing one entry. + current_by_name = _items_by_unique_name(current) + raw_by_name = _items_by_unique_name(raw) + loaded_by_name = _items_by_unique_name(loaded_expanded) + if current_by_name is not None and raw_by_name is not None: + return [ + _preserve_env_ref_templates( + item, + raw_by_name.get(item.get("name")), + loaded_by_name.get(item.get("name")) if loaded_by_name is not None else None, + ) + for item in current + ] + return [ + _preserve_env_ref_templates( + item, + raw[index] if index < len(raw) else None, + loaded_expanded[index] + if isinstance(loaded_expanded, list) and index < len(loaded_expanded) + else None, + ) + for index, item in enumerate(current) + ] + + return current + + def _normalize_root_model_keys(config: Dict[str, Any]) -> Dict[str, Any]: """Move stale root-level provider/base_url into model section. @@ -2407,7 +3083,6 @@ def read_raw_config() -> Dict[str, Any]: def load_config() -> Dict[str, Any]: """Load configuration from ~/.hermes/config.yaml.""" - import copy ensure_hermes_home() config_path = get_config_path() @@ -2428,8 +3103,11 @@ def load_config() -> Dict[str, Any]: config = _deep_merge(config, user_config) except Exception as e: print(f"Warning: Failed to load config: {e}") - - return _expand_env_vars(_normalize_root_model_keys(_normalize_max_turns_config(config))) + + normalized = _normalize_root_model_keys(_normalize_max_turns_config(config)) + expanded = _expand_env_vars(normalized) + _LAST_EXPANDED_CONFIG_BY_PATH[str(config_path)] = copy.deepcopy(expanded) + return expanded _SECURITY_COMMENT = """ @@ -2464,24 +3142,11 @@ def load_config() -> Dict[str, Any]: # minimax (MINIMAX_API_KEY) — MiniMax # minimax-cn (MINIMAX_CN_API_KEY) — MiniMax (China) # -# For custom OpenAI-compatible endpoints, add base_url and api_key_env. +# For custom OpenAI-compatible endpoints, add base_url and key_env. # # fallback_model: # provider: openrouter # model: anthropic/claude-sonnet-4 -# -# ── Smart Model Routing ──────────────────────────────────────────────── -# Optional cheap-vs-strong routing for simple turns. -# Keeps the primary model for complex work, but can route short/simple -# messages to a cheaper model across providers. -# -# smart_model_routing: -# enabled: true -# max_simple_chars: 160 -# max_simple_words: 28 -# cheap_model: -# provider: openrouter -# model: google/gemini-2.5-flash """ @@ -2508,24 +3173,11 @@ def load_config() -> Dict[str, Any]: # minimax (MINIMAX_API_KEY) — MiniMax # minimax-cn (MINIMAX_CN_API_KEY) — MiniMax (China) # -# For custom OpenAI-compatible endpoints, add base_url and api_key_env. +# For custom OpenAI-compatible endpoints, add base_url and key_env. # # fallback_model: # provider: openrouter # model: anthropic/claude-sonnet-4 -# -# ── Smart Model Routing ──────────────────────────────────────────────── -# Optional cheap-vs-strong routing for simple turns. -# Keeps the primary model for complex work, but can route short/simple -# messages to a cheaper model across providers. -# -# smart_model_routing: -# enabled: true -# max_simple_chars: 160 -# max_simple_words: 28 -# cheap_model: -# provider: openrouter -# model: google/gemini-2.5-flash """ @@ -2538,7 +3190,15 @@ def save_config(config: Dict[str, Any]): ensure_hermes_home() config_path = get_config_path() - normalized = _normalize_root_model_keys(_normalize_max_turns_config(config)) + current_normalized = _normalize_root_model_keys(_normalize_max_turns_config(config)) + normalized = current_normalized + raw_existing = _normalize_root_model_keys(_normalize_max_turns_config(read_raw_config())) + if raw_existing: + normalized = _preserve_env_ref_templates( + normalized, + raw_existing, + _LAST_EXPANDED_CONFIG_BY_PATH.get(str(config_path)), + ) # Build optional commented-out sections for features that are off by # default or only relevant when explicitly configured. @@ -2547,7 +3207,7 @@ def save_config(config: Dict[str, Any]): if not sec or sec.get("redact_secrets") is None: parts.append(_SECURITY_COMMENT) fb = normalized.get("fallback_model", {}) - if not fb or not (fb.get("provider") and fb.get("model")): + if not fb or not isinstance(fb, dict) or not (fb.get("provider") and fb.get("model")): parts.append(_FALLBACK_COMMENT) atomic_yaml_write( @@ -2556,6 +3216,7 @@ def save_config(config: Dict[str, Any]): extra_content="".join(parts) if parts else None, ) _secure_file(config_path) + _LAST_EXPANDED_CONFIG_BY_PATH[str(config_path)] = copy.deepcopy(current_normalized) def load_env() -> Dict[str, str]: @@ -2683,6 +3344,46 @@ def sanitize_env_file() -> int: return fixes +def _check_non_ascii_credential(key: str, value: str) -> str: + """Warn and strip non-ASCII characters from credential values. + + API keys and tokens must be pure ASCII — they are sent as HTTP header + values which httpx/httpcore encode as ASCII. Non-ASCII characters + (commonly introduced by copy-pasting from rich-text editors or PDFs + that substitute lookalike Unicode glyphs for ASCII letters) cause + ``UnicodeEncodeError: 'ascii' codec can't encode character`` at + request time. + + Returns the sanitized (ASCII-only) value. Prints a warning if any + non-ASCII characters were found and removed. + """ + try: + value.encode("ascii") + return value # all ASCII — nothing to do + except UnicodeEncodeError: + pass + + # Build a readable list of the offending characters + bad_chars: list[str] = [] + for i, ch in enumerate(value): + if ord(ch) > 127: + bad_chars.append(f" position {i}: {ch!r} (U+{ord(ch):04X})") + sanitized = value.encode("ascii", errors="ignore").decode("ascii") + + print( + f"\n Warning: {key} contains non-ASCII characters that will break API requests.\n" + f" This usually happens when copy-pasting from a PDF, rich-text editor,\n" + f" or web page that substitutes lookalike Unicode glyphs for ASCII letters.\n" + f"\n" + + "\n".join(f" {line}" for line in bad_chars[:5]) + + ("\n ... and more" if len(bad_chars) > 5 else "") + + f"\n\n The non-ASCII characters have been stripped automatically.\n" + f" If authentication fails, re-copy the key from the provider's dashboard.\n", + file=sys.stderr, + ) + return sanitized + + def save_env_value(key: str, value: str): """Save or update a value in ~/.hermes/.env.""" if is_managed(): @@ -2691,6 +3392,8 @@ def save_env_value(key: str, value: str): if not _ENV_VAR_NAME_RE.match(key): raise ValueError(f"Invalid environment variable name: {key!r}") value = value.replace("\n", "").replace("\r", "") + # API keys / tokens must be ASCII — strip non-ASCII with a warning. + value = _check_non_ascii_credential(key, value) ensure_hermes_home() env_path = get_env_path() @@ -2721,12 +3424,25 @@ def save_env_value(key: str, value: str): lines.append(f"{key}={value}\n") fd, tmp_path = tempfile.mkstemp(dir=str(env_path.parent), suffix='.tmp', prefix='.env_') + # Preserve original permissions so Docker volume mounts aren't clobbered. + original_mode = None + if env_path.exists(): + try: + original_mode = stat.S_IMODE(env_path.stat().st_mode) + except OSError: + pass try: with os.fdopen(fd, 'w', **write_kw) as f: f.writelines(lines) f.flush() os.fsync(f.fileno()) os.replace(tmp_path, env_path) + # Restore original permissions before _secure_file may tighten them. + if original_mode is not None: + try: + os.chmod(env_path, original_mode) + except OSError: + pass except BaseException: try: os.unlink(tmp_path) @@ -2737,13 +3453,6 @@ def save_env_value(key: str, value: str): os.environ[key] = value - # Restrict .env permissions to owner-only (contains API keys) - if not _IS_WINDOWS: - try: - os.chmod(env_path, stat.S_IRUSR | stat.S_IWUSR) - except OSError: - pass - def remove_env_value(key: str) -> bool: """Remove a key from ~/.hermes/.env and os.environ. @@ -2772,12 +3481,23 @@ def remove_env_value(key: str) -> bool: if found: fd, tmp_path = tempfile.mkstemp(dir=str(env_path.parent), suffix='.tmp', prefix='.env_') + # Preserve original permissions so Docker volume mounts aren't clobbered. + original_mode = None + try: + original_mode = stat.S_IMODE(env_path.stat().st_mode) + except OSError: + pass try: with os.fdopen(fd, 'w', **write_kw) as f: f.writelines(new_lines) f.flush() os.fsync(f.fileno()) os.replace(tmp_path, env_path) + if original_mode is not None: + try: + os.chmod(env_path, original_mode) + except OSError: + pass except BaseException: try: os.unlink(tmp_path) @@ -2919,6 +3639,10 @@ def show_config(): print(f" Personality: {display.get('personality', 'kawaii')}") print(f" Reasoning: {'on' if display.get('show_reasoning', False) else 'off'}") print(f" Bell: {'on' if display.get('bell_on_complete', False) else 'off'}") + ump = display.get('user_message_preview', {}) if isinstance(display.get('user_message_preview', {}), dict) else {} + ump_first = ump.get('first_lines', 2) + ump_last = ump.get('last_lines', 2) + print(f" User preview: first {ump_first} line(s), last {ump_last} line(s)") # Terminal print() diff --git a/hermes_cli/curses_ui.py b/hermes_cli/curses_ui.py index 4880171fd4b7..b05295f1e61d 100644 --- a/hermes_cli/curses_ui.py +++ b/hermes_cli/curses_ui.py @@ -166,6 +166,7 @@ def curses_radiolist( selected: int = 0, *, cancel_returns: int | None = None, + description: str | None = None, ) -> int: """Curses single-select radio list. Returns the selected index. @@ -174,6 +175,9 @@ def curses_radiolist( items: Display labels for each row. selected: Index that starts selected (pre-selected). cancel_returns: Returned on ESC/q. Defaults to the original *selected*. + description: Optional multi-line text shown between the title and + the item list. Useful for context that should survive the + curses screen clear. """ if cancel_returns is None: cancel_returns = selected @@ -181,6 +185,10 @@ def curses_radiolist( if not sys.stdin.isatty(): return cancel_returns + desc_lines: list[str] = [] + if description: + desc_lines = description.splitlines() + try: import curses result_holder: list = [None] @@ -199,22 +207,35 @@ def _draw(stdscr): stdscr.clear() max_y, max_x = stdscr.getmaxyx() + row = 0 + # Header try: hattr = curses.A_BOLD if curses.has_colors(): hattr |= curses.color_pair(2) - stdscr.addnstr(0, 0, title, max_x - 1, hattr) + stdscr.addnstr(row, 0, title, max_x - 1, hattr) + row += 1 + + # Description lines + for dline in desc_lines: + if row >= max_y - 1: + break + stdscr.addnstr(row, 0, dline, max_x - 1, curses.A_NORMAL) + row += 1 + stdscr.addnstr( - 1, 0, + row, 0, " \u2191\u2193 navigate ENTER/SPACE select ESC cancel", max_x - 1, curses.A_DIM, ) + row += 1 except curses.error: pass # Scrollable item list - visible_rows = max_y - 4 + items_start = row + 1 + visible_rows = max_y - items_start - 1 if cursor < scroll_offset: scroll_offset = cursor elif cursor >= scroll_offset + visible_rows: @@ -223,7 +244,7 @@ def _draw(stdscr): for draw_i, i in enumerate( range(scroll_offset, min(len(items), scroll_offset + visible_rows)) ): - y = draw_i + 3 + y = draw_i + items_start if y >= max_y - 1: break radio = "\u25cf" if i == selected else "\u25cb" diff --git a/hermes_cli/debug.py b/hermes_cli/debug.py index 3607db9231b1..8915d8a6a730 100644 --- a/hermes_cli/debug.py +++ b/hermes_cli/debug.py @@ -6,10 +6,14 @@ """ import io +import json +import os import sys +import time import urllib.error import urllib.parse import urllib.request +from dataclasses import dataclass from pathlib import Path from typing import Optional @@ -27,6 +31,213 @@ # paste.rs caps at ~1 MB; we stay under that with headroom. _MAX_LOG_BYTES = 512_000 +# Auto-delete pastes after this many seconds (6 hours). +_AUTO_DELETE_SECONDS = 21600 + + +# --------------------------------------------------------------------------- +# Pending-deletion tracking (replaces the old fork-and-sleep subprocess). +# --------------------------------------------------------------------------- + +def _pending_file() -> Path: + """Path to ``~/.hermes/pastes/pending.json``. + + Each entry: ``{"url": "...", "expire_at": }``. Scheduled + DELETEs used to be handled by spawning a detached Python process per + paste that slept for 6 hours; those accumulated forever if the user + ran ``hermes debug share`` repeatedly. We now persist the schedule + to disk and sweep expired entries on the next debug invocation. + """ + return get_hermes_home() / "pastes" / "pending.json" + + +def _load_pending() -> list[dict]: + path = _pending_file() + if not path.exists(): + return [] + try: + data = json.loads(path.read_text(encoding="utf-8")) + if isinstance(data, list): + # Filter to well-formed entries only + return [ + e for e in data + if isinstance(e, dict) and "url" in e and "expire_at" in e + ] + except (OSError, ValueError, json.JSONDecodeError): + pass + return [] + + +def _save_pending(entries: list[dict]) -> None: + path = _pending_file() + try: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(".json.tmp") + tmp.write_text(json.dumps(entries, indent=2), encoding="utf-8") + os.replace(tmp, path) + except OSError: + # Non-fatal — worst case the user has to run ``hermes debug delete`` + # manually. + pass + + +def _record_pending(urls: list[str], delay_seconds: int = _AUTO_DELETE_SECONDS) -> None: + """Record *urls* for deletion at ``now + delay_seconds``. + + Only paste.rs URLs are recorded (dpaste.com auto-expires). Entries + are merged into any existing pending.json. + """ + paste_rs_urls = [u for u in urls if _extract_paste_id(u)] + if not paste_rs_urls: + return + + entries = _load_pending() + # Dedupe by URL: keep the later expire_at if same URL appears twice + by_url: dict[str, float] = {e["url"]: float(e["expire_at"]) for e in entries} + expire_at = time.time() + delay_seconds + for u in paste_rs_urls: + by_url[u] = max(expire_at, by_url.get(u, 0.0)) + merged = [{"url": u, "expire_at": ts} for u, ts in by_url.items()] + _save_pending(merged) + + +def _sweep_expired_pastes(now: Optional[float] = None) -> tuple[int, int]: + """Synchronously DELETE any pending pastes whose ``expire_at`` has passed. + + Returns ``(deleted, remaining)``. Best-effort: failed deletes stay in + the pending file and will be retried on the next sweep. Silent — + intended to be called from every ``hermes debug`` invocation with + minimal noise. + """ + entries = _load_pending() + if not entries: + return (0, 0) + + current = time.time() if now is None else now + deleted = 0 + remaining: list[dict] = [] + + for entry in entries: + try: + expire_at = float(entry.get("expire_at", 0)) + except (TypeError, ValueError): + continue # drop malformed entries + if expire_at > current: + remaining.append(entry) + continue + + url = entry.get("url", "") + try: + if delete_paste(url): + deleted += 1 + continue + except Exception: + # Network hiccup, 404 (already gone), etc. — drop the entry + # after a grace period; don't retry forever. + pass + + # Retain failed deletes for up to 24h past expiration, then give up. + if expire_at + 86400 > current: + remaining.append(entry) + else: + deleted += 1 # count as reaped (paste.rs will GC eventually) + + if deleted: + _save_pending(remaining) + + return (deleted, len(remaining)) + + +def _best_effort_sweep_expired_pastes() -> None: + """Attempt pending-paste cleanup without letting /debug fail offline.""" + try: + _sweep_expired_pastes() + except Exception: + pass + + +# --------------------------------------------------------------------------- +# Privacy / delete helpers +# --------------------------------------------------------------------------- + +_PRIVACY_NOTICE = """\ +⚠️ This will upload the following to a public paste service: + • System info (OS, Python version, Hermes version, provider, which API keys + are configured — NOT the actual keys) + • Recent log lines (agent.log, errors.log, gateway.log — may contain + conversation fragments and file paths) + • Full agent.log and gateway.log (up to 512 KB each — likely contains + conversation content, tool outputs, and file paths) + +Pastes auto-delete after 6 hours. +""" + +_GATEWAY_PRIVACY_NOTICE = ( + "⚠️ **Privacy notice:** This uploads system info + recent log tails " + "(may contain conversation fragments) to a public paste service. " + "Full logs are NOT included from the gateway — use `hermes debug share` " + "from the CLI for full log uploads.\n" + "Pastes auto-delete after 6 hours." +) + + +def _extract_paste_id(url: str) -> Optional[str]: + """Extract the paste ID from a paste.rs or dpaste.com URL. + + Returns the ID string, or None if the URL doesn't match a known service. + """ + url = url.strip().rstrip("/") + for prefix in ("https://paste.rs/", "http://paste.rs/"): + if url.startswith(prefix): + return url[len(prefix):] + return None + + +def delete_paste(url: str) -> bool: + """Delete a paste from paste.rs. Returns True on success. + + Only paste.rs supports unauthenticated DELETE. dpaste.com pastes + expire automatically but cannot be deleted via API. + """ + paste_id = _extract_paste_id(url) + if not paste_id: + raise ValueError( + f"Cannot delete: only paste.rs URLs are supported. Got: {url}" + ) + + target = f"{_PASTE_RS_URL}{paste_id}" + req = urllib.request.Request( + target, method="DELETE", + headers={"User-Agent": "hermes-agent/debug-share"}, + ) + with urllib.request.urlopen(req, timeout=30) as resp: + return 200 <= resp.status < 300 + + +def _schedule_auto_delete(urls: list[str], delay_seconds: int = _AUTO_DELETE_SECONDS): + """Record *urls* for deletion ``delay_seconds`` from now. + + Previously this spawned a detached Python subprocess per call that slept + for 6 hours and then issued DELETE requests. Those subprocesses leaked — + every ``hermes debug share`` invocation added ~20 MB of resident Python + interpreters that never exited until the sleep completed. + + The replacement is stateless: we append to ``~/.hermes/pastes/pending.json`` + and rely on opportunistic sweeps (``_sweep_expired_pastes``) called from + every ``hermes debug`` invocation. If the user never runs ``hermes debug`` + again, paste.rs's own retention policy handles cleanup. + """ + _record_pending(urls, delay_seconds=delay_seconds) + + +def _delete_hint(url: str) -> str: + """Return a one-liner delete command for the given paste URL.""" + paste_id = _extract_paste_id(url) + if paste_id: + return f"hermes debug delete {url}" + # dpaste.com — no API delete, expires on its own. + return "(auto-expires per dpaste.com policy)" + def _upload_paste_rs(content: str) -> str: """Upload to paste.rs. Returns the paste URL. @@ -112,72 +323,128 @@ def upload_to_pastebin(content: str, expiry_days: int = 7) -> str: # Log file reading # --------------------------------------------------------------------------- -def _resolve_log_path(log_name: str) -> Optional[Path]: - """Find the log file for *log_name*, falling back to the .1 rotation. - Returns the path if found, or None. - """ +@dataclass +class LogSnapshot: + """Single-read snapshot of a log file used by debug-share.""" + + path: Optional[Path] + tail_text: str + full_text: Optional[str] + + +def _primary_log_path(log_name: str) -> Optional[Path]: + """Where *log_name* would live if present. Doesn't check existence.""" from hermes_cli.logs import LOG_FILES filename = LOG_FILES.get(log_name) - if not filename: + return (get_hermes_home() / "logs" / filename) if filename else None + + +def _resolve_log_path(log_name: str) -> Optional[Path]: + """Find the log file for *log_name*, falling back to the .1 rotation. + + Returns the first non-empty candidate (primary, then .1), or None. + Callers distinguish 'empty primary' from 'truly missing' via + :func:`_primary_log_path`. + """ + primary = _primary_log_path(log_name) + if primary is None: return None - log_dir = get_hermes_home() / "logs" - primary = log_dir / filename if primary.exists() and primary.stat().st_size > 0: return primary - # Fall back to the most recent rotated file (.1). - rotated = log_dir / f"{filename}.1" + rotated = primary.parent / f"{primary.name}.1" if rotated.exists() and rotated.stat().st_size > 0: return rotated return None -def _read_log_tail(log_name: str, num_lines: int) -> str: - """Read the last *num_lines* from a log file, or return a placeholder.""" - from hermes_cli.logs import _read_last_n_lines - - log_path = _resolve_log_path(log_name) - if log_path is None: - return "(file not found)" - - try: - lines = _read_last_n_lines(log_path, num_lines) - return "".join(lines).rstrip("\n") - except Exception as exc: - return f"(error reading: {exc})" - +def _capture_log_snapshot( + log_name: str, + *, + tail_lines: int, + max_bytes: int = _MAX_LOG_BYTES, +) -> LogSnapshot: + """Capture a log once and derive summary/full-log views from it. -def _read_full_log(log_name: str, max_bytes: int = _MAX_LOG_BYTES) -> Optional[str]: - """Read a log file for standalone upload. - - Returns the file content (last *max_bytes* if truncated), or None if the - file doesn't exist or is empty. + The report tail and standalone log upload must come from the same file + snapshot. Otherwise a rotation/truncate between reads can make the report + look newer than the uploaded ``agent.log`` paste. """ log_path = _resolve_log_path(log_name) if log_path is None: - return None + primary = _primary_log_path(log_name) + tail = "(file empty)" if primary and primary.exists() else "(file not found)" + return LogSnapshot(path=None, tail_text=tail, full_text=None) try: size = log_path.stat().st_size if size == 0: - return None + # race: file was truncated between _resolve_log_path and stat + return LogSnapshot(path=log_path, tail_text="(file empty)", full_text=None) - if size <= max_bytes: - return log_path.read_text(encoding="utf-8", errors="replace") - - # File is larger than max_bytes — read the tail. with open(log_path, "rb") as f: - f.seek(size - max_bytes) - # Skip partial line at the seek point. - f.readline() - content = f.read().decode("utf-8", errors="replace") - return f"[... truncated — showing last ~{max_bytes // 1024}KB ...]\n{content}" - except Exception: - return None + if size <= max_bytes: + raw = f.read() + truncated = False + else: + # Read from the end until we have enough bytes for the + # standalone upload and enough newline context to render the + # summary tail from the same snapshot. + chunk_size = 8192 + pos = size + chunks: list[bytes] = [] + total = 0 + newline_count = 0 + + while pos > 0 and (total < max_bytes or newline_count <= tail_lines + 1) and total < max_bytes * 2: + read_size = min(chunk_size, pos) + pos -= read_size + f.seek(pos) + chunk = f.read(read_size) + chunks.insert(0, chunk) + total += len(chunk) + newline_count += chunk.count(b"\n") + chunk_size = min(chunk_size * 2, 65536) + + raw = b"".join(chunks) + truncated = pos > 0 + + full_raw = raw + if truncated and len(full_raw) > max_bytes: + cut = len(full_raw) - max_bytes + # Check whether the cut lands exactly on a line boundary. If the + # byte just before the cut position is a newline the first retained + # byte starts a complete line and we should keep it. Only drop a + # partial first line when we're genuinely mid-line. + on_boundary = cut > 0 and full_raw[cut - 1 : cut] == b"\n" + full_raw = full_raw[cut:] + if not on_boundary and b"\n" in full_raw: + full_raw = full_raw.split(b"\n", 1)[1] + + all_text = raw.decode("utf-8", errors="replace") + tail_text = "".join(all_text.splitlines(keepends=True)[-tail_lines:]).rstrip("\n") + + full_text = full_raw.decode("utf-8", errors="replace") + if truncated: + full_text = f"[... truncated — showing last ~{max_bytes // 1024}KB ...]\n{full_text}" + + return LogSnapshot(path=log_path, tail_text=tail_text, full_text=full_text) + except Exception as exc: + return LogSnapshot(path=log_path, tail_text=f"(error reading: {exc})", full_text=None) + + +def _capture_default_log_snapshots(log_lines: int) -> dict[str, LogSnapshot]: + """Capture all logs used by debug-share exactly once.""" + errors_lines = min(log_lines, 100) + return { + "agent": _capture_log_snapshot("agent", tail_lines=log_lines), + "errors": _capture_log_snapshot("errors", tail_lines=errors_lines), + "gateway": _capture_log_snapshot("gateway", tail_lines=errors_lines), + } # --------------------------------------------------------------------------- @@ -203,7 +470,12 @@ class _FakeArgs: return capture.getvalue() -def collect_debug_report(*, log_lines: int = 200, dump_text: str = "") -> str: +def collect_debug_report( + *, + log_lines: int = 200, + dump_text: str = "", + log_snapshots: Optional[dict[str, LogSnapshot]] = None, +) -> str: """Build the summary debug report: system dump + log tails. Parameters @@ -222,19 +494,22 @@ def collect_debug_report(*, log_lines: int = 200, dump_text: str = "") -> str: dump_text = _capture_dump() buf.write(dump_text) + if log_snapshots is None: + log_snapshots = _capture_default_log_snapshots(log_lines) + # ── Recent log tails (summary only) ────────────────────────────────── buf.write("\n\n") buf.write(f"--- agent.log (last {log_lines} lines) ---\n") - buf.write(_read_log_tail("agent", log_lines)) + buf.write(log_snapshots["agent"].tail_text) buf.write("\n\n") errors_lines = min(log_lines, 100) buf.write(f"--- errors.log (last {errors_lines} lines) ---\n") - buf.write(_read_log_tail("errors", errors_lines)) + buf.write(log_snapshots["errors"].tail_text) buf.write("\n\n") buf.write(f"--- gateway.log (last {errors_lines} lines) ---\n") - buf.write(_read_log_tail("gateway", errors_lines)) + buf.write(log_snapshots["gateway"].tail_text) buf.write("\n") return buf.getvalue() @@ -246,18 +521,28 @@ def collect_debug_report(*, log_lines: int = 200, dump_text: str = "") -> str: def run_debug_share(args): """Collect debug report + full logs, upload each, print URLs.""" + _best_effort_sweep_expired_pastes() + log_lines = getattr(args, "lines", 200) expiry = getattr(args, "expire", 7) local_only = getattr(args, "local", False) + if not local_only: + print(_PRIVACY_NOTICE) + print("Collecting debug report...") # Capture dump once — prepended to every paste for context. dump_text = _capture_dump() + log_snapshots = _capture_default_log_snapshots(log_lines) - report = collect_debug_report(log_lines=log_lines, dump_text=dump_text) - agent_log = _read_full_log("agent") - gateway_log = _read_full_log("gateway") + report = collect_debug_report( + log_lines=log_lines, + dump_text=dump_text, + log_snapshots=log_snapshots, + ) + agent_log = log_snapshots["agent"].full_text + gateway_log = log_snapshots["gateway"].full_text # Prepend dump header to each full log so every paste is self-contained. if agent_log: @@ -315,22 +600,66 @@ def run_debug_share(args): if failures: print(f"\n (failed to upload: {', '.join(failures)})") + # Schedule auto-deletion after 6 hours + _schedule_auto_delete(list(urls.values())) + print(f"\n⏱ Pastes will auto-delete in 6 hours.") + + # Manual delete fallback + print(f"To delete now: hermes debug delete ") + print(f"\nShare these links with the Hermes team for support.") +def run_debug_delete(args): + """Delete one or more paste URLs uploaded by /debug.""" + urls = getattr(args, "urls", []) + if not urls: + print("Usage: hermes debug delete [ ...]") + print(" Deletes paste.rs pastes uploaded by 'hermes debug share'.") + return + + for url in urls: + try: + ok = delete_paste(url) + if ok: + print(f" ✓ Deleted: {url}") + else: + print(f" ✗ Failed to delete: {url} (unexpected response)") + except ValueError as exc: + print(f" ✗ {exc}") + except Exception as exc: + print(f" ✗ Could not delete {url}: {exc}") + + def run_debug(args): """Route debug subcommands.""" + # Opportunistic sweep of expired pastes on every ``hermes debug`` call. + # Replaces the old per-paste sleeping subprocess that used to leak as + # one orphaned Python interpreter per scheduled deletion. Silent and + # best-effort — any failure is swallowed so ``hermes debug`` stays + # reliable even when offline. + try: + _sweep_expired_pastes() + except Exception: + pass + subcmd = getattr(args, "debug_command", None) if subcmd == "share": run_debug_share(args) + elif subcmd == "delete": + run_debug_delete(args) else: # Default: show help - print("Usage: hermes debug share [--lines N] [--expire N] [--local]") + print("Usage: hermes debug ") print() print("Commands:") print(" share Upload debug report to a paste service and print URL") + print(" delete Delete a previously uploaded paste") print() - print("Options:") + print("Options (share):") print(" --lines N Number of log lines to include (default: 200)") print(" --expire N Paste expiry in days (default: 7)") print(" --local Print report locally instead of uploading") + print() + print("Options (delete):") + print(" ... One or more paste URLs to delete") diff --git a/hermes_cli/dingtalk_auth.py b/hermes_cli/dingtalk_auth.py new file mode 100644 index 000000000000..e1034c53da62 --- /dev/null +++ b/hermes_cli/dingtalk_auth.py @@ -0,0 +1,294 @@ +""" +DingTalk Device Flow authorization. + +Implements the same 3-step registration flow as dingtalk-openclaw-connector: + 1. POST /app/registration/init → get nonce + 2. POST /app/registration/begin → get device_code + verification_uri_complete + 3. POST /app/registration/poll → poll until SUCCESS → get client_id + client_secret + +The verification_uri_complete is rendered as a QR code in the terminal so the +user can scan it with DingTalk to authorize, yielding AppKey + AppSecret +automatically. +""" + +from __future__ import annotations + +import io +import os +import sys +import time +import logging +from typing import Optional, Tuple + +import requests + +logger = logging.getLogger(__name__) + +# ── Configuration ────────────────────────────────────────────────────────── + +REGISTRATION_BASE_URL = os.environ.get( + "DINGTALK_REGISTRATION_BASE_URL", "https://oapi.dingtalk.com" +).rstrip("/") + +REGISTRATION_SOURCE = os.environ.get("DINGTALK_REGISTRATION_SOURCE", "openClaw") + + +# ── API helpers ──────────────────────────────────────────────────────────── + +class RegistrationError(Exception): + """Raised when a DingTalk registration API call fails.""" + + +def _api_post(path: str, payload: dict) -> dict: + """POST to the registration API and return the parsed JSON body.""" + url = f"{REGISTRATION_BASE_URL}{path}" + try: + resp = requests.post(url, json=payload, timeout=15) + resp.raise_for_status() + data = resp.json() + except requests.RequestException as exc: + raise RegistrationError(f"Network error calling {url}: {exc}") from exc + + errcode = data.get("errcode", -1) + if errcode != 0: + errmsg = data.get("errmsg", "unknown error") + raise RegistrationError(f"API error [{path}]: {errmsg} (errcode={errcode})") + return data + + +# ── Core flow ────────────────────────────────────────────────────────────── + +def begin_registration() -> dict: + """Start a device-flow registration. + + Returns a dict with keys: + device_code, verification_uri_complete, expires_in, interval + """ + # Step 1: init → nonce + init_data = _api_post("/app/registration/init", {"source": REGISTRATION_SOURCE}) + nonce = str(init_data.get("nonce", "")).strip() + if not nonce: + raise RegistrationError("init response missing nonce") + + # Step 2: begin → device_code, verification_uri_complete + begin_data = _api_post("/app/registration/begin", {"nonce": nonce}) + device_code = str(begin_data.get("device_code", "")).strip() + verification_uri_complete = str(begin_data.get("verification_uri_complete", "")).strip() + if not device_code: + raise RegistrationError("begin response missing device_code") + if not verification_uri_complete: + raise RegistrationError("begin response missing verification_uri_complete") + + return { + "device_code": device_code, + "verification_uri_complete": verification_uri_complete, + "expires_in": int(begin_data.get("expires_in", 7200)), + "interval": max(int(begin_data.get("interval", 3)), 2), + } + + +def poll_registration(device_code: str) -> dict: + """Poll the registration status once. + + Returns a dict with keys: status, client_id?, client_secret?, fail_reason? + """ + data = _api_post("/app/registration/poll", {"device_code": device_code}) + status_raw = str(data.get("status", "")).strip().upper() + if status_raw not in ("WAITING", "SUCCESS", "FAIL", "EXPIRED"): + status_raw = "UNKNOWN" + return { + "status": status_raw, + "client_id": str(data.get("client_id", "")).strip() or None, + "client_secret": str(data.get("client_secret", "")).strip() or None, + "fail_reason": str(data.get("fail_reason", "")).strip() or None, + } + + +def wait_for_registration_success( + device_code: str, + interval: int = 3, + expires_in: int = 7200, + on_waiting: Optional[callable] = None, +) -> Tuple[str, str]: + """Block until the registration succeeds or times out. + + Returns (client_id, client_secret). + """ + deadline = time.monotonic() + expires_in + retry_window = 120 # 2 minutes for transient errors + retry_start = 0.0 + + while time.monotonic() < deadline: + time.sleep(interval) + try: + result = poll_registration(device_code) + except RegistrationError: + if retry_start == 0: + retry_start = time.monotonic() + if time.monotonic() - retry_start < retry_window: + continue + raise + + status = result["status"] + if status == "WAITING": + retry_start = 0 + if on_waiting: + on_waiting() + continue + if status == "SUCCESS": + cid = result["client_id"] + csecret = result["client_secret"] + if not cid or not csecret: + raise RegistrationError("authorization succeeded but credentials are missing") + return cid, csecret + # FAIL / EXPIRED / UNKNOWN + if retry_start == 0: + retry_start = time.monotonic() + if time.monotonic() - retry_start < retry_window: + continue + reason = result.get("fail_reason") or status + raise RegistrationError(f"authorization failed: {reason}") + + raise RegistrationError("authorization timed out, please retry") + + +# ── QR code rendering ───────────────────────────────────────────────────── + +def _ensure_qrcode_installed() -> bool: + """Try to import qrcode; if missing, auto-install it via pip/uv.""" + try: + import qrcode # noqa: F401 + return True + except ImportError: + pass + + import subprocess + + # Try uv first (Hermes convention), then pip + for cmd in ( + [sys.executable, "-m", "uv", "pip", "install", "qrcode"], + [sys.executable, "-m", "pip", "install", "-q", "qrcode"], + ): + try: + subprocess.check_call(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + import qrcode # noqa: F401,F811 + return True + except (subprocess.CalledProcessError, ImportError, FileNotFoundError): + continue + return False + + +def render_qr_to_terminal(url: str) -> bool: + """Render *url* as a compact QR code in the terminal. + + Returns True if the QR code was printed, False if the library is missing. + """ + try: + import qrcode + except ImportError: + return False + + qr = qrcode.QRCode( + version=1, + error_correction=qrcode.constants.ERROR_CORRECT_L, + box_size=1, + border=1, + ) + qr.add_data(url) + qr.make(fit=True) + + # Use half-block characters for compact rendering (2 rows per character) + matrix = qr.get_matrix() + rows = len(matrix) + lines: list[str] = [] + + TOP_HALF = "\u2580" # ▀ + BOTTOM_HALF = "\u2584" # ▄ + FULL_BLOCK = "\u2588" # █ + EMPTY = " " + + for r in range(0, rows, 2): + line_chars: list[str] = [] + for c in range(len(matrix[r])): + top = matrix[r][c] + bottom = matrix[r + 1][c] if r + 1 < rows else False + if top and bottom: + line_chars.append(FULL_BLOCK) + elif top: + line_chars.append(TOP_HALF) + elif bottom: + line_chars.append(BOTTOM_HALF) + else: + line_chars.append(EMPTY) + lines.append(" " + "".join(line_chars)) + + print("\n".join(lines)) + return True + + +# ── High-level entry point for the setup wizard ─────────────────────────── + +def dingtalk_qr_auth() -> Optional[Tuple[str, str]]: + """Run the interactive QR-code device-flow authorization. + + Returns (client_id, client_secret) on success, or None if the user + cancelled or the flow failed. + """ + from hermes_cli.setup import print_info, print_success, print_warning, print_error + + print() + print_info(" Initializing DingTalk device authorization...") + print_info(" Note: the scan page is branded 'OpenClaw' — DingTalk's") + print_info(" ecosystem onboarding bridge. Safe to use.") + + try: + reg = begin_registration() + except RegistrationError as exc: + print_error(f" Authorization init failed: {exc}") + return None + + url = reg["verification_uri_complete"] + + # Ensure qrcode library is available (auto-install if missing) + if not _ensure_qrcode_installed(): + print_warning(" qrcode library install failed, will show link only.") + + print() + print_info(" Please scan the QR code below with DingTalk to authorize:") + print() + + if not render_qr_to_terminal(url): + print_warning(f" QR code render failed, please open the link below to authorize:") + + print() + print_info(f" Or open this link manually: {url}") + print() + print_info(" Waiting for QR scan authorization... (timeout: 2 hours)") + + dot_count = 0 + + def _on_waiting(): + nonlocal dot_count + dot_count += 1 + if dot_count % 10 == 0: + sys.stdout.write(".") + sys.stdout.flush() + + try: + client_id, client_secret = wait_for_registration_success( + device_code=reg["device_code"], + interval=reg["interval"], + expires_in=reg["expires_in"], + on_waiting=_on_waiting, + ) + except RegistrationError as exc: + print() + print_error(f" Authorization failed: {exc}") + return None + + print() + print_success(" QR scan authorization successful!") + print_success(f" Client ID: {client_id}") + print_success(f" Client Secret: {client_secret[:8]}{'*' * (len(client_secret) - 8)}") + + return client_id, client_secret diff --git a/hermes_cli/doctor.py b/hermes_cli/doctor.py index a01690cbaaf0..064b1d68d1e2 100644 --- a/hermes_cli/doctor.py +++ b/hermes_cli/doctor.py @@ -8,6 +8,7 @@ import sys import subprocess import shutil +from pathlib import Path from hermes_cli.config import get_project_root, get_hermes_home, get_env_path from hermes_constants import display_hermes_home @@ -29,6 +30,7 @@ from hermes_cli.colors import Colors, color from hermes_constants import OPENROUTER_MODELS_URL +from utils import base_url_host_matches _PROVIDER_ENV_HINTS = ( @@ -42,6 +44,7 @@ "ZAI_API_KEY", "Z_AI_API_KEY", "KIMI_API_KEY", + "KIMI_CN_API_KEY", "MINIMAX_API_KEY", "MINIMAX_CN_API_KEY", "KILOCODE_API_KEY", @@ -275,6 +278,86 @@ def run_doctor(args): config_path = HERMES_HOME / 'config.yaml' if config_path.exists(): check_ok(f"{_DHH}/config.yaml exists") + + # Validate model.provider and model.default values + try: + import yaml as _yaml + cfg = _yaml.safe_load(config_path.read_text(encoding="utf-8")) or {} + model_section = cfg.get("model") or {} + provider_raw = (model_section.get("provider") or "").strip() + provider = provider_raw.lower() + default_model = (model_section.get("default") or model_section.get("model") or "").strip() + + known_providers: set = set() + try: + from hermes_cli.auth import PROVIDER_REGISTRY + known_providers = set(PROVIDER_REGISTRY.keys()) | {"openrouter", "custom", "auto"} + except Exception: + pass + try: + from hermes_cli.auth import resolve_provider as _resolve_provider + except Exception: + _resolve_provider = None + + canonical_provider = provider + if provider and _resolve_provider is not None and provider != "auto": + try: + canonical_provider = _resolve_provider(provider) + except Exception: + canonical_provider = None + + if provider and provider != "auto": + if canonical_provider is None or (known_providers and canonical_provider not in known_providers): + known_list = ", ".join(sorted(known_providers)) if known_providers else "(unavailable)" + check_fail( + f"model.provider '{provider_raw}' is not a recognised provider", + f"(known: {known_list})", + ) + issues.append( + f"model.provider '{provider_raw}' is unknown. " + f"Valid providers: {known_list}. " + f"Fix: run 'hermes config set model.provider '" + ) + + # Warn if model is set to a provider-prefixed name on a provider that doesn't use them + if default_model and "/" in default_model and canonical_provider and canonical_provider not in ("openrouter", "custom", "auto", "ai-gateway", "kilocode", "opencode-zen", "huggingface", "nous"): + check_warn( + f"model.default '{default_model}' uses a vendor/model slug but provider is '{provider_raw}'", + "(vendor-prefixed slugs belong to aggregators like openrouter)", + ) + issues.append( + f"model.default '{default_model}' is vendor-prefixed but model.provider is '{provider_raw}'. " + "Either set model.provider to 'openrouter', or drop the vendor prefix." + ) + + # Check credentials for the configured provider. + # Limit to API-key providers in PROVIDER_REGISTRY — other provider + # types (OAuth, SDK, openrouter/anthropic/custom/auto) have their + # own env-var checks elsewhere in doctor, and get_auth_status() + # returns a bare {logged_in: False} for anything it doesn't + # explicitly dispatch, which would produce false positives. + if canonical_provider and canonical_provider not in ("auto", "custom", "openrouter"): + try: + from hermes_cli.auth import PROVIDER_REGISTRY, get_auth_status + pconfig = PROVIDER_REGISTRY.get(canonical_provider) + if pconfig and getattr(pconfig, "auth_type", "") == "api_key": + status = get_auth_status(canonical_provider) or {} + configured = bool(status.get("configured") or status.get("logged_in") or status.get("api_key")) + if not configured: + check_fail( + f"model.provider '{canonical_provider}' is set but no API key is configured", + "(check ~/.hermes/.env or run 'hermes setup')", + ) + issues.append( + f"No credentials found for provider '{canonical_provider}'. " + f"Run 'hermes setup' or set the provider's API key in {_DHH}/.env, " + f"or switch providers with 'hermes config set model.provider '" + ) + except Exception: + pass + + except Exception as e: + check_warn("Could not validate model/provider config", f"({e})") else: fallback_config = PROJECT_ROOT / 'cli-config.yaml' if fallback_config.exists(): @@ -371,7 +454,11 @@ def run_doctor(args): print(color("◆ Auth Providers", Colors.CYAN, Colors.BOLD)) try: - from hermes_cli.auth import get_nous_auth_status, get_codex_auth_status + from hermes_cli.auth import ( + get_nous_auth_status, + get_codex_auth_status, + get_gemini_oauth_auth_status, + ) nous_status = get_nous_auth_status() if nous_status.get("logged_in"): @@ -386,6 +473,20 @@ def run_doctor(args): check_warn("OpenAI Codex auth", "(not logged in)") if codex_status.get("error"): check_info(codex_status["error"]) + + gemini_status = get_gemini_oauth_auth_status() + if gemini_status.get("logged_in"): + email = gemini_status.get("email") or "" + project = gemini_status.get("project_id") or "" + pieces = [] + if email: + pieces.append(email) + if project: + pieces.append(f"project={project}") + suffix = f" ({', '.join(pieces)})" if pieces else "" + check_ok("Google Gemini OAuth", f"(logged in{suffix})") + else: + check_warn("Google Gemini OAuth", "(not logged in)") except Exception as e: check_warn("Auth provider status", f"(could not check: {e})") @@ -512,7 +613,87 @@ def run_doctor(args): pass _check_gateway_service_linger(issues) - + + # ========================================================================= + # Check: Command installation (hermes bin symlink) + # ========================================================================= + if sys.platform != "win32": + print() + print(color("◆ Command Installation", Colors.CYAN, Colors.BOLD)) + + # Determine the venv entry point location + _venv_bin = None + for _venv_name in ("venv", ".venv"): + _candidate = PROJECT_ROOT / _venv_name / "bin" / "hermes" + if _candidate.exists(): + _venv_bin = _candidate + break + + # Determine the expected command link directory (mirrors install.sh logic) + _prefix = os.environ.get("PREFIX", "") + _is_termux_env = bool(os.environ.get("TERMUX_VERSION")) or "com.termux/files/usr" in _prefix + if _is_termux_env and _prefix: + _cmd_link_dir = Path(_prefix) / "bin" + _cmd_link_display = "$PREFIX/bin" + else: + _cmd_link_dir = Path.home() / ".local" / "bin" + _cmd_link_display = "~/.local/bin" + _cmd_link = _cmd_link_dir / "hermes" + + if _venv_bin is None: + check_warn( + "Venv entry point not found", + "(hermes not in venv/bin/ or .venv/bin/ — reinstall with pip install -e '.[all]')" + ) + manual_issues.append( + f"Reinstall entry point: cd {PROJECT_ROOT} && source venv/bin/activate && pip install -e '.[all]'" + ) + else: + check_ok(f"Venv entry point exists ({_venv_bin.relative_to(PROJECT_ROOT)})") + + # Check the symlink at the command link location + if _cmd_link.is_symlink(): + _target = _cmd_link.resolve() + _expected = _venv_bin.resolve() + if _target == _expected: + check_ok(f"{_cmd_link_display}/hermes → correct target") + else: + check_warn( + f"{_cmd_link_display}/hermes points to wrong target", + f"(→ {_target}, expected → {_expected})" + ) + if should_fix: + _cmd_link.unlink() + _cmd_link.symlink_to(_venv_bin) + check_ok(f"Fixed symlink: {_cmd_link_display}/hermes → {_venv_bin}") + fixed_count += 1 + else: + issues.append(f"Broken symlink at {_cmd_link_display}/hermes — run 'hermes doctor --fix'") + elif _cmd_link.exists(): + # It's a regular file, not a symlink — possibly a wrapper script + check_ok(f"{_cmd_link_display}/hermes exists (non-symlink)") + else: + check_fail( + f"{_cmd_link_display}/hermes not found", + "(hermes command may not work outside the venv)" + ) + if should_fix: + _cmd_link_dir.mkdir(parents=True, exist_ok=True) + _cmd_link.symlink_to(_venv_bin) + check_ok(f"Created symlink: {_cmd_link_display}/hermes → {_venv_bin}") + fixed_count += 1 + + # Check if the link dir is on PATH + _path_dirs = os.environ.get("PATH", "").split(os.pathsep) + if str(_cmd_link_dir) not in _path_dirs: + check_warn( + f"{_cmd_link_display} is not on your PATH", + "(add it to your shell config: export PATH=\"$HOME/.local/bin:$PATH\")" + ) + manual_issues.append(f"Add {_cmd_link_display} to your PATH") + else: + issues.append(f"Missing {_cmd_link_display}/hermes symlink — run 'hermes doctor --fix'") + # ========================================================================= # Check: External tools # ========================================================================= @@ -678,6 +859,16 @@ def run_doctor(args): elif response.status_code == 401: print(f"\r {color('✗', Colors.RED)} OpenRouter API {color('(invalid API key)', Colors.DIM)} ") issues.append("Check OPENROUTER_API_KEY in .env") + elif response.status_code == 402: + print(f"\r {color('✗', Colors.RED)} OpenRouter API {color('(out of credits — payment required)', Colors.DIM)}") + issues.append( + "OpenRouter account has insufficient credits. " + "Fix: run 'hermes config set model.provider ' to switch providers, " + "or fund your OpenRouter account at https://openrouter.ai/settings/credits" + ) + elif response.status_code == 429: + print(f"\r {color('✗', Colors.RED)} OpenRouter API {color('(rate limited)', Colors.DIM)} ") + issues.append("OpenRouter rate limit hit — consider switching to a different provider or waiting") else: print(f"\r {color('✗', Colors.RED)} OpenRouter API {color(f'(HTTP {response.status_code})', Colors.DIM)} ") except Exception as e: @@ -721,17 +912,21 @@ def run_doctor(args): _apikey_providers = [ ("Z.AI / GLM", ("GLM_API_KEY", "ZAI_API_KEY", "Z_AI_API_KEY"), "https://api.z.ai/api/paas/v4/models", "GLM_BASE_URL", True), ("Kimi / Moonshot", ("KIMI_API_KEY",), "https://api.moonshot.ai/v1/models", "KIMI_BASE_URL", True), + ("StepFun Step Plan", ("STEPFUN_API_KEY",), "https://api.stepfun.ai/step_plan/v1/models", "STEPFUN_BASE_URL", True), ("Kimi / Moonshot (China)", ("KIMI_CN_API_KEY",), "https://api.moonshot.cn/v1/models", None, True), + ("Arcee AI", ("ARCEEAI_API_KEY",), "https://api.arcee.ai/api/v1/models", "ARCEE_BASE_URL", True), ("DeepSeek", ("DEEPSEEK_API_KEY",), "https://api.deepseek.com/v1/models", "DEEPSEEK_BASE_URL", True), ("Hugging Face", ("HF_TOKEN",), "https://router.huggingface.co/v1/models", "HF_BASE_URL", True), + ("NVIDIA NIM", ("NVIDIA_API_KEY",), "https://integrate.api.nvidia.com/v1/models", "NVIDIA_BASE_URL", True), ("Alibaba/DashScope", ("DASHSCOPE_API_KEY",), "https://dashscope-intl.aliyuncs.com/compatible-mode/v1/models", "DASHSCOPE_BASE_URL", True), # MiniMax: the /anthropic endpoint doesn't support /models, but the /v1 endpoint does. ("MiniMax", ("MINIMAX_API_KEY",), "https://api.minimax.io/v1/models", "MINIMAX_BASE_URL", True), ("MiniMax (China)", ("MINIMAX_CN_API_KEY",), "https://api.minimaxi.com/v1/models", "MINIMAX_CN_BASE_URL", True), - ("AI Gateway", ("AI_GATEWAY_API_KEY",), "https://ai-gateway.vercel.sh/v1/models", "AI_GATEWAY_BASE_URL", True), + ("Vercel AI Gateway", ("AI_GATEWAY_API_KEY",), "https://ai-gateway.vercel.sh/v1/models", "AI_GATEWAY_BASE_URL", True), ("Kilo Code", ("KILOCODE_API_KEY",), "https://api.kilo.ai/api/gateway/models", "KILOCODE_BASE_URL", True), ("OpenCode Zen", ("OPENCODE_ZEN_API_KEY",), "https://opencode.ai/zen/v1/models", "OPENCODE_ZEN_BASE_URL", True), - ("OpenCode Go", ("OPENCODE_GO_API_KEY",), "https://opencode.ai/zen/go/v1/models", "OPENCODE_GO_BASE_URL", True), + # OpenCode Go has no shared /models endpoint; skip the health check. + ("OpenCode Go", ("OPENCODE_GO_API_KEY",), None, "OPENCODE_GO_BASE_URL", False), ] for _pname, _env_vars, _default_url, _base_env, _supports_health_check in _apikey_providers: _key = "" @@ -748,19 +943,23 @@ def run_doctor(args): print(f" Checking {_pname} API...", end="", flush=True) try: import httpx - _base = os.getenv(_base_env, "") - # Auto-detect Kimi Code keys (sk-kimi-) → api.kimi.com + _base = os.getenv(_base_env, "") if _base_env else "" + # Auto-detect Kimi Code keys (sk-kimi-) → api.kimi.com/coding/v1 + # (OpenAI-compat surface, which exposes /models for health check). if not _base and _key.startswith("sk-kimi-"): _base = "https://api.kimi.com/coding/v1" - # Anthropic-compat endpoints (/anthropic) don't support /models. - # Rewrite to the OpenAI-compat /v1 surface for health checks. + # Anthropic-compat endpoints (/anthropic, api.kimi.com/coding + # with no /v1) don't support /models. Rewrite to the OpenAI-compat + # /v1 surface for health checks. if _base and _base.rstrip("/").endswith("/anthropic"): from agent.auxiliary_client import _to_openai_base_url _base = _to_openai_base_url(_base) + if base_url_host_matches(_base, "api.kimi.com") and _base.rstrip("/").endswith("/coding"): + _base = _base.rstrip("/") + "/v1" _url = (_base.rstrip("/") + "/models") if _base else _default_url _headers = {"Authorization": f"Bearer {_key}"} - if "api.kimi.com" in _url.lower(): - _headers["User-Agent"] = "KimiCLI/1.30.0" + if base_url_host_matches(_base, "api.kimi.com"): + _headers["User-Agent"] = "claude-code/0.1.0" _resp = httpx.get( _url, headers=_headers, @@ -776,6 +975,31 @@ def run_doctor(args): except Exception as _e: print(f"\r {color('⚠', Colors.YELLOW)} {_label} {color(f'({_e})', Colors.DIM)} ") + # -- AWS Bedrock -- + # Bedrock uses the AWS SDK credential chain, not API keys. + try: + from agent.bedrock_adapter import has_aws_credentials, resolve_aws_auth_env_var, resolve_bedrock_region + if has_aws_credentials(): + _auth_var = resolve_aws_auth_env_var() + _region = resolve_bedrock_region() + _label = "AWS Bedrock".ljust(20) + print(f" Checking AWS Bedrock...", end="", flush=True) + try: + import boto3 + _br_client = boto3.client("bedrock", region_name=_region) + _br_resp = _br_client.list_foundation_models() + _model_count = len(_br_resp.get("modelSummaries", [])) + print(f"\r {color('✓', Colors.GREEN)} {_label} {color(f'({_auth_var}, {_region}, {_model_count} models)', Colors.DIM)} ") + except ImportError: + print(f"\r {color('⚠', Colors.YELLOW)} {_label} {color(f'(boto3 not installed — {sys.executable} -m pip install boto3)', Colors.DIM)} ") + issues.append(f"Install boto3 for Bedrock: {sys.executable} -m pip install boto3") + except Exception as _e: + _err_name = type(_e).__name__ + print(f"\r {color('⚠', Colors.YELLOW)} {_label} {color(f'({_err_name}: {_e})', Colors.DIM)} ") + issues.append(f"AWS Bedrock: {_err_name} — check IAM permissions for bedrock:ListFoundationModels") + except ImportError: + pass # bedrock_adapter not available — skip silently + # ========================================================================= # Check: Submodules # ========================================================================= diff --git a/hermes_cli/dump.py b/hermes_cli/dump.py index 491bf6e2c377..90364a261ac3 100644 --- a/hermes_cli/dump.py +++ b/hermes_cli/dump.py @@ -43,41 +43,20 @@ def _redact(value: str) -> str: def _gateway_status() -> str: """Return a short gateway status string.""" - if sys.platform.startswith("linux"): - from hermes_constants import is_container - if is_container(): - try: - from hermes_cli.gateway import find_gateway_pids - pids = find_gateway_pids() - if pids: - return f"running (docker, pid {pids[0]})" - return "stopped (docker)" - except Exception: - return "stopped (docker)" - try: - from hermes_cli.gateway import get_service_name - svc = get_service_name() - except Exception: - svc = "hermes-gateway" - try: - r = subprocess.run( - ["systemctl", "--user", "is-active", svc], - capture_output=True, text=True, timeout=5, - ) - return "running (systemd)" if r.stdout.strip() == "active" else "stopped" - except Exception: - return "unknown" - elif sys.platform == "darwin": - try: - from hermes_cli.gateway import get_launchd_label - r = subprocess.run( - ["launchctl", "list", get_launchd_label()], - capture_output=True, text=True, timeout=5, - ) - return "loaded (launchd)" if r.returncode == 0 else "not loaded" - except Exception: - return "unknown" - return "N/A" + try: + from hermes_cli.gateway import get_gateway_runtime_snapshot + + snapshot = get_gateway_runtime_snapshot() + if snapshot.running: + mode = snapshot.manager + if snapshot.has_process_service_mismatch: + mode = "manual" + return f"running ({mode}, pid {snapshot.gateway_pids[0]})" + if snapshot.service_installed and not snapshot.service_running: + return f"stopped ({snapshot.manager})" + return f"stopped ({snapshot.manager})" + except Exception: + return "unknown" if sys.platform.startswith(("linux", "darwin")) else "N/A" def _count_skills(hermes_home: Path) -> int: @@ -131,6 +110,7 @@ def _configured_platforms() -> list[str]: "wecom": "WECOM_BOT_ID", "wecom_callback": "WECOM_CALLBACK_CORP_ID", "weixin": "WEIXIN_ACCOUNT_ID", + "qqbot": "QQ_APP_ID", } return [name for name, env in checks.items() if os.getenv(env)] @@ -180,7 +160,6 @@ def _config_overrides(config: dict) -> dict[str, str]: ("display", "streaming"), ("display", "skin"), ("display", "show_reasoning"), - ("smart_model_routing", "enabled"), ("privacy", "redact_pii"), ("tts", "provider"), ] @@ -295,6 +274,7 @@ def run_dump(args): ("DEEPSEEK_API_KEY", "deepseek"), ("DASHSCOPE_API_KEY", "dashscope"), ("HF_TOKEN", "huggingface"), + ("NVIDIA_API_KEY", "nvidia"), ("AI_GATEWAY_API_KEY", "ai_gateway"), ("OPENCODE_ZEN_API_KEY", "opencode_zen"), ("OPENCODE_GO_API_KEY", "opencode_go"), diff --git a/hermes_cli/env_loader.py b/hermes_cli/env_loader.py index 8d6a1449d965..009f3de273b2 100644 --- a/hermes_cli/env_loader.py +++ b/hermes_cli/env_loader.py @@ -3,16 +3,94 @@ from __future__ import annotations import os +import sys from pathlib import Path from dotenv import load_dotenv +# Env var name suffixes that indicate credential values. These are the +# only env vars whose values we sanitize on load — we must not silently +# alter arbitrary user env vars, but credentials are known to require +# pure ASCII (they become HTTP header values). +_CREDENTIAL_SUFFIXES = ("_API_KEY", "_TOKEN", "_SECRET", "_KEY") + +# Names we've already warned about during this process, so repeated +# load_hermes_dotenv() calls (user env + project env, gateway hot-reload, +# tests) don't spam the same warning multiple times. +_WARNED_KEYS: set[str] = set() + + +def _format_offending_chars(value: str, limit: int = 3) -> str: + """Return a compact 'U+XXXX ('c'), ...' summary of non-ASCII codepoints.""" + seen: list[str] = [] + for ch in value: + if ord(ch) > 127: + label = f"U+{ord(ch):04X}" + if ch.isprintable(): + label += f" ({ch!r})" + if label not in seen: + seen.append(label) + if len(seen) >= limit: + break + return ", ".join(seen) + + +def _sanitize_loaded_credentials() -> None: + """Strip non-ASCII characters from credential env vars in os.environ. + + Called after dotenv loads so the rest of the codebase never sees + non-ASCII API keys. Only touches env vars whose names end with + known credential suffixes (``_API_KEY``, ``_TOKEN``, etc.). + + Emits a one-line warning to stderr when characters are stripped. + Silent stripping would mask copy-paste corruption (Unicode lookalike + glyphs from PDFs / rich-text editors, ZWSP from web pages) as opaque + provider-side "invalid API key" errors (see #6843). + """ + for key, value in list(os.environ.items()): + if not any(key.endswith(suffix) for suffix in _CREDENTIAL_SUFFIXES): + continue + try: + value.encode("ascii") + continue + except UnicodeEncodeError: + pass + cleaned = value.encode("ascii", errors="ignore").decode("ascii") + os.environ[key] = cleaned + if key in _WARNED_KEYS: + continue + _WARNED_KEYS.add(key) + stripped = len(value) - len(cleaned) + detail = _format_offending_chars(value) or "non-printable" + print( + f" Warning: {key} contained {stripped} non-ASCII character" + f"{'s' if stripped != 1 else ''} ({detail}) — stripped so the " + f"key can be sent as an HTTP header.", + file=sys.stderr, + ) + print( + " This usually means the key was copy-pasted from a PDF, " + "rich-text editor, or web page that substituted lookalike\n" + " Unicode glyphs for ASCII letters. If authentication fails " + "(e.g. \"API key not valid\"), re-copy the key from the\n" + " provider's dashboard and run `hermes setup` (or edit the " + ".env file in a plain-text editor).", + file=sys.stderr, + ) + + def _load_dotenv_with_fallback(path: Path, *, override: bool) -> None: try: load_dotenv(dotenv_path=path, override=override, encoding="utf-8") except UnicodeDecodeError: load_dotenv(dotenv_path=path, override=override, encoding="latin-1") + # Strip non-ASCII characters from credential env vars that were just + # loaded. API keys must be pure ASCII since they're sent as HTTP + # header values (httpx encodes headers as ASCII). Non-ASCII chars + # typically come from copy-pasting keys from PDFs or rich-text editors + # that substitute Unicode lookalike glyphs (e.g. ʋ U+028B for v). + _sanitize_loaded_credentials() def _sanitize_env_file_if_needed(path: Path) -> None: @@ -82,6 +160,8 @@ def load_hermes_dotenv( # Fix corrupted .env files before python-dotenv parses them (#8908). if user_env.exists(): _sanitize_env_file_if_needed(user_env) + if project_env_path and project_env_path.exists(): + _sanitize_env_file_if_needed(project_env_path) if user_env.exists(): _load_dotenv_with_fallback(user_env, override=True) diff --git a/hermes_cli/gateway.py b/hermes_cli/gateway.py index c049c0f96635..3b828fecf594 100644 --- a/hermes_cli/gateway.py +++ b/hermes_cli/gateway.py @@ -10,6 +10,7 @@ import signal import subprocess import sys +from dataclasses import dataclass from pathlib import Path PROJECT_ROOT = Path(__file__).parent.parent.resolve() @@ -41,6 +42,23 @@ # Process Management (for manual gateway runs) # ============================================================================= + +@dataclass(frozen=True) +class GatewayRuntimeSnapshot: + manager: str + service_installed: bool = False + service_running: bool = False + gateway_pids: tuple[int, ...] = () + service_scope: str | None = None + + @property + def running(self) -> bool: + return self.service_running or bool(self.gateway_pids) + + @property + def has_process_service_mismatch(self) -> bool: + return self.service_installed and self.running and not self.service_running + def _get_service_pids() -> set: """Return PIDs currently managed by systemd or launchd gateway services. @@ -157,20 +175,76 @@ def _request_gateway_self_restart(pid: int) -> bool: return True -def find_gateway_pids(exclude_pids: set | None = None, all_profiles: bool = False) -> list: - """Find PIDs of running gateway processes. +def _graceful_restart_via_sigusr1(pid: int, drain_timeout: float) -> bool: + """Send SIGUSR1 to a gateway PID and wait for it to exit gracefully. + + SIGUSR1 is wired in gateway/run.py to ``request_restart(via_service=True)`` + which drains in-flight agent runs (up to ``agent.restart_drain_timeout`` + seconds), then exits with code 75. Both systemd (``Restart=on-failure`` + + ``RestartForceExitStatus=75``) and launchd (``KeepAlive.SuccessfulExit + = false``) relaunch the process after the graceful exit. + + This is the drain-aware alternative to ``systemctl restart`` / ``SIGTERM``, + which SIGKILL in-flight agents after a short timeout. Args: - exclude_pids: PIDs to exclude from the result (e.g. service-managed - PIDs that should not be killed during a stale-process sweep). - all_profiles: When ``True``, return gateway PIDs across **all** - profiles (the pre-7923 global behaviour). ``hermes update`` - needs this because a code update affects every profile. - When ``False`` (default), only PIDs belonging to the current - Hermes profile are returned. + pid: Gateway process PID (systemd MainPID, launchd PID, or bare + process PID). + drain_timeout: Seconds to wait for the process to exit after sending + SIGUSR1. Should be slightly larger than the gateway's + ``agent.restart_drain_timeout`` to allow the drain loop to + finish cleanly. + + Returns: + True if the PID was signalled and exited within the timeout. + False if SIGUSR1 couldn't be sent or the process didn't exit in + time (caller should fall back to a harder restart path). """ - _exclude = exclude_pids or set() - pids = [pid for pid in _get_service_pids() if pid not in _exclude] + if not hasattr(signal, "SIGUSR1"): + return False + if pid <= 0: + return False + try: + os.kill(pid, signal.SIGUSR1) + except ProcessLookupError: + # Already gone — nothing to drain. + return True + except (PermissionError, OSError): + return False + + import time as _time + + deadline = _time.monotonic() + max(drain_timeout, 1.0) + while _time.monotonic() < deadline: + try: + os.kill(pid, 0) # signal 0 — probe liveness + except ProcessLookupError: + return True + except PermissionError: + # Process still exists but we can't signal it. Treat as alive + # so the caller falls back. + pass + _time.sleep(0.5) + # Drain didn't finish in time. + return False + + +def _append_unique_pid(pids: list[int], pid: int | None, exclude_pids: set[int]) -> None: + if pid is None or pid <= 0: + return + if pid == os.getpid() or pid in exclude_pids or pid in pids: + return + pids.append(pid) + + +def _scan_gateway_pids(exclude_pids: set[int], all_profiles: bool = False) -> list[int]: + """Best-effort process-table scan for gateway PIDs. + + This supplements the profile-scoped PID file so status views can still spot + a live gateway when the PID file is stale/missing, and ``--all`` sweeps can + discover gateways outside the current profile. + """ + pids: list[int] = [] patterns = [ "hermes_cli.main gateway", "hermes_cli.main --profile", @@ -203,33 +277,39 @@ def _matches_current_profile(command: str) -> bool: if is_windows(): result = subprocess.run( ["wmic", "process", "get", "ProcessId,CommandLine", "/FORMAT:LIST"], - capture_output=True, text=True, timeout=10 + capture_output=True, + text=True, + timeout=10, ) + if result.returncode != 0: + return [] current_cmd = "" - for line in result.stdout.split('\n'): + for line in result.stdout.split("\n"): line = line.strip() if line.startswith("CommandLine="): current_cmd = line[len("CommandLine="):] elif line.startswith("ProcessId="): pid_str = line[len("ProcessId="):] - if any(p in current_cmd for p in patterns) and (all_profiles or _matches_current_profile(current_cmd)): + if any(p in current_cmd for p in patterns) and ( + all_profiles or _matches_current_profile(current_cmd) + ): try: - pid = int(pid_str) - if pid != os.getpid() and pid not in pids and pid not in _exclude: - pids.append(pid) + _append_unique_pid(pids, int(pid_str), exclude_pids) except ValueError: pass current_cmd = "" else: result = subprocess.run( - ["ps", "eww", "-ax", "-o", "pid=,command="], + ["ps", "-A", "eww", "-o", "pid=,command="], capture_output=True, text=True, timeout=10, ) - for line in result.stdout.split('\n'): + if result.returncode != 0: + return [] + for line in result.stdout.split("\n"): stripped = line.strip() - if not stripped or 'grep' in stripped: + if not stripped or "grep" in stripped: continue pid = None @@ -251,16 +331,278 @@ def _matches_current_profile(command: str) -> bool: if pid is None: continue - if pid == os.getpid() or pid in pids or pid in _exclude: - continue - if any(pattern in command for pattern in patterns) and (all_profiles or _matches_current_profile(command)): - pids.append(pid) + if any(pattern in command for pattern in patterns) and ( + all_profiles or _matches_current_profile(command) + ): + _append_unique_pid(pids, pid, exclude_pids) except (OSError, subprocess.TimeoutExpired): - pass + return [] + + return pids + + +def find_gateway_pids(exclude_pids: set | None = None, all_profiles: bool = False) -> list: + """Find PIDs of running gateway processes. + + Args: + exclude_pids: PIDs to exclude from the result (e.g. service-managed + PIDs that should not be killed during a stale-process sweep). + all_profiles: When ``True``, return gateway PIDs across **all** + profiles (the pre-7923 global behaviour). ``hermes update`` + needs this because a code update affects every profile. + When ``False`` (default), only PIDs belonging to the current + Hermes profile are returned. + """ + _exclude = set(exclude_pids or set()) + pids: list[int] = [] + if not all_profiles: + try: + from gateway.status import get_running_pid + _append_unique_pid(pids, get_running_pid(), _exclude) + except Exception: + pass + for pid in _get_service_pids(): + _append_unique_pid(pids, pid, _exclude) + for pid in _scan_gateway_pids(_exclude, all_profiles=all_profiles): + _append_unique_pid(pids, pid, _exclude) return pids +def _probe_systemd_service_running(system: bool = False) -> tuple[bool, bool]: + selected_system = _select_systemd_scope(system) + unit_exists = get_systemd_unit_path(system=selected_system).exists() + if not unit_exists: + return selected_system, False + try: + result = _run_systemctl( + ["is-active", get_service_name()], + system=selected_system, + capture_output=True, + text=True, + timeout=10, + ) + except (RuntimeError, subprocess.TimeoutExpired): + return selected_system, False + return selected_system, result.stdout.strip() == "active" + + +def _read_systemd_unit_properties( + system: bool = False, + properties: tuple[str, ...] = ( + "ActiveState", + "SubState", + "Result", + "ExecMainStatus", + ), +) -> dict[str, str]: + """Return selected ``systemctl show`` properties for the gateway unit.""" + selected_system = _select_systemd_scope(system) + try: + result = _run_systemctl( + [ + "show", + get_service_name(), + "--no-pager", + "--property", + ",".join(properties), + ], + system=selected_system, + capture_output=True, + text=True, + timeout=10, + ) + except (RuntimeError, subprocess.TimeoutExpired, OSError): + return {} + + if result.returncode != 0: + return {} + + parsed: dict[str, str] = {} + for line in result.stdout.splitlines(): + if "=" not in line: + continue + key, value = line.split("=", 1) + parsed[key] = value.strip() + return parsed + + +def _wait_for_systemd_service_restart( + *, + system: bool = False, + previous_pid: int | None = None, + timeout: float = 60.0, +) -> bool: + """Wait for the gateway service to become active after a restart handoff.""" + import time + + svc = get_service_name() + scope_label = _service_scope_label(system).capitalize() + deadline = time.time() + timeout + + while time.time() < deadline: + props = _read_systemd_unit_properties(system=system) + active_state = props.get("ActiveState", "") + sub_state = props.get("SubState", "") + new_pid = None + try: + from gateway.status import get_running_pid + + new_pid = get_running_pid() + except Exception: + new_pid = None + + if active_state == "active": + if new_pid and (previous_pid is None or new_pid != previous_pid): + print(f"✓ {scope_label} service restarted (PID {new_pid})") + return True + if previous_pid is None: + print(f"✓ {scope_label} service restarted") + return True + + if active_state == "activating" and sub_state == "auto-restart": + time.sleep(1) + continue + + time.sleep(2) + + print( + f"⚠ {scope_label} service did not become active within {int(timeout)}s.\n" + f" Check status: {'sudo ' if system else ''}hermes gateway status\n" + f" Check logs: journalctl {'--user ' if not system else ''}-u {svc} -l --since '2 min ago'" + ) + return False + + +def _recover_pending_systemd_restart(system: bool = False, previous_pid: int | None = None) -> bool: + """Recover a planned service restart that is stuck in systemd state.""" + props = _read_systemd_unit_properties(system=system) + if not props: + return False + + try: + from gateway.status import read_runtime_status + except Exception: + return False + + runtime_state = read_runtime_status() or {} + if not runtime_state.get("restart_requested"): + return False + + active_state = props.get("ActiveState", "") + sub_state = props.get("SubState", "") + exec_main_status = props.get("ExecMainStatus", "") + result = props.get("Result", "") + + if active_state == "activating" and sub_state == "auto-restart": + print("⏳ Service restart already pending — waiting for systemd relaunch...") + return _wait_for_systemd_service_restart( + system=system, + previous_pid=previous_pid, + ) + + if active_state == "failed" and ( + exec_main_status == str(GATEWAY_SERVICE_RESTART_EXIT_CODE) + or result == "exit-code" + ): + svc = get_service_name() + scope_label = _service_scope_label(system).capitalize() + print(f"↻ Clearing failed state for pending {scope_label.lower()} service restart...") + _run_systemctl( + ["reset-failed", svc], + system=system, + check=False, + timeout=30, + ) + _run_systemctl( + ["start", svc], + system=system, + check=False, + timeout=90, + ) + return _wait_for_systemd_service_restart( + system=system, + previous_pid=previous_pid, + ) + + return False + + +def _probe_launchd_service_running() -> bool: + if not get_launchd_plist_path().exists(): + return False + try: + result = subprocess.run( + ["launchctl", "list", get_launchd_label()], + capture_output=True, + text=True, + timeout=10, + ) + except subprocess.TimeoutExpired: + return False + return result.returncode == 0 + + +def get_gateway_runtime_snapshot(system: bool = False) -> GatewayRuntimeSnapshot: + """Return a unified view of gateway liveness for the current profile.""" + gateway_pids = tuple(find_gateway_pids()) + if is_termux(): + return GatewayRuntimeSnapshot( + manager="Termux / manual process", + gateway_pids=gateway_pids, + ) + + from hermes_constants import is_container + + if is_linux() and is_container(): + return GatewayRuntimeSnapshot( + manager="docker (foreground)", + gateway_pids=gateway_pids, + ) + + if supports_systemd_services(): + selected_system, service_running = _probe_systemd_service_running(system=system) + scope_label = _service_scope_label(selected_system) + return GatewayRuntimeSnapshot( + manager=f"systemd ({scope_label})", + service_installed=get_systemd_unit_path(system=selected_system).exists(), + service_running=service_running, + gateway_pids=gateway_pids, + service_scope=scope_label, + ) + + if is_macos(): + return GatewayRuntimeSnapshot( + manager="launchd", + service_installed=get_launchd_plist_path().exists(), + service_running=_probe_launchd_service_running(), + gateway_pids=gateway_pids, + service_scope="launchd", + ) + + return GatewayRuntimeSnapshot( + manager="manual process", + gateway_pids=gateway_pids, + ) + + +def _format_gateway_pids(pids: tuple[int, ...] | list[int], *, limit: int | None = 3) -> str: + rendered = [str(pid) for pid in pids[:limit] if pid > 0] if limit is not None else [str(pid) for pid in pids if pid > 0] + if limit is not None and len(pids) > limit: + rendered.append("...") + return ", ".join(rendered) + + +def _print_gateway_process_mismatch(snapshot: GatewayRuntimeSnapshot) -> None: + if not snapshot.has_process_service_mismatch: + return + print() + print("⚠ Gateway process is running for this profile, but the service is not active") + print(f" PID(s): {_format_gateway_pids(snapshot.gateway_pids, limit=None)}") + print(" This is usually a manual foreground/tmux/nohup run, so `hermes gateway`") + print(" can refuse to start another copy until this process stops.") + + def kill_gateway_processes(force: bool = False, exclude_pids: set | None = None, all_profiles: bool = False) -> int: """Kill any running gateway processes. Returns count killed. @@ -323,7 +665,8 @@ def stop_profile_gateway() -> bool: except (ProcessLookupError, PermissionError): break - remove_pid_file() + if get_running_pid() is None: + remove_pid_file() return True @@ -340,25 +683,44 @@ def _wsl_systemd_operational() -> bool: WSL2 with ``systemd=true`` in wsl.conf has working systemd. WSL2 without it (or WSL1) does not — systemctl commands fail. """ + return _systemd_operational(system=True) + + +def _systemd_operational(system: bool = False) -> bool: + """Return True when the requested systemd scope is usable.""" try: - result = subprocess.run( - ["systemctl", "is-system-running"], - capture_output=True, text=True, timeout=5, + result = _run_systemctl( + ["is-system-running"], + system=system, + capture_output=True, + text=True, + timeout=5, ) # "running", "degraded", "starting" all mean systemd is PID 1 status = result.stdout.strip().lower() return status in ("running", "degraded", "starting", "initializing") - except (FileNotFoundError, subprocess.TimeoutExpired, OSError): + except (RuntimeError, subprocess.TimeoutExpired, OSError): return False +def _container_systemd_operational() -> bool: + """Return True when a container exposes working user or system systemd.""" + if _systemd_operational(system=False): + return True + if _systemd_operational(system=True): + return True + return False + + def supports_systemd_services() -> bool: - if not is_linux() or is_termux() or is_container(): + if not is_linux() or is_termux(): return False if shutil.which("systemctl") is None: return False if is_wsl(): return _wsl_systemd_operational() + if is_container(): + return _container_systemd_operational() return True @@ -453,6 +815,21 @@ def get_systemd_unit_path(system: bool = False) -> Path: return Path.home() / ".config" / "systemd" / "user" / f"{name}.service" +class UserSystemdUnavailableError(RuntimeError): + """Raised when ``systemctl --user`` cannot reach the user D-Bus session. + + Typically hit on fresh RHEL/Debian SSH sessions where linger is disabled + and no user@.service is running, so ``/run/user/$UID/bus`` never exists. + Carries a user-facing remediation message in ``args[0]``. + """ + + +def _user_dbus_socket_path() -> Path: + """Return the expected per-user D-Bus socket path (regardless of existence).""" + xdg = os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}" + return Path(xdg) / "bus" + + def _ensure_user_systemd_env() -> None: """Ensure DBUS_SESSION_BUS_ADDRESS and XDG_RUNTIME_DIR are set for systemctl --user. @@ -475,6 +852,126 @@ def _ensure_user_systemd_env() -> None: os.environ["DBUS_SESSION_BUS_ADDRESS"] = f"unix:path={bus_path}" +def _wait_for_user_dbus_socket(timeout: float = 3.0) -> bool: + """Poll for the user D-Bus socket to appear, up to ``timeout`` seconds. + + Linger-enabled user@.service can take a second or two to spawn the socket + after ``loginctl enable-linger`` runs. Returns True once the socket exists. + """ + import time + + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if _user_dbus_socket_path().exists(): + _ensure_user_systemd_env() + return True + time.sleep(0.2) + return _user_dbus_socket_path().exists() + + +def _preflight_user_systemd(*, auto_enable_linger: bool = True) -> None: + """Ensure ``systemctl --user`` will reach the user D-Bus session bus. + + No-op when the bus socket is already there (the common case on desktops + and linger-enabled servers). On fresh SSH sessions where the socket is + missing: + + * If linger is already enabled, wait briefly for user@.service to spawn + the socket. + * If linger is disabled and ``auto_enable_linger`` is True, try + ``loginctl enable-linger $USER`` (works as non-root when polkit permits + it, otherwise needs sudo). + * If the socket is still missing afterwards, raise + :class:`UserSystemdUnavailableError` with a precise remediation message. + + Callers should treat the exception as a terminal condition for user-scope + systemd operations and surface the message to the user. + """ + _ensure_user_systemd_env() + bus_path = _user_dbus_socket_path() + if bus_path.exists(): + return + + import getpass + + username = getpass.getuser() + linger_enabled, linger_detail = get_systemd_linger_status() + + if linger_enabled is True: + if _wait_for_user_dbus_socket(timeout=3.0): + return + # Linger is on but socket still missing — unusual; fall through to error. + _raise_user_systemd_unavailable( + username, + reason="User D-Bus socket is missing even though linger is enabled.", + fix_hint=( + f" systemctl start user@{os.getuid()}.service\n" + " (may require sudo; try again after the command succeeds)" + ), + ) + + if auto_enable_linger and shutil.which("loginctl"): + try: + result = subprocess.run( + ["loginctl", "enable-linger", username], + capture_output=True, + text=True, + check=False, + timeout=30, + ) + except Exception as exc: + _raise_user_systemd_unavailable( + username, + reason=f"loginctl enable-linger failed ({exc}).", + fix_hint=f" sudo loginctl enable-linger {username}", + ) + else: + if result.returncode == 0: + if _wait_for_user_dbus_socket(timeout=5.0): + print(f"✓ Enabled linger for {username} — user D-Bus now available") + return + # enable-linger succeeded but the socket never appeared. + _raise_user_systemd_unavailable( + username, + reason="Linger was enabled, but the user D-Bus socket did not appear.", + fix_hint=( + " Log out and log back in, then re-run the command.\n" + f" Or reboot and run: systemctl --user start {get_service_name()}" + ), + ) + detail = (result.stderr or result.stdout or f"exit {result.returncode}").strip() + _raise_user_systemd_unavailable( + username, + reason=f"loginctl enable-linger was denied: {detail}", + fix_hint=f" sudo loginctl enable-linger {username}", + ) + + _raise_user_systemd_unavailable( + username, + reason=( + "User D-Bus session is not available " + f"({linger_detail or 'linger disabled'})." + ), + fix_hint=f" sudo loginctl enable-linger {username}", + ) + + +def _raise_user_systemd_unavailable(username: str, *, reason: str, fix_hint: str) -> None: + """Build a user-facing error message and raise UserSystemdUnavailableError.""" + msg = ( + f"{reason}\n" + " systemctl --user cannot reach the user D-Bus session in this shell.\n" + "\n" + " To fix:\n" + f"{fix_hint}\n" + "\n" + " Alternative: run the gateway in the foreground (stays up until\n" + " you exit / close the terminal):\n" + " hermes gateway run" + ) + raise UserSystemdUnavailableError(msg) + + def _systemctl_cmd(system: bool = False) -> list[str]: if not system: _ensure_user_systemd_env() @@ -521,6 +1018,195 @@ def has_conflicting_systemd_units() -> bool: return len(get_installed_systemd_scopes()) > 1 +# Legacy service names from older Hermes installs that predate the +# hermes-gateway rename. Kept as an explicit allowlist (NOT a glob) so +# profile units (hermes-gateway-*.service) and unrelated third-party +# "hermes" units are never matched. +_LEGACY_SERVICE_NAMES: tuple[str, ...] = ("hermes.service",) + +# ExecStart content markers that identify a unit as running our gateway. +# A legacy unit is only flagged when its file contains one of these. +_LEGACY_UNIT_EXECSTART_MARKERS: tuple[str, ...] = ( + "hermes_cli.main gateway", + "hermes_cli/main.py gateway", + "gateway/run.py", + " hermes gateway ", + "/hermes gateway ", +) + + +def _legacy_unit_search_paths() -> list[tuple[bool, Path]]: + """Return ``[(is_system, base_dir), ...]`` — directories to scan for legacy units. + + Factored out so tests can monkeypatch the search roots without touching + real filesystem paths. + """ + return [ + (False, Path.home() / ".config" / "systemd" / "user"), + (True, Path("/etc/systemd/system")), + ] + + +def _find_legacy_hermes_units() -> list[tuple[str, Path, bool]]: + """Return ``[(unit_name, unit_path, is_system)]`` for legacy Hermes gateway units. + + Detects unit files installed by older Hermes versions that used a + different service name (e.g. ``hermes.service`` before the rename to + ``hermes-gateway.service``). When both a legacy unit and the current + ``hermes-gateway.service`` are active, they fight over the same bot + token — the PR #5646 signal-recovery change turns this into a 30-second + SIGTERM flap loop. + + Safety guards: + + * Explicit allowlist of legacy names (no globbing). Profile units such + as ``hermes-gateway-coder.service`` and unrelated third-party + ``hermes-*`` services are never matched. + * ExecStart content check — only flag units that invoke our gateway + entrypoint. A user-created ``hermes.service`` running an unrelated + binary is left untouched. + * Results are returned purely for caller inspection; this function + never mutates or removes anything. + """ + results: list[tuple[str, Path, bool]] = [] + for is_system, base in _legacy_unit_search_paths(): + for name in _LEGACY_SERVICE_NAMES: + unit_path = base / name + try: + if not unit_path.exists(): + continue + text = unit_path.read_text(encoding="utf-8", errors="ignore") + except (OSError, PermissionError): + continue + if not any(marker in text for marker in _LEGACY_UNIT_EXECSTART_MARKERS): + # Not our gateway — leave alone + continue + results.append((name, unit_path, is_system)) + return results + + +def has_legacy_hermes_units() -> bool: + """Return True when any legacy Hermes gateway unit files exist.""" + return bool(_find_legacy_hermes_units()) + + +def print_legacy_unit_warning() -> None: + """Warn about legacy Hermes gateway unit files if any are installed. + + Idempotent: prints nothing when no legacy units are detected. Safe to + call from any status/install/setup path. + """ + legacy = _find_legacy_hermes_units() + if not legacy: + return + print_warning("Legacy Hermes gateway unit(s) detected from an older install:") + for name, path, is_system in legacy: + scope = "system" if is_system else "user" + print_info(f" {path} ({scope} scope)") + print_info(" These run alongside the current hermes-gateway service and") + print_info(" cause SIGTERM flap loops — both try to use the same bot token.") + print_info(" Remove them with:") + print_info(" hermes gateway migrate-legacy") + + +def remove_legacy_hermes_units( + interactive: bool = True, + dry_run: bool = False, +) -> tuple[int, list[Path]]: + """Stop, disable, and remove legacy Hermes gateway unit files. + + Iterates over whatever ``_find_legacy_hermes_units()`` returns — which is + an explicit allowlist of legacy names (not a glob). Profile units and + unrelated third-party services are never touched. + + Args: + interactive: When True, prompt before removing. When False, remove + without asking (used when another prompt has already confirmed, + e.g. from the install flow). + dry_run: When True, list what would be removed and return. + + Returns: + ``(removed_count, remaining_paths)`` — remaining includes units we + couldn't remove (typically system-scope when not running as root). + """ + legacy = _find_legacy_hermes_units() + if not legacy: + print("No legacy Hermes gateway units found.") + return 0, [] + + user_units = [(n, p) for n, p, is_sys in legacy if not is_sys] + system_units = [(n, p) for n, p, is_sys in legacy if is_sys] + + print() + print("Legacy Hermes gateway unit(s) found:") + for name, path, is_system in legacy: + scope = "system" if is_system else "user" + print(f" {path} ({scope} scope)") + print() + + if dry_run: + print("(dry-run — nothing removed)") + return 0, [p for _, p, _ in legacy] + + if interactive and not prompt_yes_no("Remove these legacy units?", True): + print("Skipped. Run again with: hermes gateway migrate-legacy") + return 0, [p for _, p, _ in legacy] + + removed = 0 + remaining: list[Path] = [] + + # User-scope removal + for name, path in user_units: + try: + _run_systemctl(["stop", name], system=False, check=False, timeout=90) + _run_systemctl(["disable", name], system=False, check=False, timeout=30) + path.unlink(missing_ok=True) + print(f" ✓ Removed {path}") + removed += 1 + except (OSError, RuntimeError) as e: + print(f" ⚠ Could not remove {path}: {e}") + remaining.append(path) + + if user_units: + try: + _run_systemctl(["daemon-reload"], system=False, check=False, timeout=30) + except RuntimeError: + pass + + # System-scope removal (needs root) + if system_units: + if os.geteuid() != 0: + print() + print_warning("System-scope legacy units require root to remove.") + print_info(" Re-run with: sudo hermes gateway migrate-legacy") + for _, path in system_units: + remaining.append(path) + else: + for name, path in system_units: + try: + _run_systemctl(["stop", name], system=True, check=False, timeout=90) + _run_systemctl(["disable", name], system=True, check=False, timeout=30) + path.unlink(missing_ok=True) + print(f" ✓ Removed {path}") + removed += 1 + except (OSError, RuntimeError) as e: + print(f" ⚠ Could not remove {path}: {e}") + remaining.append(path) + + try: + _run_systemctl(["daemon-reload"], system=True, check=False, timeout=30) + except RuntimeError: + pass + + print() + if remaining: + print_warning(f"{len(remaining)} legacy unit(s) still present — see messages above.") + else: + print_success(f"Removed {removed} legacy unit(s).") + + return removed, remaining + + def print_systemd_scope_conflict_warning() -> None: scopes = get_installed_systemd_scopes() if len(scopes) < 2: @@ -639,8 +1325,6 @@ def get_systemd_linger_status() -> tuple[bool | None, str]: if not is_linux(): return None, "not supported on this platform" - import shutil - if not shutil.which("loginctl"): return None, "loginctl not found" @@ -715,7 +1399,9 @@ def _detect_venv_dir() -> Path | None: """Detect the active virtualenv directory. Checks ``sys.prefix`` first (works regardless of the directory name), - then falls back to probing common directory names under PROJECT_ROOT. + then ``VIRTUAL_ENV`` env var (covers uv-managed environments where + sys.prefix == sys.base_prefix), then falls back to probing common + directory names under PROJECT_ROOT. Returns ``None`` when no virtualenv can be found. """ # If we're running inside a virtualenv, sys.prefix points to it. @@ -724,6 +1410,15 @@ def _detect_venv_dir() -> Path | None: if venv.is_dir(): return venv + # uv and some other tools set VIRTUAL_ENV without changing sys.prefix. + # This catches `uv run` where sys.prefix == sys.base_prefix but the + # environment IS a venv. (#8620) + _virtual_env = os.environ.get("VIRTUAL_ENV") + if _virtual_env: + venv = Path(_virtual_env) + if venv.is_dir(): + return venv + # Fallback: check common virtualenv directory names under the project root. for candidate in (".venv", "venv"): venv = PROJECT_ROOT / candidate @@ -828,7 +1523,14 @@ def generate_systemd_unit(system: bool = False, run_as_user: str | None = None) path_entries.append(resolved_node_dir) common_bin_paths = ["/usr/local/sbin", "/usr/local/bin", "/usr/sbin", "/usr/bin", "/sbin", "/bin"] - restart_timeout = max(60, int(_get_restart_drain_timeout() or 0)) + # systemd's TimeoutStopSec must exceed the gateway's drain_timeout so + # there's budget left for post-interrupt cleanup (tool subprocess kill, + # adapter disconnect, session DB close) before systemd escalates to + # SIGKILL on the cgroup — otherwise bash/sleep tool-call children left + # by a force-interrupted agent get reaped by systemd instead of us + # (#8202). 30s of headroom covers the worst case we've observed. + _drain_timeout = int(_get_restart_drain_timeout() or 0) + restart_timeout = max(60, _drain_timeout) + 30 if system: username, group_name, home_dir = _system_service_identity(run_as_user) @@ -981,7 +1683,6 @@ def _ensure_linger_enabled() -> None: return import getpass - import shutil username = getpass.getuser() linger_file = Path(f"/var/lib/systemd/linger/{username}") @@ -1043,6 +1744,19 @@ def systemd_install(force: bool = False, system: bool = False, run_as_user: str if system: _require_root_for_system_service("install") + # Offer to remove legacy units (hermes.service from pre-rename installs) + # before installing the new hermes-gateway.service. If both remain, they + # flap-fight for the Telegram bot token on every gateway startup. + # Only removes units matching _LEGACY_SERVICE_NAMES + our ExecStart + # signature — profile units are never touched. + if has_legacy_hermes_units(): + print() + print_legacy_unit_warning() + print() + if prompt_yes_no("Remove the legacy unit(s) before installing?", True): + remove_legacy_hermes_units(interactive=False) + print() + unit_path = get_systemd_unit_path(system=system) scope_flag = " --system" if system else "" @@ -1081,6 +1795,7 @@ def systemd_install(force: bool = False, system: bool = False, run_as_user: str _ensure_linger_enabled() print_systemd_scope_conflict_warning() + print_legacy_unit_warning() def systemd_uninstall(system: bool = False): @@ -1104,6 +1819,11 @@ def systemd_start(system: bool = False): system = _select_systemd_scope(system) if system: _require_root_for_system_service("start") + else: + # Fail fast with actionable guidance if the user D-Bus session is not + # reachable (common on fresh RHEL/Debian SSH sessions without linger). + # Raises UserSystemdUnavailableError with a remediation message. + _preflight_user_systemd() refresh_systemd_unit_if_needed(system=system) _run_systemctl(["start", get_service_name()], system=system, check=True, timeout=30) print(f"✓ {_service_scope_label(system).capitalize()} service started") @@ -1123,19 +1843,64 @@ def systemd_restart(system: bool = False): system = _select_systemd_scope(system) if system: _require_root_for_system_service("restart") + else: + _preflight_user_systemd() refresh_systemd_unit_if_needed(system=system) from gateway.status import get_running_pid pid = get_running_pid() if pid is not None and _request_gateway_self_restart(pid): - print(f"✓ {_service_scope_label(system).capitalize()} service restart requested") + import time + scope_label = _service_scope_label(system).capitalize() + svc = get_service_name() + + # Phase 1: wait for old process to exit (drain + shutdown) + print(f"⏳ {scope_label} service draining active work...") + deadline = time.time() + 90 + while time.time() < deadline: + try: + os.kill(pid, 0) + time.sleep(1) + except (ProcessLookupError, PermissionError): + break # old process is gone + else: + print(f"⚠ Old process (PID {pid}) still alive after 90s") + + # The gateway exits with code 75 for a planned service restart. + # systemd can sit in the RestartSec window or even wedge itself into a + # failed/rate-limited state if the operator asks for another restart in + # the middle of that handoff. Clear any stale failed state and kick the + # unit immediately so `hermes gateway restart` behaves idempotently. + _run_systemctl( + ["reset-failed", svc], + system=system, + check=False, + timeout=30, + ) + _run_systemctl( + ["start", svc], + system=system, + check=False, + timeout=90, + ) + _wait_for_systemd_service_restart(system=system, previous_pid=pid) return + + if _recover_pending_systemd_restart(system=system, previous_pid=pid): + return + + _run_systemctl( + ["reset-failed", get_service_name()], + system=system, + check=False, + timeout=30, + ) _run_systemctl(["reload-or-restart", get_service_name()], system=system, check=True, timeout=90) print(f"✓ {_service_scope_label(system).capitalize()} service restarted") -def systemd_status(deep: bool = False, system: bool = False): +def systemd_status(deep: bool = False, system: bool = False, full: bool = False): system = _select_systemd_scope(system) unit_path = get_systemd_unit_path(system=system) scope_flag = " --system" if system else "" @@ -1149,13 +1914,21 @@ def systemd_status(deep: bool = False, system: bool = False): print_systemd_scope_conflict_warning() print() + if has_legacy_hermes_units(): + print_legacy_unit_warning() + print() + if not systemd_unit_is_current(system=system): print("⚠ Installed gateway service definition is outdated") print(f" Run: {'sudo ' if system else ''}hermes gateway restart{scope_flag} # auto-refreshes the unit") print() + status_cmd = ["status", get_service_name(), "--no-pager"] + if full: + status_cmd.append("-l") + _run_systemctl( - ["status", get_service_name(), "--no-pager"], + status_cmd, system=system, capture_output=False, timeout=10, @@ -1188,6 +1961,19 @@ def systemd_status(deep: bool = False, system: bool = False): for line in runtime_lines: print(f" {line}") + unit_props = _read_systemd_unit_properties(system=system) + active_state = unit_props.get("ActiveState", "") + sub_state = unit_props.get("SubState", "") + exec_main_status = unit_props.get("ExecMainStatus", "") + result_code = unit_props.get("Result", "") + if active_state == "activating" and sub_state == "auto-restart": + print(" ⏳ Restart pending: systemd is waiting to relaunch the gateway") + elif active_state == "failed" and exec_main_status == str(GATEWAY_SERVICE_RESTART_EXIT_CODE): + print(" ⚠ Planned restart is stuck in systemd failed state (exit 75)") + print(f" Run: systemctl {'--user ' if not system else ''}reset-failed {get_service_name()} && {'sudo ' if system else ''}hermes gateway start{scope_flag}") + elif active_state == "failed" and result_code: + print(f" ⚠ Systemd unit result: {result_code}") + if system: print("✓ System service starts at boot without requiring systemd linger") elif deep: @@ -1203,7 +1989,10 @@ def systemd_status(deep: bool = False, system: bool = False): if deep: print() print("Recent logs:") - subprocess.run(_journalctl_cmd(system) + ["-u", get_service_name(), "-n", "20", "--no-pager"], timeout=10) + log_cmd = _journalctl_cmd(system) + ["-u", get_service_name(), "-n", "20", "--no-pager"] + if full: + log_cmd.append("-l") + subprocess.run(log_cmd, timeout=10) # ============================================================================= @@ -1217,7 +2006,6 @@ def get_launchd_label() -> str: def _launchd_domain() -> str: - import os return f"gui/{os.getuid()}" @@ -1634,7 +2422,7 @@ def run_gateway(verbose: int = 0, quiet: bool = False, replace: bool = False): " Create an App-Level Token with scope: connections:write → copy xapp-... token", "3. Add Bot Token Scopes: Features → OAuth & Permissions → Scopes", " Required: chat:write, app_mentions:read, channels:history, channels:read,", - " groups:history, im:history, im:read, im:write, users:read, files:write", + " groups:history, im:history, im:read, im:write, users:read, files:read, files:write", "4. Subscribe to Events: Features → Event Subscriptions → Enable", " Required events: message.im, message.channels, app_mention", " Optional: message.groups (for private channels)", @@ -1913,6 +2701,29 @@ def run_gateway(verbose: int = 0, quiet: bool = False, replace: bool = False): "help": "Phone number or Apple ID to deliver cron results and notifications to."}, ], }, + { + "key": "qqbot", + "label": "QQ Bot", + "emoji": "🐧", + "token_var": "QQ_APP_ID", + "setup_instructions": [ + "1. Register a QQ Bot application at q.qq.com", + "2. Note your App ID and App Secret from the application page", + "3. Enable the required intents (C2C, Group, Guild messages)", + "4. Configure sandbox or publish the bot", + ], + "vars": [ + {"name": "QQ_APP_ID", "prompt": "QQ Bot App ID", "password": False, + "help": "Your QQ Bot App ID from q.qq.com."}, + {"name": "QQ_CLIENT_SECRET", "prompt": "QQ Bot App Secret", "password": True, + "help": "Your QQ Bot App Secret from q.qq.com."}, + {"name": "QQ_ALLOWED_USERS", "prompt": "Allowed user OpenIDs (comma-separated, leave empty for open access)", "password": False, + "is_allowlist": True, + "help": "Optional — restrict DM access to specific user OpenIDs."}, + {"name": "QQBOT_HOME_CHANNEL", "prompt": "Home channel (user/group OpenID for cron delivery, or empty)", "password": False, + "help": "OpenID to deliver cron results and notifications to."}, + ], + }, ] @@ -2122,15 +2933,179 @@ def _setup_sms(): def _setup_dingtalk(): - """Configure DingTalk via the standard platform setup.""" + """Configure DingTalk — QR scan (recommended) or manual credential entry.""" + from hermes_cli.setup import ( + prompt_choice, prompt_yes_no, print_info, print_success, print_warning, + ) + dingtalk_platform = next(p for p in _PLATFORMS if p["key"] == "dingtalk") - _setup_standard_platform(dingtalk_platform) + emoji = dingtalk_platform["emoji"] + label = dingtalk_platform["label"] + + print() + print(color(f" ─── {emoji} {label} Setup ───", Colors.CYAN)) + + existing = get_env_value("DINGTALK_CLIENT_ID") + if existing: + print() + print_success(f"{label} is already configured (Client ID: {existing}).") + if not prompt_yes_no(f" Reconfigure {label}?", False): + return + + print() + method = prompt_choice( + " Choose setup method", + [ + "QR Code Scan (Recommended, auto-obtain Client ID and Client Secret)", + "Manual Input (Client ID and Client Secret)", + ], + default=0, + ) + + if method == 0: + # ── QR-code device-flow authorization ── + try: + from hermes_cli.dingtalk_auth import dingtalk_qr_auth + except ImportError as exc: + print_warning(f" QR auth module failed to load ({exc}), falling back to manual input.") + _setup_standard_platform(dingtalk_platform) + return + + result = dingtalk_qr_auth() + if result is None: + print_warning(" QR auth incomplete, falling back to manual input.") + _setup_standard_platform(dingtalk_platform) + return + + client_id, client_secret = result + save_env_value("DINGTALK_CLIENT_ID", client_id) + save_env_value("DINGTALK_CLIENT_SECRET", client_secret) + save_env_value("DINGTALK_ALLOW_ALL_USERS", "true") + print() + print_success(f"{emoji} {label} configured via QR scan!") + else: + # ── Manual entry ── + _setup_standard_platform(dingtalk_platform) + # Also enable allow-all by default for convenience + if get_env_value("DINGTALK_CLIENT_ID"): + save_env_value("DINGTALK_ALLOW_ALL_USERS", "true") def _setup_wecom(): - """Configure WeCom (Enterprise WeChat) via the standard platform setup.""" - wecom_platform = next(p for p in _PLATFORMS if p["key"] == "wecom") - _setup_standard_platform(wecom_platform) + """Interactive setup for WeCom — scan QR code or manual credential input.""" + print() + print(color(" ─── 💬 WeCom (Enterprise WeChat) Setup ───", Colors.CYAN)) + + existing_bot_id = get_env_value("WECOM_BOT_ID") + existing_secret = get_env_value("WECOM_SECRET") + if existing_bot_id and existing_secret: + print() + print_success("WeCom is already configured.") + if not prompt_yes_no(" Reconfigure WeCom?", False): + return + + # ── Choose setup method ── + print() + method_choices = [ + "Scan QR code to obtain Bot ID and Secret automatically (recommended)", + "Enter existing Bot ID and Secret manually", + ] + method_idx = prompt_choice(" How would you like to set up WeCom?", method_choices, 0) + + bot_id = None + secret = None + + if method_idx == 0: + # ── QR scan flow ── + try: + from gateway.platforms.wecom import qr_scan_for_bot_info + except Exception as exc: + print_error(f" WeCom QR scan import failed: {exc}") + qr_scan_for_bot_info = None + + if qr_scan_for_bot_info is not None: + try: + credentials = qr_scan_for_bot_info() + except KeyboardInterrupt: + print() + print_warning(" WeCom setup cancelled.") + return + except Exception as exc: + print_warning(f" QR scan failed: {exc}") + credentials = None + if credentials: + bot_id = credentials.get("bot_id", "") + secret = credentials.get("secret", "") + print_success(" ✔ QR scan successful! Bot ID and Secret obtained.") + + if not bot_id or not secret: + print_info(" QR scan did not complete. Continuing with manual input.") + bot_id = None + secret = None + + # ── Manual credential input ── + if not bot_id or not secret: + print() + print_info(" 1. Go to WeCom Application → Workspace → Smart Robot -> Create smart robots") + print_info(" 2. Select API Mode") + print_info(" 3. Copy the Bot ID and Secret from the bot's credentials info") + print_info(" 4. The bot connects via WebSocket — no public endpoint needed") + print() + bot_id = prompt(" Bot ID", password=False) + if not bot_id: + print_warning(" Skipped — WeCom won't work without a Bot ID.") + return + secret = prompt(" Secret", password=True) + if not secret: + print_warning(" Skipped — WeCom won't work without a Secret.") + return + + # ── Save core credentials ── + save_env_value("WECOM_BOT_ID", bot_id) + save_env_value("WECOM_SECRET", secret) + + # ── Allowed users (deny-by-default security) ── + print() + print_info(" The gateway DENIES all users by default for security.") + print_info(" Enter user IDs to create an allowlist, or leave empty.") + allowed = prompt(" Allowed user IDs (comma-separated, or empty)", password=False) + if allowed: + cleaned = allowed.replace(" ", "") + save_env_value("WECOM_ALLOWED_USERS", cleaned) + print_success(" Saved — only these users can interact with the bot.") + else: + print() + access_choices = [ + "Enable open access (anyone can message the bot)", + "Use DM pairing (unknown users request access, you approve with 'hermes pairing approve')", + "Disable direct messages", + "Skip for now (bot will deny all users until configured)", + ] + access_idx = prompt_choice(" How should unauthorized users be handled?", access_choices, 1) + if access_idx == 0: + save_env_value("WECOM_DM_POLICY", "open") + save_env_value("GATEWAY_ALLOW_ALL_USERS", "true") + print_warning(" Open access enabled — anyone can use your bot!") + elif access_idx == 1: + save_env_value("WECOM_DM_POLICY", "pairing") + print_success(" DM pairing mode — users will receive a code to request access.") + print_info(" Approve with: hermes pairing approve ") + elif access_idx == 2: + save_env_value("WECOM_DM_POLICY", "disabled") + print_warning(" Direct messages disabled.") + else: + print_info(" Skipped — configure later with 'hermes gateway setup'") + + # ── Home channel (optional) ── + print() + print_info(" Chat ID for scheduled results and notifications.") + home = prompt(" Home chat ID (optional, for cron/notifications)", password=False) + if home: + save_env_value("WECOM_HOME_CHANNEL", home) + print_success(f" Home channel set to {home}") + + print() + print_success("💬 WeCom configured!") def _is_service_installed() -> bool: @@ -2483,6 +3458,116 @@ def _setup_feishu(): print_info(f" Bot: {bot_name}") +def _setup_qqbot(): + """Interactive setup for QQ Bot — scan-to-configure or manual credentials.""" + print() + print(color(" ─── 🐧 QQ Bot Setup ───", Colors.CYAN)) + + existing_app_id = get_env_value("QQ_APP_ID") + existing_secret = get_env_value("QQ_CLIENT_SECRET") + if existing_app_id and existing_secret: + print() + print_success("QQ Bot is already configured.") + if not prompt_yes_no(" Reconfigure QQ Bot?", False): + return + + # ── Choose setup method ── + print() + method_choices = [ + "Scan QR code to add bot automatically (recommended)", + "Enter existing App ID and App Secret manually", + ] + method_idx = prompt_choice(" How would you like to set up QQ Bot?", method_choices, 0) + + credentials = None + used_qr = False + + if method_idx == 0: + # ── QR scan-to-configure ── + try: + from gateway.platforms.qqbot import qr_register + credentials = qr_register() + except KeyboardInterrupt: + print() + print_warning(" QQ Bot setup cancelled.") + return + if credentials: + used_qr = True + if not credentials: + print_info(" QR setup did not complete. Continuing with manual input.") + + # ── Manual credential input ── + if not credentials: + print() + print_info(" Go to https://q.qq.com to register a QQ Bot application.") + print_info(" Note your App ID and App Secret from the application page.") + print() + app_id = prompt(" App ID", password=False) + if not app_id: + print_warning(" Skipped — QQ Bot won't work without an App ID.") + return + app_secret = prompt(" App Secret", password=True) + if not app_secret: + print_warning(" Skipped — QQ Bot won't work without an App Secret.") + return + credentials = {"app_id": app_id.strip(), "client_secret": app_secret.strip(), "user_openid": ""} + + # ── Save core credentials ── + save_env_value("QQ_APP_ID", credentials["app_id"]) + save_env_value("QQ_CLIENT_SECRET", credentials["client_secret"]) + + user_openid = credentials.get("user_openid", "") + + # ── DM security policy ── + print() + access_choices = [ + "Use DM pairing approval (recommended)", + "Allow all direct messages", + "Only allow listed user OpenIDs", + ] + access_idx = prompt_choice(" How should direct messages be authorized?", access_choices, 0) + if access_idx == 0: + save_env_value("QQ_ALLOW_ALL_USERS", "false") + if user_openid: + print() + if prompt_yes_no(f" Add yourself ({user_openid}) to the allow list?", True): + save_env_value("QQ_ALLOWED_USERS", user_openid) + print_success(f" Allow list set to {user_openid}") + else: + save_env_value("QQ_ALLOWED_USERS", "") + else: + save_env_value("QQ_ALLOWED_USERS", "") + print_success(" DM pairing enabled.") + print_info(" Unknown users can request access; approve with `hermes pairing approve`.") + elif access_idx == 1: + save_env_value("QQ_ALLOW_ALL_USERS", "true") + save_env_value("QQ_ALLOWED_USERS", "") + print_warning(" Open DM access enabled for QQ Bot.") + else: + default_allow = user_openid or "" + allowlist = prompt(" Allowed user OpenIDs (comma-separated)", default_allow, password=False).replace(" ", "") + save_env_value("QQ_ALLOW_ALL_USERS", "false") + save_env_value("QQ_ALLOWED_USERS", allowlist) + print_success(" Allowlist saved.") + + # ── Home channel ── + if user_openid: + print() + if prompt_yes_no(f" Use your QQ user ID ({user_openid}) as the home channel?", True): + save_env_value("QQBOT_HOME_CHANNEL", user_openid) + print_success(f" Home channel set to {user_openid}") + else: + print() + home_channel = prompt(" Home channel OpenID (for cron/notifications, or empty)", password=False) + if home_channel: + save_env_value("QQBOT_HOME_CHANNEL", home_channel.strip()) + print_success(f" Home channel set to {home_channel.strip()}") + + print() + print_success("🐧 QQ Bot configured!") + print_info(f" App ID: {credentials['app_id']}") + + def _setup_signal(): """Interactive setup for Signal messenger.""" import shutil @@ -2620,6 +3705,10 @@ def gateway_setup(): print_systemd_scope_conflict_warning() print() + if supports_systemd_services() and has_legacy_hermes_units(): + print_legacy_unit_warning() + print() + if service_installed and service_running: print_success("Gateway service is installed and running.") elif service_installed: @@ -2630,6 +3719,10 @@ def gateway_setup(): systemd_start() elif is_macos(): launchd_start() + except UserSystemdUnavailableError as e: + print_error(" Failed to start — user systemd not reachable:") + for line in str(e).splitlines(): + print(f" {line}") except subprocess.CalledProcessError as e: print_error(f" Failed to start: {e}") else: @@ -2660,8 +3753,14 @@ def gateway_setup(): _setup_signal() elif platform["key"] == "weixin": _setup_weixin() + elif platform["key"] == "dingtalk": + _setup_dingtalk() elif platform["key"] == "feishu": _setup_feishu() + elif platform["key"] == "qqbot": + _setup_qqbot() + elif platform["key"] == "wecom": + _setup_wecom() else: _setup_standard_platform(platform) @@ -2688,6 +3787,10 @@ def gateway_setup(): else: stop_profile_gateway() print_info("Start manually: hermes gateway") + except UserSystemdUnavailableError as e: + print_error(" Restart failed — user systemd not reachable:") + for line in str(e).splitlines(): + print(f" {line}") except subprocess.CalledProcessError as e: print_error(f" Restart failed: {e}") elif service_installed: @@ -2697,6 +3800,10 @@ def gateway_setup(): systemd_start() elif is_macos(): launchd_start() + except UserSystemdUnavailableError as e: + print_error(" Start failed — user systemd not reachable:") + for line in str(e).splitlines(): + print(f" {line}") except subprocess.CalledProcessError as e: print_error(f" Start failed: {e}") else: @@ -2720,6 +3827,10 @@ def gateway_setup(): systemd_start(system=installed_scope == "system") else: launchd_start() + except UserSystemdUnavailableError as e: + print_error(" Start failed — user systemd not reachable:") + for line in str(e).splitlines(): + print(f" {line}") except subprocess.CalledProcessError as e: print_error(f" Start failed: {e}") except subprocess.CalledProcessError as e: @@ -2757,6 +3868,18 @@ def gateway_setup(): def gateway_command(args): """Handle gateway subcommands.""" + try: + return _gateway_command_inner(args) + except UserSystemdUnavailableError as e: + # Clean, actionable message instead of a traceback when the user D-Bus + # session is unreachable (fresh SSH shell, no linger, container, etc.). + print_error("User systemd not reachable:") + for line in str(e).splitlines(): + print(f" {line}") + sys.exit(1) + + +def _gateway_command_inner(args): subcmd = getattr(args, 'gateway_command', None) # Default to run if no subcommand @@ -2841,6 +3964,15 @@ def gateway_command(args): elif subcmd == "start": system = getattr(args, 'system', False) + start_all = getattr(args, 'all', False) + + if start_all: + # Kill all stale gateway processes across all profiles before starting + killed = kill_gateway_processes(all_profiles=True) + if killed: + print(f"✓ Killed {killed} stale gateway process(es) across all profiles") + _wait_for_gateway_exit(timeout=10.0, force_after=5.0) + if is_termux(): print("Gateway service start is not supported on Termux because there is no system service manager.") print("Run manually: hermes gateway") @@ -2926,7 +4058,39 @@ def gateway_command(args): # Try service first, fall back to killing and restarting service_available = False system = getattr(args, 'system', False) + restart_all = getattr(args, 'all', False) service_configured = False + + if restart_all: + # --all: stop every gateway process across all profiles, then start fresh + service_stopped = False + if supports_systemd_services() and (get_systemd_unit_path(system=False).exists() or get_systemd_unit_path(system=True).exists()): + try: + systemd_stop(system=system) + service_stopped = True + except subprocess.CalledProcessError: + pass + elif is_macos() and get_launchd_plist_path().exists(): + try: + launchd_stop() + service_stopped = True + except subprocess.CalledProcessError: + pass + killed = kill_gateway_processes(all_profiles=True) + total = killed + (1 if service_stopped else 0) + if total: + print(f"✓ Stopped {total} gateway process(es) across all profiles") + _wait_for_gateway_exit(timeout=10.0, force_after=5.0) + + # Start the current profile's service fresh + print("Starting gateway...") + if supports_systemd_services() and (get_systemd_unit_path(system=False).exists() or get_systemd_unit_path(system=True).exists()): + systemd_start(system=system) + elif is_macos() and get_launchd_plist_path().exists(): + launchd_start() + else: + run_gateway(verbose=0) + return if supports_systemd_services() and (get_systemd_unit_path(system=False).exists() or get_systemd_unit_path(system=True).exists()): service_configured = True @@ -2979,16 +4143,20 @@ def gateway_command(args): elif subcmd == "status": deep = getattr(args, 'deep', False) + full = getattr(args, 'full', False) system = getattr(args, 'system', False) + snapshot = get_gateway_runtime_snapshot(system=system) # Check for service first if supports_systemd_services() and (get_systemd_unit_path(system=False).exists() or get_systemd_unit_path(system=True).exists()): - systemd_status(deep, system=system) + systemd_status(deep, system=system, full=full) + _print_gateway_process_mismatch(snapshot) elif is_macos() and get_launchd_plist_path().exists(): launchd_status(deep) + _print_gateway_process_mismatch(snapshot) else: # Check for manually running processes - pids = find_gateway_pids() + pids = list(snapshot.gateway_pids) if pids: print(f"✓ Gateway is running (PID: {', '.join(map(str, pids))})") print(" (Running manually, not as a system service)") @@ -3029,3 +4197,14 @@ def gateway_command(args): else: print(" hermes gateway install # Install as user service") print(" sudo hermes gateway install --system # Install as boot-time system service") + + elif subcmd == "migrate-legacy": + # Stop, disable, and remove legacy Hermes gateway unit files from + # pre-rename installs (e.g. hermes.service). Profile units and + # unrelated third-party services are never touched. + dry_run = getattr(args, 'dry_run', False) + yes = getattr(args, 'yes', False) + if not supports_systemd_services() and not is_macos(): + print("Legacy unit migration only applies to systemd-based Linux hosts.") + return + remove_legacy_hermes_units(interactive=not yes, dry_run=dry_run) diff --git a/hermes_cli/hooks.py b/hermes_cli/hooks.py new file mode 100644 index 000000000000..97d9e36b30ea --- /dev/null +++ b/hermes_cli/hooks.py @@ -0,0 +1,385 @@ +"""hermes hooks — inspect and manage shell-script hooks. + +Usage:: + + hermes hooks list + hermes hooks test [--for-tool X] [--payload-file F] + hermes hooks revoke + hermes hooks doctor + +Consent records live under ``~/.hermes/shell-hooks-allowlist.json`` and +hook definitions come from the ``hooks:`` block in ``~/.hermes/config.yaml`` +(the same config read by the CLI / gateway at startup). + +This module is a thin CLI shell over :mod:`agent.shell_hooks`; every +shared concern (payload serialisation, response parsing, allowlist +format) lives there. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +from typing import Any, Dict, List, Optional + + +def hooks_command(args) -> None: + """Entry point for ``hermes hooks`` — dispatches to the requested action.""" + sub = getattr(args, "hooks_action", None) + + if not sub: + print("Usage: hermes hooks {list|test|revoke|doctor}") + print("Run 'hermes hooks --help' for details.") + return + + if sub in ("list", "ls"): + _cmd_list(args) + elif sub == "test": + _cmd_test(args) + elif sub in ("revoke", "remove", "rm"): + _cmd_revoke(args) + elif sub == "doctor": + _cmd_doctor(args) + else: + print(f"Unknown hooks subcommand: {sub}") + + +# --------------------------------------------------------------------------- +# list +# --------------------------------------------------------------------------- + +def _cmd_list(_args) -> None: + from hermes_cli.config import load_config + from agent import shell_hooks + + specs = shell_hooks.iter_configured_hooks(load_config()) + + if not specs: + print("No shell hooks configured in ~/.hermes/config.yaml.") + print("See `hermes hooks --help` or") + print(" website/docs/user-guide/features/hooks.md") + print("for the config schema and worked examples.") + return + + by_event: Dict[str, List] = {} + for spec in specs: + by_event.setdefault(spec.event, []).append(spec) + + allowlist = shell_hooks.load_allowlist() + approved = { + (e.get("event"), e.get("command")) + for e in allowlist.get("approvals", []) + if isinstance(e, dict) + } + + print(f"Configured shell hooks ({len(specs)} total):\n") + + for event in sorted(by_event.keys()): + print(f" [{event}]") + for spec in by_event[event]: + is_approved = (spec.event, spec.command) in approved + status = "✓ allowed" if is_approved else "✗ not allowlisted" + matcher_part = f" matcher={spec.matcher!r}" if spec.matcher else "" + print( + f" - {spec.command}{matcher_part} " + f"(timeout={spec.timeout}s, {status})" + ) + + if is_approved: + entry = shell_hooks.allowlist_entry_for(spec.event, spec.command) + if entry and entry.get("approved_at"): + print(f" approved_at: {entry['approved_at']}") + mtime_now = shell_hooks.script_mtime_iso(spec.command) + mtime_at = entry.get("script_mtime_at_approval") + if mtime_now and mtime_at and mtime_now > mtime_at: + print( + f" ⚠ script modified since approval " + f"(was {mtime_at}, now {mtime_now}) — " + f"run `hermes hooks doctor` to re-validate" + ) + print() + + +# --------------------------------------------------------------------------- +# test +# --------------------------------------------------------------------------- + +# Synthetic kwargs matching the real invoke_hook() call sites — these are +# passed verbatim to agent.shell_hooks.run_once(), which routes them through +# the same _serialize_payload() that production firings use. That way the +# stdin a script sees under `hermes hooks test` and `hermes hooks doctor` +# is identical in shape to what it will see at runtime. +_DEFAULT_PAYLOADS = { + "pre_tool_call": { + "tool_name": "terminal", + "args": {"command": "echo hello"}, + "session_id": "test-session", + "task_id": "test-task", + "tool_call_id": "test-call", + }, + "post_tool_call": { + "tool_name": "terminal", + "args": {"command": "echo hello"}, + "session_id": "test-session", + "task_id": "test-task", + "tool_call_id": "test-call", + "result": '{"output": "hello"}', + }, + "pre_llm_call": { + "session_id": "test-session", + "user_message": "What is the weather?", + "conversation_history": [], + "is_first_turn": True, + "model": "gpt-4", + "platform": "cli", + }, + "post_llm_call": { + "session_id": "test-session", + "model": "gpt-4", + "platform": "cli", + }, + "on_session_start": {"session_id": "test-session"}, + "on_session_end": {"session_id": "test-session"}, + "on_session_finalize": {"session_id": "test-session"}, + "on_session_reset": {"session_id": "test-session"}, + "pre_api_request": { + "session_id": "test-session", + "task_id": "test-task", + "platform": "cli", + "model": "claude-sonnet-4-6", + "provider": "anthropic", + "base_url": "https://api.anthropic.com", + "api_mode": "anthropic_messages", + "api_call_count": 1, + "message_count": 4, + "tool_count": 12, + "approx_input_tokens": 2048, + "request_char_count": 8192, + "max_tokens": 4096, + }, + "post_api_request": { + "session_id": "test-session", + "task_id": "test-task", + "platform": "cli", + "model": "claude-sonnet-4-6", + "provider": "anthropic", + "base_url": "https://api.anthropic.com", + "api_mode": "anthropic_messages", + "api_call_count": 1, + "api_duration": 1.234, + "finish_reason": "stop", + "message_count": 4, + "response_model": "claude-sonnet-4-6", + "usage": {"input_tokens": 2048, "output_tokens": 512}, + "assistant_content_chars": 1200, + "assistant_tool_call_count": 0, + }, + "subagent_stop": { + "parent_session_id": "parent-sess", + "child_role": None, + "child_summary": "Synthetic summary for hooks test", + "child_status": "completed", + "duration_ms": 1234, + }, +} + + +def _cmd_test(args) -> None: + from hermes_cli.config import load_config + from hermes_cli.plugins import VALID_HOOKS + from agent import shell_hooks + + event = args.event + if event not in VALID_HOOKS: + print(f"Unknown event: {event!r}") + print(f"Valid events: {', '.join(sorted(VALID_HOOKS))}") + return + + # Synthetic kwargs in the same shape invoke_hook() would pass. Merged + # with --for-tool (overrides tool_name) and --payload-file (extra kwargs). + payload = dict(_DEFAULT_PAYLOADS.get(event, {"session_id": "test-session"})) + + if getattr(args, "for_tool", None): + payload["tool_name"] = args.for_tool + + if getattr(args, "payload_file", None): + try: + custom = json.loads(Path(args.payload_file).read_text()) + if isinstance(custom, dict): + payload.update(custom) + else: + print(f"Warning: {args.payload_file} is not a JSON object; ignoring") + except Exception as exc: + print(f"Error reading payload file: {exc}") + return + + specs = shell_hooks.iter_configured_hooks(load_config()) + specs = [s for s in specs if s.event == event] + + if getattr(args, "for_tool", None): + specs = [ + s for s in specs + if s.event not in ("pre_tool_call", "post_tool_call") + or s.matches_tool(args.for_tool) + ] + + if not specs: + print(f"No shell hooks configured for event: {event}") + if getattr(args, "for_tool", None): + print(f"(with matcher filter --for-tool={args.for_tool})") + return + + print(f"Firing {len(specs)} hook(s) for event '{event}':\n") + for spec in specs: + print(f" → {spec.command}") + result = shell_hooks.run_once(spec, payload) + _print_run_result(result) + print() + + +def _print_run_result(result: Dict[str, Any]) -> None: + if result.get("error"): + print(f" ✗ error: {result['error']}") + return + if result.get("timed_out"): + print(f" ✗ timed out after {result['elapsed_seconds']}s") + return + + rc = result.get("returncode") + elapsed = result.get("elapsed_seconds", 0) + print(f" exit={rc} elapsed={elapsed}s") + + stdout = (result.get("stdout") or "").strip() + stderr = (result.get("stderr") or "").strip() + if stdout: + print(f" stdout: {_truncate(stdout, 400)}") + if stderr: + print(f" stderr: {_truncate(stderr, 400)}") + + parsed = result.get("parsed") + if parsed: + print(f" parsed (Hermes wire shape): {json.dumps(parsed)}") + else: + print(" parsed: ") + + +def _truncate(s: str, n: int) -> str: + return s if len(s) <= n else s[: n - 3] + "..." + + +# --------------------------------------------------------------------------- +# revoke +# --------------------------------------------------------------------------- + +def _cmd_revoke(args) -> None: + from agent import shell_hooks + + removed = shell_hooks.revoke(args.command) + if removed == 0: + print(f"No allowlist entry found for command: {args.command}") + return + print(f"Removed {removed} allowlist entry/entries for: {args.command}") + print( + "Note: currently running CLI / gateway processes keep their " + "already-registered callbacks until they restart." + ) + + +# --------------------------------------------------------------------------- +# doctor +# --------------------------------------------------------------------------- + +def _cmd_doctor(_args) -> None: + from hermes_cli.config import load_config + from agent import shell_hooks + + specs = shell_hooks.iter_configured_hooks(load_config()) + + if not specs: + print("No shell hooks configured — nothing to check.") + return + + print(f"Checking {len(specs)} configured shell hook(s)...\n") + + problems = 0 + for spec in specs: + print(f" [{spec.event}] {spec.command}") + problems += _doctor_one(spec, shell_hooks) + print() + + if problems: + print(f"{problems} issue(s) found. Fix before relying on these hooks.") + else: + print("All shell hooks look healthy.") + + +def _doctor_one(spec, shell_hooks) -> int: + problems = 0 + + # 1. Script exists and is executable + if shell_hooks.script_is_executable(spec.command): + print(" ✓ script exists and is executable") + else: + problems += 1 + print(" ✗ script missing or not executable " + "(chmod +x the file, or fix the path)") + + # 2. Allowlist status + entry = shell_hooks.allowlist_entry_for(spec.event, spec.command) + if entry: + print(f" ✓ allowlisted (approved {entry.get('approved_at', '?')})") + else: + problems += 1 + print(" ✗ not allowlisted — hook will NOT fire at runtime " + "(run with --accept-hooks once, or confirm at the TTY prompt)") + + # 3. Mtime drift + if entry and entry.get("script_mtime_at_approval"): + mtime_now = shell_hooks.script_mtime_iso(spec.command) + mtime_at = entry["script_mtime_at_approval"] + if mtime_now and mtime_at and mtime_now > mtime_at: + problems += 1 + print(f" ⚠ script modified since approval " + f"(was {mtime_at}, now {mtime_now}) — review changes, " + f"then `hermes hooks revoke` + re-approve to refresh") + elif mtime_now and mtime_at and mtime_now == mtime_at: + print(" ✓ script unchanged since approval") + + # 4. Produces valid JSON for a synthetic payload — only when the entry + # is already allowlisted. Otherwise `hermes hooks doctor` would execute + # every script listed in a freshly-pulled config before the user has + # reviewed them, which directly contradicts the documented workflow + # ("spot newly-added hooks *before they register*"). + if not entry: + print(" ℹ skipped JSON smoke test — not allowlisted yet. " + "Approve the hook first (via TTY prompt or --accept-hooks), " + "then re-run `hermes hooks doctor`.") + elif shell_hooks.script_is_executable(spec.command): + payload = _DEFAULT_PAYLOADS.get(spec.event, {"extra": {}}) + result = shell_hooks.run_once(spec, payload) + if result.get("timed_out"): + problems += 1 + print(f" ✗ timed out after {result['elapsed_seconds']}s " + f"on synthetic payload (timeout={spec.timeout}s)") + elif result.get("error"): + problems += 1 + print(f" ✗ execution error: {result['error']}") + else: + rc = result.get("returncode") + elapsed = result.get("elapsed_seconds", 0) + stdout = (result.get("stdout") or "").strip() + if stdout: + try: + json.loads(stdout) + print(f" ✓ produced valid JSON on synthetic payload " + f"(exit={rc}, {elapsed}s)") + except json.JSONDecodeError: + problems += 1 + print(f" ✗ stdout was not valid JSON (exit={rc}, " + f"{elapsed}s): {_truncate(stdout, 120)}") + else: + print(f" ✓ ran clean with empty stdout " + f"(exit={rc}, {elapsed}s) — hook is observer-only") + + return problems diff --git a/hermes_cli/main.py b/hermes_cli/main.py index 97281d5a95c8..d7de309607da 100644 --- a/hermes_cli/main.py +++ b/hermes_cli/main.py @@ -45,11 +45,26 @@ import argparse import os +import shutil import subprocess import sys from pathlib import Path from typing import Optional +def _add_accept_hooks_flag(parser) -> None: + """Attach the ``--accept-hooks`` flag. Shared across every agent + subparser so the flag works regardless of CLI position.""" + parser.add_argument( + "--accept-hooks", + action="store_true", + default=argparse.SUPPRESS, + help=( + "Auto-approve unseen shell hooks without a TTY prompt " + "(equivalent to HERMES_ACCEPT_HOOKS=1 / hooks_auto_accept: true)." + ), + ) + + def _require_tty(command_name: str) -> None: """Exit with a clear error if stdin is not a terminal. @@ -71,6 +86,7 @@ def _require_tty(command_name: str) -> None: PROJECT_ROOT = Path(__file__).parent.parent.resolve() sys.path.insert(0, str(PROJECT_ROOT)) + # --------------------------------------------------------------------------- # Profile override — MUST happen before any hermes module import. # @@ -101,6 +117,7 @@ def _apply_profile_override() -> None: if profile_name is None: try: from hermes_constants import get_default_hermes_root + active_path = get_default_hermes_root() / "active_profile" if active_path.exists(): name = active_path.read_text().strip() @@ -114,13 +131,17 @@ def _apply_profile_override() -> None: if profile_name is not None: try: from hermes_cli.profiles import resolve_profile_env + hermes_home = resolve_profile_env(profile_name) except (ValueError, FileNotFoundError) as exc: print(f"Error: {exc}", file=sys.stderr) sys.exit(1) except Exception as exc: # A bug in profiles.py must NEVER prevent hermes from starting - print(f"Warning: profile override failed ({exc}), using default", file=sys.stderr) + print( + f"Warning: profile override failed ({exc}), using default", + file=sys.stderr, + ) return os.environ["HERMES_HOME"] = hermes_home # Strip the flag from argv so argparse doesn't choke @@ -128,25 +149,28 @@ def _apply_profile_override() -> None: for i, arg in enumerate(argv): if arg in ("--profile", "-p"): start = i + 1 # +1 because argv is sys.argv[1:] - sys.argv = sys.argv[:start] + sys.argv[start + consume:] + sys.argv = sys.argv[:start] + sys.argv[start + consume :] break elif arg.startswith("--profile="): start = i + 1 - sys.argv = sys.argv[:start] + sys.argv[start + 1:] + sys.argv = sys.argv[:start] + sys.argv[start + 1 :] break + _apply_profile_override() # Load .env from ~/.hermes/.env first, then project root as dev fallback. # User-managed env files should override stale shell exports on restart. from hermes_cli.config import get_hermes_home from hermes_cli.env_loader import load_hermes_dotenv -load_hermes_dotenv(project_env=PROJECT_ROOT / '.env') + +load_hermes_dotenv(project_env=PROJECT_ROOT / ".env") # Initialize centralized file logging early — all `hermes` subcommands # (chat, setup, gateway, config, etc.) write to agent.log + errors.log. try: from hermes_logging import setup_logging as _setup_logging + _setup_logging(mode="cli") except Exception: pass # best-effort — don't crash the CLI if logging setup fails @@ -155,6 +179,7 @@ def _apply_profile_override() -> None: try: from hermes_cli.config import load_config as _load_config_early from hermes_constants import apply_ipv4_preference as _apply_ipv4 + _early_cfg = _load_config_early() _net = _early_cfg.get("network", {}) if isinstance(_net, dict) and _net.get("force_ipv4"): @@ -168,7 +193,7 @@ def _apply_profile_override() -> None: from datetime import datetime from hermes_cli import __version__, __release_date__ -from hermes_constants import OPENROUTER_BASE_URL +from hermes_constants import AI_GATEWAY_BASE_URL, OPENROUTER_BASE_URL logger = logging.getLogger(__name__) @@ -201,6 +226,7 @@ def _has_any_provider_configured() -> bool: # tool credentials (Claude Code, Codex CLI) that shouldn't silently skip # the setup wizard on a fresh install. from hermes_cli.config import DEFAULT_CONFIG + _DEFAULT_MODEL = DEFAULT_CONFIG.get("model", "") cfg = load_config() model_cfg = cfg.get("model") @@ -218,7 +244,13 @@ def _has_any_provider_configured() -> bool: from hermes_cli.auth import PROVIDER_REGISTRY # Collect all provider env vars - provider_env_vars = {"OPENROUTER_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY", "ANTHROPIC_TOKEN", "OPENAI_BASE_URL"} + provider_env_vars = { + "OPENROUTER_API_KEY", + "OPENAI_API_KEY", + "ANTHROPIC_API_KEY", + "ANTHROPIC_TOKEN", + "OPENAI_BASE_URL", + } for pconfig in PROVIDER_REGISTRY.values(): if pconfig.auth_type == "api_key": provider_env_vars.update(pconfig.api_key_env_vars) @@ -256,6 +288,7 @@ def _has_any_provider_configured() -> bool: if auth_file.exists(): try: import json + auth = json.loads(auth_file.read_text()) active = auth.get("active_provider") if active: @@ -265,7 +298,6 @@ def _has_any_provider_configured() -> bool: except Exception: pass - # Check config.yaml — if model is a dict with an explicit provider set, # the user has gone through setup (fresh installs have model as a plain # string). Also covers custom endpoints that store api_key/base_url in @@ -282,9 +314,15 @@ def _has_any_provider_configured() -> bool: # being installed doesn't mean the user wants Hermes to use their tokens. if _has_hermes_config: try: - from agent.anthropic_adapter import read_claude_code_credentials, is_claude_code_token_valid + from agent.anthropic_adapter import ( + read_claude_code_credentials, + is_claude_code_token_valid, + ) + creds = read_claude_code_credentials() - if creds and (is_claude_code_token_valid(creds) or creds.get("refreshToken")): + if creds and ( + is_claude_code_token_valid(creds) or creds.get("refreshToken") + ): return True except Exception: pass @@ -346,10 +384,10 @@ def _curses_browse(stdscr): if curses.has_colors(): curses.start_color() curses.use_default_colors() - curses.init_pair(1, curses.COLOR_GREEN, -1) # selected + curses.init_pair(1, curses.COLOR_GREEN, -1) # selected curses.init_pair(2, curses.COLOR_YELLOW, -1) # header - curses.init_pair(3, curses.COLOR_CYAN, -1) # search - curses.init_pair(4, 8, -1) # dim + curses.init_pair(3, curses.COLOR_CYAN, -1) # search + curses.init_pair(4, 8, -1) # dim cursor = 0 scroll_offset = 0 @@ -390,7 +428,9 @@ def _curses_browse(stdscr): name_width = max(20, max_x - fixed_cols) col_header = f" {'Title / Preview':<{name_width}} {'Active':<10} {'Src':<5} {'ID'}" try: - dim_attr = curses.color_pair(4) if curses.has_colors() else curses.A_DIM + dim_attr = ( + curses.color_pair(4) if curses.has_colors() else curses.A_DIM + ) stdscr.addnstr(1, 0, col_header, max_x - 1, dim_attr) except curses.error: pass @@ -417,10 +457,12 @@ def _curses_browse(stdscr): elif cursor >= scroll_offset + visible_rows: scroll_offset = cursor - visible_rows + 1 - for draw_i, i in enumerate(range( - scroll_offset, - min(len(filtered), scroll_offset + visible_rows) - )): + for draw_i, i in enumerate( + range( + scroll_offset, + min(len(filtered), scroll_offset + visible_rows), + ) + ): y = draw_i + 3 if y >= max_y - 1: break @@ -446,18 +488,23 @@ def _curses_browse(stdscr): else: footer = f" 0/{len(sessions)} sessions" try: - stdscr.addnstr(footer_y, 0, footer, max_x - 1, - curses.color_pair(4) if curses.has_colors() else curses.A_DIM) + stdscr.addnstr( + footer_y, + 0, + footer, + max_x - 1, + curses.color_pair(4) if curses.has_colors() else curses.A_DIM, + ) except curses.error: pass stdscr.refresh() key = stdscr.getch() - if key in (curses.KEY_UP, ): + if key in (curses.KEY_UP,): if filtered: cursor = (cursor - 1) % len(filtered) - elif key in (curses.KEY_DOWN, ): + elif key in (curses.KEY_DOWN,): if filtered: cursor = (cursor + 1) % len(filtered) elif key in (curses.KEY_ENTER, 10, 13): @@ -483,7 +530,7 @@ def _curses_browse(stdscr): filtered = list(sessions) cursor = 0 scroll_offset = 0 - elif key == ord('q') and not search_text: + elif key == ord("q") and not search_text: return elif 32 <= key <= 126: # Printable character → add to search filter @@ -526,12 +573,13 @@ def _curses_browse(stdscr): return None -def _resolve_last_cli_session() -> Optional[str]: - """Look up the most recent CLI session ID from SQLite. Returns None if unavailable.""" +def _resolve_last_session(source: str = "cli") -> Optional[str]: + """Look up the most recent session ID for a source.""" try: from hermes_state import SessionDB + db = SessionDB() - sessions = db.search_sessions(source="cli", limit=1) + sessions = db.search_sessions(source=source, limit=1) db.close() if sessions: return sessions[0]["id"] @@ -570,7 +618,6 @@ def _exec_in_container(container_info: dict, cli_args: list): container_info: dict with backend, container_name, exec_user, hermes_bin cli_args: the original CLI arguments (everything after 'hermes') """ - import shutil backend = container_info["backend"] container_name = container_info["container_name"] @@ -579,8 +626,10 @@ def _exec_in_container(container_info: dict, cli_args: list): runtime = shutil.which(backend) if not runtime: - print(f"Error: {backend} not found on PATH. Cannot route to container.", - file=sys.stderr) + print( + f"Error: {backend} not found on PATH. Cannot route to container.", + file=sys.stderr, + ) sys.exit(1) # Rootful containers (NixOS systemd service) are invisible to unprivileged @@ -588,14 +637,16 @@ def _exec_in_container(container_info: dict, cli_args: list): # Probe whether the runtime can see the container; if not, try via sudo. sudo_path = None probe = _probe_container( - [runtime, "inspect", "--format", "ok", container_name], backend, + [runtime, "inspect", "--format", "ok", container_name], + backend, ) if probe.returncode != 0: sudo_path = shutil.which("sudo") if sudo_path: probe2 = _probe_container( [sudo_path, "-n", runtime, "inspect", "--format", "ok", container_name], - backend, via_sudo=True, + backend, + via_sudo=True, ) if probe2.returncode != 0: print( @@ -608,10 +659,10 @@ def _exec_in_container(container_info: dict, cli_args: list): f"\n" f"On NixOS:\n" f"\n" - f' security.sudo.extraRules = [{{\n' + f" security.sudo.extraRules = [{{\n" f' users = [ "{os.getenv("USER", "your-user")}" ];\n' f' commands = [{{ command = "{runtime}"; options = [ "NOPASSWD" ]; }}];\n' - f' }}];\n' + f" }}];\n" f"\n" f"Or run: sudo hermes {' '.join(cli_args)}", file=sys.stderr, @@ -636,7 +687,8 @@ def _exec_in_container(container_info: dict, cli_args: list): cmd_prefix = [sudo_path, "-n", runtime] if sudo_path else [runtime] exec_cmd = ( - cmd_prefix + ["exec"] + cmd_prefix + + ["exec"] + tty_flags + ["-u", exec_user] + env_flags @@ -653,29 +705,347 @@ def _resolve_session_by_name_or_id(name_or_id: str) -> Optional[str]: - If it looks like a session ID (contains underscore + hex), try direct lookup first. - Otherwise, treat it as a title and use resolve_session_by_title (auto-latest). - Falls back to the other method if the first doesn't match. + - If the resolved session is a compression root, follow the chain forward + to the latest continuation. Users who remember the old root ID (e.g. + from an exit summary printed before the bug fix, or from notes) get + resumed at the live tip instead of a stale parent with no messages. """ try: from hermes_state import SessionDB + db = SessionDB() # Try as exact session ID first session = db.get_session(name_or_id) + resolved_id: Optional[str] = None if session: - db.close() - return session["id"] + resolved_id = session["id"] + else: + # Try as title (with auto-latest for lineage) + resolved_id = db.resolve_session_by_title(name_or_id) + + if resolved_id: + # Project forward through compression chain so resumes land on + # the live tip instead of a dead compressed parent. + try: + resolved_id = db.get_compression_tip(resolved_id) or resolved_id + except Exception: + pass - # Try as title (with auto-latest for lineage) - session_id = db.resolve_session_by_title(name_or_id) db.close() - return session_id + return resolved_id except Exception: pass return None +def _print_tui_exit_summary(session_id: Optional[str]) -> None: + """Print a shell-visible epilogue after TUI exits.""" + target = session_id or _resolve_last_session(source="tui") + if not target: + return + + db = None + try: + from hermes_state import SessionDB + + db = SessionDB() + session = db.get_session(target) + if not session: + return + + title = db.get_session_title(target) + message_count = int(session.get("message_count") or 0) + input_tokens = int(session.get("input_tokens") or 0) + output_tokens = int(session.get("output_tokens") or 0) + cache_read_tokens = int(session.get("cache_read_tokens") or 0) + cache_write_tokens = int(session.get("cache_write_tokens") or 0) + reasoning_tokens = int(session.get("reasoning_tokens") or 0) + total_tokens = ( + input_tokens + + output_tokens + + cache_read_tokens + + cache_write_tokens + + reasoning_tokens + ) + except Exception: + return + finally: + if db is not None: + db.close() + + print() + print("Resume this session with:") + print(f" hermes --tui --resume {target}") + if title: + print(f' hermes --tui -c "{title}"') + print() + print(f"Session: {target}") + if title: + print(f"Title: {title}") + print(f"Messages: {message_count}") + print( + "Tokens: " + f"{total_tokens} (in {input_tokens}, out {output_tokens}, " + f"cache {cache_read_tokens + cache_write_tokens}, reasoning {reasoning_tokens})" + ) + + +def _tui_need_npm_install(root: Path) -> bool: + """True when @hermes/ink is missing or node_modules is behind package-lock.json (post-pull).""" + ink = root / "node_modules" / "@hermes" / "ink" / "package.json" + if not ink.is_file(): + return True + lock = root / "package-lock.json" + if not lock.is_file(): + return False + marker = root / "node_modules" / ".package-lock.json" + if not marker.is_file(): + return True + return lock.stat().st_mtime > marker.stat().st_mtime + + +def _find_bundled_tui(tui_dir: Path) -> Optional[Path]: + """Directory whose dist/entry.js we should run: HERMES_TUI_DIR first, else repo ui-tui.""" + env = os.environ.get("HERMES_TUI_DIR") + if env: + p = Path(env) + if (p / "dist" / "entry.js").exists() and not _tui_need_npm_install(p): + return p + if (tui_dir / "dist" / "entry.js").exists() and not _tui_need_npm_install(tui_dir): + return tui_dir + return None + + +def _tui_build_needed(tui_dir: Path) -> bool: + entry = tui_dir / "dist" / "entry.js" + if not entry.exists(): + return True + dist_m = entry.stat().st_mtime + skip = frozenset({"node_modules", "dist"}) + for dirpath, dirnames, filenames in os.walk(tui_dir, topdown=True): + dirnames[:] = [d for d in dirnames if d not in skip] + for fn in filenames: + if fn.endswith((".ts", ".tsx")): + if os.path.getmtime(os.path.join(dirpath, fn)) > dist_m: + return True + for meta in ( + "package.json", + "package-lock.json", + "tsconfig.json", + "tsconfig.build.json", + ): + mp = tui_dir / meta + if mp.exists() and mp.stat().st_mtime > dist_m: + return True + return False + + +def _hermes_ink_bundle_stale(tui_dir: Path) -> bool: + ink_root = tui_dir / "packages" / "hermes-ink" + bundle = ink_root / "dist" / "ink-bundle.js" + if not bundle.exists(): + return True + bm = bundle.stat().st_mtime + skip = frozenset({"node_modules", "dist"}) + for dirpath, dirnames, filenames in os.walk(ink_root, topdown=True): + dirnames[:] = [d for d in dirnames if d not in skip] + for fn in filenames: + if fn.endswith((".ts", ".tsx")): + if os.path.getmtime(os.path.join(dirpath, fn)) > bm: + return True + mp = ink_root / "package.json" + if mp.exists() and mp.stat().st_mtime > bm: + return True + return False + + +def _ensure_tui_node() -> None: + """Make sure `node` + `npm` are on PATH for the TUI. + + If either is missing and scripts/lib/node-bootstrap.sh is available, source + it and call `ensure_node` (fnm/nvm/proto/brew/bundled cascade). After + install, capture the resolved node binary path from the bash subprocess + and prepend its directory to os.environ["PATH"] so shutil.which finds the + new binaries in this Python process — regardless of which version manager + was used (nvm, fnm, proto, brew, or the bundled fallback). + + Idempotent no-op when node+npm are already discoverable. Set + ``HERMES_SKIP_NODE_BOOTSTRAP=1`` to disable auto-install. + """ + if shutil.which("node") and shutil.which("npm"): + return + if os.environ.get("HERMES_SKIP_NODE_BOOTSTRAP"): + return + + helper = PROJECT_ROOT / "scripts" / "lib" / "node-bootstrap.sh" + if not helper.is_file(): + return + + hermes_home = os.environ.get("HERMES_HOME") or str(Path.home() / ".hermes") + try: + # Helper writes logs to stderr; we ask bash to print `command -v node` + # on stdout once ensure_node succeeds. Subshell PATH edits don't leak + # back into Python, so the stdout capture is the bridge. + result = subprocess.run( + [ + "bash", + "-c", + f'source "{helper}" >&2 && ensure_node >&2 && command -v node', + ], + env={**os.environ, "HERMES_HOME": hermes_home}, + capture_output=True, + text=True, + check=False, + ) + except (OSError, subprocess.SubprocessError): + return + + parts = os.environ.get("PATH", "").split(os.pathsep) + extras: list[Path] = [] + + resolved = (result.stdout or "").strip() + if resolved: + extras.append(Path(resolved).resolve().parent) + + extras.extend([Path(hermes_home) / "node" / "bin", Path.home() / ".local" / "bin"]) + + for extra in extras: + s = str(extra) + if extra.is_dir() and s not in parts: + parts.insert(0, s) + os.environ["PATH"] = os.pathsep.join(parts) + + +def _make_tui_argv(tui_dir: Path, tui_dev: bool) -> tuple[list[str], Path]: + """TUI: --dev → tsx src; else node dist (HERMES_TUI_DIR or ui-tui, build when stale).""" + _ensure_tui_node() + + def _node_bin(bin: str) -> str: + if bin == "node": + env_node = os.environ.get("HERMES_NODE") + if env_node and os.path.isfile(env_node) and os.access(env_node, os.X_OK): + return env_node + path = shutil.which(bin) + if not path: + print(f"{bin} not found — install Node.js to use the TUI.") + sys.exit(1) + return path + + # pre-built dist + node_modules (nix / full HERMES_TUI_DIR) skips npm. + if not tui_dev: + ext_dir = os.environ.get("HERMES_TUI_DIR") + if ext_dir: + p = Path(ext_dir) + if (p / "dist" / "entry.js").exists() and not _tui_need_npm_install(p): + node = _node_bin("node") + return [node, str(p / "dist" / "entry.js")], p + + npm = _node_bin("npm") + if _tui_need_npm_install(tui_dir): + if not os.environ.get("HERMES_QUIET"): + print("Installing TUI dependencies…") + result = subprocess.run( + [npm, "install", "--silent", "--no-fund", "--no-audit", "--progress=false"], + cwd=str(tui_dir), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + text=True, + env={**os.environ, "CI": "1"}, + ) + if result.returncode != 0: + err = (result.stderr or "").strip() + preview = "\n".join(err.splitlines()[-30:]) + print("npm install failed.") + if preview: + print(preview) + sys.exit(1) + + if tui_dev: + if _hermes_ink_bundle_stale(tui_dir): + result = subprocess.run( + [npm, "run", "build", "--prefix", "packages/hermes-ink"], + cwd=str(tui_dir), + capture_output=True, + text=True, + ) + if result.returncode != 0: + combined = f"{result.stdout or ''}{result.stderr or ''}".strip() + preview = "\n".join(combined.splitlines()[-30:]) + print("@hermes/ink build failed.") + if preview: + print(preview) + sys.exit(1) + tsx = tui_dir / "node_modules" / ".bin" / "tsx" + if tsx.exists(): + return [str(tsx), "src/entry.tsx"], tui_dir + return [npm, "start"], tui_dir + + if _tui_build_needed(tui_dir): + result = subprocess.run( + [npm, "run", "build"], + cwd=str(tui_dir), + capture_output=True, + text=True, + ) + if result.returncode != 0: + combined = f"{result.stdout or ''}{result.stderr or ''}".strip() + preview = "\n".join(combined.splitlines()[-30:]) + print("TUI build failed.") + if preview: + print(preview) + sys.exit(1) + + root = _find_bundled_tui(tui_dir) + if not root: + print("TUI build did not produce dist/entry.js") + sys.exit(1) + + node = _node_bin("node") + return [node, str(root / "dist" / "entry.js")], root + + +def _launch_tui(resume_session_id: Optional[str] = None, tui_dev: bool = False): + """Replace current process with the TUI.""" + tui_dir = PROJECT_ROOT / "ui-tui" + + env = os.environ.copy() + env["HERMES_PYTHON_SRC_ROOT"] = os.environ.get( + "HERMES_PYTHON_SRC_ROOT", str(PROJECT_ROOT) + ) + env.setdefault("HERMES_PYTHON", sys.executable) + env.setdefault("HERMES_CWD", os.getcwd()) + # Guarantee an 8GB V8 heap + exposed GC for the TUI. Default node cap is + # ~1.5–4GB depending on version and can fatal-OOM on long sessions with + # large transcripts / reasoning blobs. Token-level merge: respect any + # user-supplied --max-old-space-size (they may have set it higher) and + # avoid duplicating --expose-gc. + _tokens = env.get("NODE_OPTIONS", "").split() + if not any(t.startswith("--max-old-space-size=") for t in _tokens): + _tokens.append("--max-old-space-size=8192") + if "--expose-gc" not in _tokens: + _tokens.append("--expose-gc") + env["NODE_OPTIONS"] = " ".join(_tokens) + if resume_session_id: + env["HERMES_TUI_RESUME"] = resume_session_id + + argv, cwd = _make_tui_argv(tui_dir, tui_dev) + try: + code = subprocess.call(argv, cwd=str(cwd), env=env) + except KeyboardInterrupt: + code = 130 + + if code in (0, 130): + _print_tui_exit_summary(resume_session_id) + + sys.exit(code) + + def cmd_chat(args): """Run interactive chat CLI.""" - # Resolve --continue into --resume with the latest CLI session or by name + use_tui = getattr(args, "tui", False) or os.environ.get("HERMES_TUI") == "1" + + # Resolve --continue into --resume with the latest session or by name continue_val = getattr(args, "continue_last", None) if continue_val and not getattr(args, "resume", None): if isinstance(continue_val, str): @@ -689,11 +1059,15 @@ def cmd_chat(args): sys.exit(1) else: # -c with no argument — continue the most recent session - last_id = _resolve_last_cli_session() + source = "tui" if use_tui else "cli" + last_id = _resolve_last_session(source=source) + if not last_id and source == "tui": + last_id = _resolve_last_session(source="cli") if last_id: args.resume = last_id else: - print("No previous CLI session found to continue.") + kind = "TUI" if use_tui else "CLI" + print(f"No previous {kind} session found to continue.") sys.exit(1) # Resolve --resume by title if it's not a direct session ID @@ -708,12 +1082,17 @@ def cmd_chat(args): # First-run guard: check if any provider is configured before launching if not _has_any_provider_configured(): print() - print("It looks like Hermes isn't configured yet -- no API keys or providers found.") + print( + "It looks like Hermes isn't configured yet -- no API keys or providers found." + ) print() print(" Run: hermes setup") print() - from hermes_cli.setup import is_interactive_stdin, print_noninteractive_setup_guidance + from hermes_cli.setup import ( + is_interactive_stdin, + print_noninteractive_setup_guidance, + ) if not is_interactive_stdin(): print_noninteractive_setup_guidance( @@ -735,6 +1114,7 @@ def cmd_chat(args): # Start update check in background (runs while other init happens) try: from hermes_cli.banner import prefetch_update_check + prefetch_update_check() except Exception: pass @@ -742,6 +1122,7 @@ def cmd_chat(args): # Sync bundled skills on every CLI launch (fast -- skips unchanged skills) try: from tools.skills_sync import sync_skills + sync_skills(quiet=True) except Exception: pass @@ -750,13 +1131,33 @@ def cmd_chat(args): if getattr(args, "yolo", False): os.environ["HERMES_YOLO_MODE"] = "1" + # --ignore-user-config: make load_cli_config() / load_config() skip the + # user's ~/.hermes/config.yaml and return built-in defaults. Set BEFORE + # importing cli (which runs `CLI_CONFIG = load_cli_config()` at module + # import time). Credentials in .env are still loaded — this flag only + # ignores behavioral/config settings. + if getattr(args, "ignore_user_config", False): + os.environ["HERMES_IGNORE_USER_CONFIG"] = "1" + + # --ignore-rules: skip auto-injection of AGENTS.md/SOUL.md/.cursorrules + # (rules), memory entries, and any preloaded skills coming from user config. + # Maps to AIAgent(skip_context_files=True, skip_memory=True). + if getattr(args, "ignore_rules", False): + os.environ["HERMES_IGNORE_RULES"] = "1" + # --source: tag session source for filtering (e.g. 'tool' for third-party integrations) if getattr(args, "source", None): os.environ["HERMES_SESSION_SOURCE"] = args.source + if use_tui: + _launch_tui( + getattr(args, "resume", None), + tui_dev=getattr(args, "tui_dev", False), + ) + # Import and run the CLI from cli import main as cli_main - + # Build kwargs from args kwargs = { "model": args.model, @@ -772,10 +1173,12 @@ def cmd_chat(args): "checkpoints": getattr(args, "checkpoints", False), "pass_session_id": getattr(args, "pass_session_id", False), "max_turns": getattr(args, "max_turns", None), + "ignore_rules": getattr(args, "ignore_rules", False), + "ignore_user_config": getattr(args, "ignore_user_config", False), } # Filter out None values kwargs = {k: v for k, v in kwargs.items() if v is not None} - + try: cli_main(**kwargs) except ValueError as e: @@ -786,14 +1189,13 @@ def cmd_chat(args): def cmd_gateway(args): """Gateway management commands.""" from hermes_cli.gateway import gateway_command + gateway_command(args) def cmd_whatsapp(args): """Set up WhatsApp: choose mode, configure, install bridge, pair via QR.""" _require_tty("whatsapp") - import subprocess - from pathlib import Path from hermes_cli.config import get_env_value, save_env_value print() @@ -808,7 +1210,9 @@ def cmd_whatsapp(args): print() print(" 1. Separate bot number (recommended)") print(" People message the bot's number directly — cleanest experience.") - print(" Requires a second phone number with WhatsApp installed on a device.") + print( + " Requires a second phone number with WhatsApp installed on a device." + ) print() print(" 2. Personal number (self-chat)") print(" You message yourself to talk to the agent.") @@ -843,7 +1247,9 @@ def cmd_whatsapp(args): print(" ✓ Mode: personal number (self-chat)") else: wa_mode = current_mode - mode_label = "separate bot number" if wa_mode == "bot" else "personal number (self-chat)" + mode_label = ( + "separate bot number" if wa_mode == "bot" else "personal number (self-chat)" + ) print(f"\n✓ Mode: {mode_label}") # ── Step 2: Enable WhatsApp ────────────────────────────────────────── @@ -865,7 +1271,9 @@ def cmd_whatsapp(args): response = "n" if response.lower() in ("y", "yes"): if wa_mode == "bot": - phone = input(" Phone numbers that can message the bot (comma-separated): ").strip() + phone = input( + " Phone numbers that can message the bot (comma-separated): " + ).strip() else: phone = input(" Your phone number (e.g. 15551234567): ").strip() if phone: @@ -875,7 +1283,9 @@ def cmd_whatsapp(args): print() if wa_mode == "bot": print(" Who should be allowed to message the bot?") - phone = input(" Phone numbers (comma-separated, or * for anyone): ").strip() + phone = input( + " Phone numbers (comma-separated, or * for anyone): " + ).strip() else: phone = input(" Your phone number (e.g. 15551234567): ").strip() if phone: @@ -894,16 +1304,27 @@ def cmd_whatsapp(args): return if not (bridge_dir / "node_modules").exists(): - print("\n→ Installing WhatsApp bridge dependencies...") - result = subprocess.run( - ["npm", "install"], - cwd=str(bridge_dir), - capture_output=True, - text=True, - timeout=120, - ) + print("\n→ Installing WhatsApp bridge dependencies (this can take a few minutes)...") + npm = shutil.which("npm") + if not npm: + print(" ✗ npm not found on PATH — install Node.js first") + return + try: + result = subprocess.run( + [npm, "install", "--no-fund", "--no-audit", "--progress=false"], + cwd=str(bridge_dir), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + text=True, + ) + except KeyboardInterrupt: + print("\n ✗ Install cancelled") + return if result.returncode != 0: - print(f" ✗ npm install failed: {result.stderr}") + err = (result.stderr or "").strip() + preview = "\n".join(err.splitlines()[-30:]) if err else "(no output)" + print(" ✗ npm install failed:") + print(preview) return print(" ✓ Dependencies installed") else: @@ -916,11 +1337,12 @@ def cmd_whatsapp(args): if (session_dir / "creds.json").exists(): print("✓ Existing WhatsApp session found") try: - response = input("\n Re-pair? This will clear the existing session. [y/N] ").strip() + response = input( + "\n Re-pair? This will clear the existing session. [y/N] " + ).strip() except (EOFError, KeyboardInterrupt): response = "n" if response.lower() in ("y", "yes"): - import shutil shutil.rmtree(session_dir, ignore_errors=True) session_dir.mkdir(parents=True, exist_ok=True) print(" ✓ Session cleared") @@ -979,6 +1401,7 @@ def cmd_whatsapp(args): def cmd_setup(args): """Interactive setup wizard.""" from hermes_cli.setup import run_setup_wizard + run_setup_wizard(args) @@ -997,9 +1420,15 @@ def select_provider_and_model(args=None): persistence. """ from hermes_cli.auth import ( - resolve_provider, AuthError, format_auth_error, + resolve_provider, + AuthError, + format_auth_error, + ) + from hermes_cli.config import ( + get_compatible_custom_providers, + load_config, + get_env_value, ) - from hermes_cli.config import get_compatible_custom_providers, load_config, get_env_value config = load_config() current_model = config.get("model") @@ -1009,16 +1438,13 @@ def select_provider_and_model(args=None): # Read effective provider the same way the CLI does at startup: # config.yaml model.provider > env var > auto-detect - import os config_provider = None model_cfg = config.get("model") if isinstance(model_cfg, dict): config_provider = model_cfg.get("provider") effective_provider = ( - config_provider - or os.getenv("HERMES_INFERENCE_PROVIDER") - or "auto" + config_provider or os.getenv("HERMES_INFERENCE_PROVIDER") or "auto" ) try: active = resolve_provider(effective_provider) @@ -1034,29 +1460,9 @@ def select_provider_and_model(args=None): if active == "openrouter" and get_env_value("OPENAI_BASE_URL"): active = "custom" - provider_labels = { - "openrouter": "OpenRouter", - "nous": "Nous Portal", - "openai-codex": "OpenAI Codex", - "qwen-oauth": "Qwen OAuth", - "copilot-acp": "GitHub Copilot ACP", - "copilot": "GitHub Copilot", - "anthropic": "Anthropic", - "gemini": "Google AI Studio", - "zai": "Z.AI / GLM", - "kimi-coding": "Kimi / Moonshot", - "kimi-coding-cn": "Kimi / Moonshot (China)", - "minimax": "MiniMax", - "minimax-cn": "MiniMax (China)", - "opencode-zen": "OpenCode Zen", - "opencode-go": "OpenCode Go", - "ai-gateway": "AI Gateway", - "kilocode": "Kilo Code", - "alibaba": "Alibaba Cloud (DashScope)", - "huggingface": "Hugging Face", - "xiaomi": "Xiaomi MiMo", - "custom": "Custom endpoint", - } + from hermes_cli.models import CANONICAL_PROVIDERS, _PROVIDER_LABELS + + provider_labels = dict(_PROVIDER_LABELS) # derive from canonical list active_label = provider_labels.get(active, active) if active else "none" print() @@ -1064,32 +1470,8 @@ def select_provider_and_model(args=None): print(f" Active provider: {active_label}") print() - # Step 1: Provider selection — top providers shown first, rest behind "More..." - top_providers = [ - ("nous", "Nous Portal (Nous Research subscription)"), - ("openrouter", "OpenRouter (100+ models, pay-per-use)"), - ("anthropic", "Anthropic (Claude models — API key or Claude Code)"), - ("openai-codex", "OpenAI Codex"), - ("qwen-oauth", "Qwen OAuth (reuses local Qwen CLI login)"), - ("copilot", "GitHub Copilot (uses GITHUB_TOKEN or gh auth token)"), - ("huggingface", "Hugging Face Inference Providers (20+ open models)"), - ] - - extended_providers = [ - ("copilot-acp", "GitHub Copilot ACP (spawns `copilot --acp --stdio`)"), - ("gemini", "Google AI Studio (Gemini models — OpenAI-compatible endpoint)"), - ("zai", "Z.AI / GLM (Zhipu AI direct API)"), - ("kimi-coding", "Kimi / Moonshot (Moonshot AI direct API)"), - ("kimi-coding-cn", "Kimi / Moonshot China (Moonshot CN direct API)"), - ("minimax", "MiniMax (global direct API)"), - ("minimax-cn", "MiniMax China (domestic direct API)"), - ("kilocode", "Kilo Code (Kilo Gateway API)"), - ("opencode-zen", "OpenCode Zen (35+ curated models, pay-as-you-go)"), - ("opencode-go", "OpenCode Go (open models, $10/month subscription)"), - ("ai-gateway", "AI Gateway (Vercel — 200+ models, pay-per-use)"), - ("alibaba", "Alibaba Cloud / DashScope Coding (Qwen + multi-provider)"), - ("xiaomi", "Xiaomi MiMo (MiMo-V2 models — pro, omni, flash)"), - ] + # Step 1: Provider selection — flat list from CANONICAL_PROVIDERS + all_providers = [(p.slug, p.tui_desc) for p in CANONICAL_PROVIDERS] def _named_custom_provider_map(cfg) -> dict[str, dict[str, str]]: custom_provider_map = {} @@ -1119,40 +1501,39 @@ def _named_custom_provider_map(cfg) -> dict[str, dict[str, str]]: return custom_provider_map # Add user-defined custom providers from config.yaml - _custom_provider_map = _named_custom_provider_map(config) # key → {name, base_url, api_key} + _custom_provider_map = _named_custom_provider_map( + config + ) # key → {name, base_url, api_key} for key, provider_info in _custom_provider_map.items(): name = provider_info["name"] base_url = provider_info["base_url"] short_url = base_url.replace("https://", "").replace("http://", "").rstrip("/") saved_model = provider_info.get("model", "") model_hint = f" — {saved_model}" if saved_model else "" - top_providers.append((key, f"{name} ({short_url}){model_hint}")) - - top_keys = {k for k, _ in top_providers} - extended_keys = {k for k, _ in extended_providers} - - # If the active provider is in the extended list, promote it into top - if active and active in extended_keys: - promoted = [(k, l) for k, l in extended_providers if k == active] - extended_providers = [(k, l) for k, l in extended_providers if k != active] - top_providers = promoted + top_providers - top_keys.add(active) + all_providers.append((key, f"{name} ({short_url}){model_hint}")) - # Build the primary menu + # Build the menu ordered = [] default_idx = 0 - for key, label in top_providers: + for key, label in all_providers: if active and key == active: ordered.append((key, f"{label} ← currently active")) default_idx = len(ordered) - 1 else: ordered.append((key, label)) - ordered.append(("more", "More providers...")) - ordered.append(("cancel", "Cancel")) + ordered.append(("custom", "Custom endpoint (enter URL manually)")) + _has_saved_custom_list = isinstance(config.get("custom_providers"), list) and bool( + config.get("custom_providers") + ) + if _has_saved_custom_list: + ordered.append(("remove-custom", "Remove a saved custom provider")) + ordered.append(("aux-config", "Configure auxiliary models...")) + ordered.append(("cancel", "Leave unchanged")) provider_idx = _prompt_provider_choice( - [label for _, label in ordered], default=default_idx, + [label for _, label in ordered], + default=default_idx, ) if provider_idx is None or ordered[provider_idx][0] == "cancel": print("No change.") @@ -1160,39 +1541,33 @@ def _named_custom_provider_map(cfg) -> dict[str, dict[str, str]]: selected_provider = ordered[provider_idx][0] - # "More providers..." — show the extended list - if selected_provider == "more": - ext_ordered = list(extended_providers) - ext_ordered.append(("custom", "Custom endpoint (enter URL manually)")) - _has_saved_custom_list = isinstance(config.get("custom_providers"), list) and bool(config.get("custom_providers")) - if _has_saved_custom_list: - ext_ordered.append(("remove-custom", "Remove a saved custom provider")) - ext_ordered.append(("cancel", "Cancel")) - - ext_idx = _prompt_provider_choice( - [label for _, label in ext_ordered], default=0, - ) - if ext_idx is None or ext_ordered[ext_idx][0] == "cancel": - print("No change.") - return - selected_provider = ext_ordered[ext_idx][0] + if selected_provider == "aux-config": + _aux_config_menu() + return # Step 2: Provider-specific setup + model selection if selected_provider == "openrouter": _model_flow_openrouter(config, current_model) + elif selected_provider == "ai-gateway": + _model_flow_ai_gateway(config, current_model) elif selected_provider == "nous": _model_flow_nous(config, current_model, args=args) elif selected_provider == "openai-codex": _model_flow_openai_codex(config, current_model) elif selected_provider == "qwen-oauth": _model_flow_qwen_oauth(config, current_model) + elif selected_provider == "google-gemini-cli": + _model_flow_google_gemini_cli(config, current_model) elif selected_provider == "copilot-acp": _model_flow_copilot_acp(config, current_model) elif selected_provider == "copilot": _model_flow_copilot(config, current_model) elif selected_provider == "custom": _model_flow_custom(config) - elif selected_provider.startswith("custom:") or selected_provider in _custom_provider_map: + elif ( + selected_provider.startswith("custom:") + or selected_provider in _custom_provider_map + ): provider_info = _named_custom_provider_map(load_config()).get(selected_provider) if provider_info is None: print( @@ -1207,15 +1582,39 @@ def _named_custom_provider_map(cfg) -> dict[str, dict[str, str]]: _model_flow_anthropic(config, current_model) elif selected_provider == "kimi-coding": _model_flow_kimi(config, current_model) - elif selected_provider in ("gemini", "zai", "kimi-coding-cn", "minimax", "minimax-cn", "kilocode", "opencode-zen", "opencode-go", "ai-gateway", "alibaba", "huggingface", "xiaomi"): + elif selected_provider == "stepfun": + _model_flow_stepfun(config, current_model) + elif selected_provider == "bedrock": + _model_flow_bedrock(config, current_model) + elif selected_provider in ( + "gemini", + "deepseek", + "xai", + "zai", + "kimi-coding-cn", + "minimax", + "minimax-cn", + "kilocode", + "opencode-zen", + "opencode-go", + "alibaba", + "huggingface", + "xiaomi", + "arcee", + "nvidia", + "ollama-cloud", + ): _model_flow_api_key_provider(config, selected_provider, current_model) # ── Post-switch cleanup: clear stale OPENAI_BASE_URL ────────────── # When the user switches to a named provider (anything except "custom"), # a leftover OPENAI_BASE_URL in ~/.hermes/.env can poison auxiliary # clients that use provider:auto. Clear it proactively. (#5161) - if selected_provider not in ("custom", "cancel", "remove-custom") \ - and not selected_provider.startswith("custom:"): + if selected_provider not in ( + "custom", + "cancel", + "remove-custom", + ) and not selected_provider.startswith("custom:"): _clear_stale_openai_base_url() @@ -1242,86 +1641,420 @@ def _clear_stale_openai_base_url(): stale_url = get_env_value("OPENAI_BASE_URL") if stale_url: save_env_value("OPENAI_BASE_URL", "") - print(f"Cleared stale OPENAI_BASE_URL from .env (was: {stale_url[:40]}...)" - if len(stale_url) > 40 - else f"Cleared stale OPENAI_BASE_URL from .env (was: {stale_url})") + print( + f"Cleared stale OPENAI_BASE_URL from .env (was: {stale_url[:40]}...)" + if len(stale_url) > 40 + else f"Cleared stale OPENAI_BASE_URL from .env (was: {stale_url})" + ) -def _prompt_provider_choice(choices, *, default=0): - """Show provider selection menu with curses arrow-key navigation. +# ───────────────────────────────────────────────────────────────────────────── +# Auxiliary model configuration +# +# Hermes uses lightweight "auxiliary" models for side tasks (vision analysis, +# context compression, web extraction, session search, etc.). Each task has +# its own provider+model pair in config.yaml under `auxiliary.`. +# +# The UI lives behind "Configure auxiliary models..." at the bottom of the +# `hermes model` provider picker. It does NOT re-run credential setup — it +# only routes already-authenticated providers to specific aux tasks. Users +# configure new providers through the normal `hermes model` flow first. +# ───────────────────────────────────────────────────────────────────────────── + +# (task_key, display_name, short_description) +_AUX_TASKS: list[tuple[str, str, str]] = [ + ("vision", "Vision", "image/screenshot analysis"), + ("compression", "Compression", "context summarization"), + ("web_extract", "Web extract", "web page summarization"), + ("session_search", "Session search", "past-conversation recall"), + ("approval", "Approval", "smart command approval"), + ("mcp", "MCP", "MCP tool reasoning"), + ("flush_memories", "Flush memories", "memory consolidation"), + ("title_generation", "Title generation", "session titles"), + ("skills_hub", "Skills hub", "skills search/install"), +] - Falls back to a numbered list when curses is unavailable (e.g. piped - stdin, non-TTY environments). Returns the selected index, or None - if the user cancels. + +def _format_aux_current(task_cfg: dict) -> str: + """Render the current aux config for display in the task menu.""" + if not isinstance(task_cfg, dict): + return "auto" + base_url = str(task_cfg.get("base_url") or "").strip() + provider = str(task_cfg.get("provider") or "auto").strip() or "auto" + model = str(task_cfg.get("model") or "").strip() + if base_url: + short = base_url.replace("https://", "").replace("http://", "").rstrip("/") + return f"custom ({short})" + (f" · {model}" if model else "") + if provider == "auto": + return "auto" + (f" · {model}" if model else "") + if model: + return f"{provider} · {model}" + return provider + + +def _save_aux_choice( + task: str, + *, + provider: str, + model: str = "", + base_url: str = "", + api_key: str = "", +) -> None: + """Persist an auxiliary task's provider/model to config.yaml. + + Only writes the four routing fields — timeout, download_timeout, and any + other task-specific settings are preserved untouched. The main model + config (``model.default``/``model.provider``) is never modified. """ - try: - from hermes_cli.setup import _curses_prompt_choice - idx = _curses_prompt_choice("Select provider:", choices, default) - if idx >= 0: - print() - return idx - except Exception: - pass + from hermes_cli.config import load_config, save_config - # Fallback: numbered list - print("Select provider:") - for i, c in enumerate(choices, 1): - marker = "→" if i - 1 == default else " " - print(f" {marker} {i}. {c}") - print() - while True: - try: - val = input(f"Choice [1-{len(choices)}] ({default + 1}): ").strip() - if not val: - return default - idx = int(val) - 1 - if 0 <= idx < len(choices): - return idx - print(f"Please enter 1-{len(choices)}") - except ValueError: - print("Please enter a number") - except (KeyboardInterrupt, EOFError): - print() - return None + cfg = load_config() + aux = cfg.setdefault("auxiliary", {}) + if not isinstance(aux, dict): + aux = {} + cfg["auxiliary"] = aux + entry = aux.setdefault(task, {}) + if not isinstance(entry, dict): + entry = {} + aux[task] = entry + entry["provider"] = provider + entry["model"] = model or "" + entry["base_url"] = base_url or "" + entry["api_key"] = api_key or "" + save_config(cfg) -def _model_flow_openrouter(config, current_model=""): - """OpenRouter provider: ensure API key, then pick model.""" - from hermes_cli.auth import _prompt_model_selection, _save_model_choice, deactivate_provider - from hermes_cli.config import get_env_value, save_env_value +def _reset_aux_to_auto() -> int: + """Reset every known aux task back to auto/empty. Returns number reset.""" + from hermes_cli.config import load_config, save_config + + cfg = load_config() + aux = cfg.setdefault("auxiliary", {}) + if not isinstance(aux, dict): + aux = {} + cfg["auxiliary"] = aux + count = 0 + for task, _name, _desc in _AUX_TASKS: + entry = aux.setdefault(task, {}) + if not isinstance(entry, dict): + entry = {} + aux[task] = entry + changed = False + if entry.get("provider") not in (None, "", "auto"): + entry["provider"] = "auto" + changed = True + for field in ("model", "base_url", "api_key"): + if entry.get(field): + entry[field] = "" + changed = True + # Preserve timeout/download_timeout — those are user-tuned, not routing + if changed: + count += 1 + save_config(cfg) + return count + + +def _aux_config_menu() -> None: + """Top-level auxiliary-model picker — choose a task to configure. + + Loops until the user picks "Back" so multiple tasks can be configured + without returning to the main provider menu. + """ + from hermes_cli.config import load_config + + while True: + cfg = load_config() + aux = cfg.get("auxiliary", {}) if isinstance(cfg.get("auxiliary"), dict) else {} - api_key = get_env_value("OPENROUTER_API_KEY") - if not api_key: - print("No OpenRouter API key configured.") - print("Get one at: https://openrouter.ai/keys") print() - try: - import getpass - key = getpass.getpass("OpenRouter API key (or Enter to cancel): ").strip() - except (KeyboardInterrupt, EOFError): - print() + print(" Auxiliary models — side-task routing") + print() + print(" Side tasks (vision, compression, web extraction, etc.) default") + print(" to your main chat model. \"auto\" means \"use my main model\" —") + print(" Hermes only falls back to a lightweight backend (OpenRouter,") + print(" Nous Portal) if the main model is unavailable. Override a") + print(" task below if you want it pinned to a specific provider/model.") + print() + + # Build the task menu with current settings inline + name_col = max(len(name) for _, name, _ in _AUX_TASKS) + 2 + desc_col = max(len(desc) for _, _, desc in _AUX_TASKS) + 4 + entries: list[tuple[str, str]] = [] + for task_key, name, desc in _AUX_TASKS: + task_cfg = aux.get(task_key, {}) if isinstance(aux.get(task_key), dict) else {} + current = _format_aux_current(task_cfg) + label = f"{name.ljust(name_col)}{('(' + desc + ')').ljust(desc_col)}{current}" + entries.append((task_key, label)) + entries.append(("__reset__", "Reset all to auto")) + entries.append(("__back__", "Back")) + + idx = _prompt_provider_choice( + [label for _, label in entries], default=0, + ) + if idx is None: return - if not key: - print("Cancelled.") + key = entries[idx][0] + if key == "__back__": return - save_env_value("OPENROUTER_API_KEY", key) - print("API key saved.") - print() + if key == "__reset__": + n = _reset_aux_to_auto() + if n: + print(f"Reset {n} auxiliary task(s) to auto.") + else: + print("All auxiliary tasks were already set to auto.") + print() + continue + # Otherwise configure the specific task + _aux_select_for_task(key) - from hermes_cli.models import model_ids, get_pricing_for_provider - openrouter_models = model_ids(force_refresh=True) - # Fetch live pricing (non-blocking — returns empty dict on failure) - pricing = get_pricing_for_provider("openrouter", force_refresh=True) +def _aux_select_for_task(task: str) -> None: + """Pick a provider + model for a single auxiliary task and persist it. - selected = _prompt_model_selection(openrouter_models, current_model=current_model, pricing=pricing) - if selected: - _save_model_choice(selected) + Uses ``list_authenticated_providers()`` to only show providers the user + has already configured. This avoids re-running OAuth/credential flows + inside the aux picker — users set up new providers through the normal + ``hermes model`` flow, then route aux tasks to them here. + """ + from hermes_cli.config import load_config + from hermes_cli.model_switch import list_authenticated_providers - # Update config provider and deactivate any OAuth provider - from hermes_cli.config import load_config, save_config - cfg = load_config() - model = cfg.get("model") + cfg = load_config() + aux = cfg.get("auxiliary", {}) if isinstance(cfg.get("auxiliary"), dict) else {} + task_cfg = aux.get(task, {}) if isinstance(aux.get(task), dict) else {} + current_provider = str(task_cfg.get("provider") or "auto").strip() or "auto" + current_model = str(task_cfg.get("model") or "").strip() + current_base_url = str(task_cfg.get("base_url") or "").strip() + + display_name = next((name for key, name, _ in _AUX_TASKS if key == task), task) + + # Gather authenticated providers (has credentials + curated model list) + try: + providers = list_authenticated_providers(current_provider=current_provider) + except Exception as exc: + print(f"Could not detect authenticated providers: {exc}") + providers = [] + + entries: list[tuple[str, str, list[str]]] = [] # (slug, label, models) + # "auto" always first + auto_marker = " ← current" if current_provider == "auto" and not current_base_url else "" + entries.append(("__auto__", f"auto (recommended){auto_marker}", [])) + + for p in providers: + slug = p.get("slug", "") + name = p.get("name") or slug + total = p.get("total_models", 0) + models = p.get("models") or [] + model_hint = f" — {total} models" if total else "" + marker = " ← current" if slug == current_provider and not current_base_url else "" + entries.append((slug, f"{name}{model_hint}{marker}", list(models))) + + # Custom endpoint (raw base_url) + custom_marker = " ← current" if current_base_url else "" + entries.append(("__custom__", f"Custom endpoint (direct URL){custom_marker}", [])) + entries.append(("__back__", "Back", [])) + + print() + print(f" Configure {display_name} — current: {_format_aux_current(task_cfg)}") + print() + + idx = _prompt_provider_choice([label for _, label, _ in entries], default=0) + if idx is None: + return + slug, _label, models = entries[idx] + + if slug == "__back__": + return + + if slug == "__auto__": + _save_aux_choice(task, provider="auto", model="", base_url="", api_key="") + print(f"{display_name}: reset to auto.") + return + + if slug == "__custom__": + _aux_flow_custom_endpoint(task, task_cfg) + return + + # Regular provider — pick a model from its curated list + _aux_flow_provider_model(task, slug, models, current_model) + + +def _aux_flow_provider_model( + task: str, + provider_slug: str, + curated_models: list, + current_model: str = "", +) -> None: + """Prompt for a model under an already-authenticated provider, save to aux.""" + from hermes_cli.auth import _prompt_model_selection + from hermes_cli.models import get_pricing_for_provider + + display_name = next((name for key, name, _ in _AUX_TASKS if key == task), task) + + # Fetch live pricing for this provider (non-blocking) + pricing: dict = {} + try: + pricing = get_pricing_for_provider(provider_slug) or {} + except Exception: + pricing = {} + + model_list = list(curated_models) + + # Let the user pick a model. _prompt_model_selection supports "Enter custom + # model name" and cancel. When there's no curated list (rare), fall back + # to a raw input prompt. + if not model_list: + print(f"No curated model list for {provider_slug}.") + print("Enter a model slug manually (blank = use provider default):") + try: + val = input("Model: ").strip() + except (KeyboardInterrupt, EOFError): + print() + return + selected = val or "" + else: + selected = _prompt_model_selection( + model_list, current_model=current_model, pricing=pricing, + ) + if selected is None: + print("No change.") + return + + _save_aux_choice(task, provider=provider_slug, model=selected or "", + base_url="", api_key="") + if selected: + print(f"{display_name}: {provider_slug} · {selected}") + else: + print(f"{display_name}: {provider_slug} (provider default model)") + + +def _aux_flow_custom_endpoint(task: str, task_cfg: dict) -> None: + """Prompt for a direct OpenAI-compatible base_url + optional api_key/model.""" + import getpass + + display_name = next((name for key, name, _ in _AUX_TASKS if key == task), task) + current_base_url = str(task_cfg.get("base_url") or "").strip() + current_model = str(task_cfg.get("model") or "").strip() + + print() + print(f" Custom endpoint for {display_name}") + print(" Provide an OpenAI-compatible base URL (e.g. http://localhost:11434/v1)") + print() + try: + url_prompt = f"Base URL [{current_base_url}]: " if current_base_url else "Base URL: " + url = input(url_prompt).strip() + except (KeyboardInterrupt, EOFError): + print() + return + url = url or current_base_url + if not url: + print("No URL provided. No change.") + return + try: + model_prompt = f"Model slug (optional) [{current_model}]: " if current_model else "Model slug (optional): " + model = input(model_prompt).strip() + except (KeyboardInterrupt, EOFError): + print() + return + model = model or current_model + try: + api_key = getpass.getpass("API key (optional, blank = use OPENAI_API_KEY): ").strip() + except (KeyboardInterrupt, EOFError): + print() + return + + _save_aux_choice( + task, provider="custom", model=model, base_url=url, api_key=api_key, + ) + short_url = url.replace("https://", "").replace("http://", "").rstrip("/") + print(f"{display_name}: custom ({short_url})" + (f" · {model}" if model else "")) + + +def _prompt_provider_choice(choices, *, default=0): + """Show provider selection menu with curses arrow-key navigation. + + Falls back to a numbered list when curses is unavailable (e.g. piped + stdin, non-TTY environments). Returns the selected index, or None + if the user cancels. + """ + try: + from hermes_cli.setup import _curses_prompt_choice + + idx = _curses_prompt_choice("Select provider:", choices, default) + if idx >= 0: + print() + return idx + except Exception: + pass + + # Fallback: numbered list + print("Select provider:") + for i, c in enumerate(choices, 1): + marker = "→" if i - 1 == default else " " + print(f" {marker} {i}. {c}") + print() + while True: + try: + val = input(f"Choice [1-{len(choices)}] ({default + 1}): ").strip() + if not val: + return default + idx = int(val) - 1 + if 0 <= idx < len(choices): + return idx + print(f"Please enter 1-{len(choices)}") + except ValueError: + print("Please enter a number") + except (KeyboardInterrupt, EOFError): + print() + return None + + +def _model_flow_openrouter(config, current_model=""): + """OpenRouter provider: ensure API key, then pick model.""" + from hermes_cli.auth import ( + _prompt_model_selection, + _save_model_choice, + deactivate_provider, + ) + from hermes_cli.config import get_env_value, save_env_value + + api_key = get_env_value("OPENROUTER_API_KEY") + if not api_key: + print("No OpenRouter API key configured.") + print("Get one at: https://openrouter.ai/keys") + print() + try: + import getpass + + key = getpass.getpass("OpenRouter API key (or Enter to cancel): ").strip() + except (KeyboardInterrupt, EOFError): + print() + return + if not key: + print("Cancelled.") + return + save_env_value("OPENROUTER_API_KEY", key) + print("API key saved.") + print() + + from hermes_cli.models import model_ids, get_pricing_for_provider + + openrouter_models = model_ids(force_refresh=True) + + # Fetch live pricing (non-blocking — returns empty dict on failure) + pricing = get_pricing_for_provider("openrouter", force_refresh=True) + + selected = _prompt_model_selection( + openrouter_models, current_model=current_model, pricing=pricing + ) + if selected: + _save_model_choice(selected) + + # Update config provider and deactivate any OAuth provider + from hermes_cli.config import load_config, save_config + + cfg = load_config() + model = cfg.get("model") if not isinstance(model, dict): model = {"default": model} if model else {} cfg["model"] = model @@ -1335,20 +2068,83 @@ def _model_flow_openrouter(config, current_model=""): print("No change.") +def _model_flow_ai_gateway(config, current_model=""): + """Vercel AI Gateway provider: ensure API key, then pick model with pricing.""" + from hermes_cli.auth import ( + _prompt_model_selection, + _save_model_choice, + deactivate_provider, + ) + from hermes_cli.config import get_env_value, save_env_value + + api_key = get_env_value("AI_GATEWAY_API_KEY") + if not api_key: + print("No Vercel AI Gateway API key configured.") + print("Create API key here: https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai-gateway&title=AI+Gateway") + print("Add a payment method to get $5 in free credits.") + print() + try: + import getpass + + key = getpass.getpass("AI Gateway API key (or Enter to cancel): ").strip() + except (KeyboardInterrupt, EOFError): + print() + return + if not key: + print("Cancelled.") + return + save_env_value("AI_GATEWAY_API_KEY", key) + print("API key saved.") + print() + + from hermes_cli.models import ai_gateway_model_ids, get_pricing_for_provider + + models_list = ai_gateway_model_ids(force_refresh=True) + pricing = get_pricing_for_provider("ai-gateway", force_refresh=True) + + selected = _prompt_model_selection( + models_list, current_model=current_model, pricing=pricing + ) + if selected: + _save_model_choice(selected) + + from hermes_cli.config import load_config, save_config + + cfg = load_config() + model = cfg.get("model") + if not isinstance(model, dict): + model = {"default": model} if model else {} + cfg["model"] = model + model["provider"] = "ai-gateway" + model["base_url"] = AI_GATEWAY_BASE_URL + model["api_mode"] = "chat_completions" + save_config(cfg) + deactivate_provider() + print(f"Default model set to: {selected} (via Vercel AI Gateway)") + else: + print("No change.") + + def _model_flow_nous(config, current_model="", args=None): """Nous Portal provider: ensure logged in, then pick model.""" from hermes_cli.auth import ( - get_provider_auth_state, _prompt_model_selection, _save_model_choice, - _update_config_for_provider, resolve_nous_runtime_credentials, - AuthError, format_auth_error, - _login_nous, PROVIDER_REGISTRY, + get_provider_auth_state, + _prompt_model_selection, + _save_model_choice, + _update_config_for_provider, + resolve_nous_runtime_credentials, + AuthError, + format_auth_error, + _login_nous, + PROVIDER_REGISTRY, ) - from hermes_cli.config import get_env_value, save_config, save_env_value - from hermes_cli.nous_subscription import ( - apply_nous_provider_defaults, - get_nous_subscription_explainer_lines, + from hermes_cli.config import ( + get_env_value, + load_config, + save_config, + save_env_value, ) - import argparse + from hermes_cli.nous_subscription import prompt_enable_tool_gateway state = get_provider_auth_state("nous") if not state or not state.get("access_token"): @@ -1366,9 +2162,12 @@ def _model_flow_nous(config, current_model="", args=None): insecure=bool(getattr(args, "insecure", False)), ) _login_nous(mock_args, PROVIDER_REGISTRY["nous"]) - print() - for line in get_nous_subscription_explainer_lines(): - print(line) + # Offer Tool Gateway enablement for paid subscribers + try: + _refreshed = load_config() or {} + prompt_enable_tool_gateway(_refreshed) + except Exception: + pass except SystemExit: print("Login cancelled or failed.") return @@ -1382,9 +2181,12 @@ def _model_flow_nous(config, current_model="", args=None): # The live /models endpoint returns hundreds of models; the curated list # shows only agentic models users recognize from OpenRouter. from hermes_cli.models import ( - _PROVIDER_MODELS, get_pricing_for_provider, filter_nous_free_models, - check_nous_free_tier, partition_nous_models_by_tier, + _PROVIDER_MODELS, + get_pricing_for_provider, + check_nous_free_tier, + partition_nous_models_by_tier, ) + model_ids = _PROVIDER_MODELS.get("nous", []) if not model_ids: print("No curated models available for Nous Portal.") @@ -1401,9 +2203,14 @@ def _model_flow_nous(config, current_model="", args=None): print("Re-authenticating with Nous Portal...\n") try: mock_args = argparse.Namespace( - portal_url=None, inference_url=None, client_id=None, - scope=None, no_browser=False, timeout=15.0, - ca_bundle=None, insecure=False, + portal_url=None, + inference_url=None, + client_id=None, + scope=None, + no_browser=False, + timeout=15.0, + ca_bundle=None, + insecure=False, ) _login_nous(mock_args, PROVIDER_REGISTRY["nous"]) except Exception as login_exc: @@ -1418,13 +2225,13 @@ def _model_flow_nous(config, current_model="", args=None): # Check if user is on free tier free_tier = check_nous_free_tier() - # For both tiers: apply the allowlist filter first (removes non-allowlisted - # free models and allowlist models that aren't actually free). - # Then for free users: partition remaining models into selectable/unavailable. - model_ids = filter_nous_free_models(model_ids, pricing) + # For free users: partition models into selectable/unavailable based on + # whether they are free per the Portal-reported pricing. unavailable_models: list[str] = [] if free_tier: - model_ids, unavailable_models = partition_nous_models_by_tier(model_ids, pricing, free_tier=True) + model_ids, unavailable_models = partition_nous_models_by_tier( + model_ids, pricing, free_tier=True + ) if not model_ids and not unavailable_models: print("No models available for Nous Portal after filtering.") @@ -1443,15 +2250,21 @@ def _model_flow_nous(config, current_model="", args=None): print("No free models currently available.") if unavailable_models: from hermes_cli.auth import DEFAULT_NOUS_PORTAL_URL + _url = (_nous_portal_url or DEFAULT_NOUS_PORTAL_URL).rstrip("/") print(f"Upgrade at {_url} to access paid models.") return - print(f"Showing {len(model_ids)} curated models — use \"Enter custom model name\" for others.") + print( + f'Showing {len(model_ids)} curated models — use "Enter custom model name" for others.' + ) selected = _prompt_model_selection( - model_ids, current_model=current_model, pricing=pricing, - unavailable_models=unavailable_models, portal_url=_nous_portal_url, + model_ids, + current_model=current_model, + pricing=pricing, + unavailable_models=unavailable_models, + portal_url=_nous_portal_url, ) if selected: _save_model_choice(selected) @@ -1476,18 +2289,10 @@ def _model_flow_nous(config, current_model="", args=None): if get_env_value("OPENAI_BASE_URL"): save_env_value("OPENAI_BASE_URL", "") save_env_value("OPENAI_API_KEY", "") - changed_defaults = apply_nous_provider_defaults(config) save_config(config) print(f"Default model set to: {selected} (via Nous Portal)") - if "tts" in changed_defaults: - print("TTS provider set to: OpenAI TTS via your Nous subscription") - else: - current_tts = str(config.get("tts", {}).get("provider") or "edge") - if current_tts.lower() not in {"", "edge"}: - print(f"Keeping your existing TTS provider: {current_tts}") - print() - for line in get_nous_subscription_explainer_lines(): - print(line) + # Offer Tool Gateway enablement for paid subscribers + prompt_enable_tool_gateway(config) else: print("No change.") @@ -1495,12 +2300,15 @@ def _model_flow_nous(config, current_model="", args=None): def _model_flow_openai_codex(config, current_model=""): """OpenAI Codex provider: ensure logged in, then pick model.""" from hermes_cli.auth import ( - get_codex_auth_status, _prompt_model_selection, _save_model_choice, - _update_config_for_provider, _login_openai_codex, - PROVIDER_REGISTRY, DEFAULT_CODEX_BASE_URL, + get_codex_auth_status, + _prompt_model_selection, + _save_model_choice, + _update_config_for_provider, + _login_openai_codex, + PROVIDER_REGISTRY, + DEFAULT_CODEX_BASE_URL, ) from hermes_cli.codex_models import get_codex_model_ids - import argparse status = get_codex_auth_status() if not status.get("logged_in"): @@ -1528,6 +2336,7 @@ def _model_flow_openai_codex(config, current_model=""): if not _codex_token: try: from hermes_cli.auth import resolve_codex_runtime_credentials + _codex_creds = resolve_codex_runtime_credentials() _codex_token = _codex_creds.get("api_key") except Exception: @@ -1544,7 +2353,6 @@ def _model_flow_openai_codex(config, current_model=""): print("No change.") - _DEFAULT_QWEN_PORTAL_MODELS = [ "qwen3-coder-plus", "qwen3-coder", @@ -1594,6 +2402,80 @@ def _model_flow_qwen_oauth(_config, current_model=""): print("No change.") +def _model_flow_google_gemini_cli(_config, current_model=""): + """Google Gemini OAuth (PKCE) via Cloud Code Assist — supports free AND paid tiers. + + Flow: + 1. Show upfront warning about Google's ToS stance (per opencode-gemini-auth). + 2. If creds missing, run PKCE browser OAuth via agent.google_oauth. + 3. Resolve project context (env -> config -> auto-discover -> free tier). + 4. Prompt user to pick a model. + 5. Save to ~/.hermes/config.yaml. + """ + from hermes_cli.auth import ( + DEFAULT_GEMINI_CLOUDCODE_BASE_URL, + get_gemini_oauth_auth_status, + resolve_gemini_oauth_runtime_credentials, + _prompt_model_selection, + _save_model_choice, + _update_config_for_provider, + ) + from hermes_cli.models import _PROVIDER_MODELS + + print() + print("⚠ Google considers using the Gemini CLI OAuth client with third-party") + print(" software a policy violation. Some users have reported account") + print(" restrictions. You can use your own API key via 'gemini' provider") + print(" for the lowest-risk experience.") + print() + try: + proceed = input("Continue with OAuth login? [y/N]: ").strip().lower() + except (EOFError, KeyboardInterrupt): + print("Cancelled.") + return + if proceed not in {"y", "yes"}: + print("Cancelled.") + return + + status = get_gemini_oauth_auth_status() + if not status.get("logged_in"): + try: + from agent.google_oauth import resolve_project_id_from_env, start_oauth_flow + + env_project = resolve_project_id_from_env() + start_oauth_flow(force_relogin=True, project_id=env_project) + except Exception as exc: + print(f"OAuth login failed: {exc}") + return + + # Verify creds resolve + trigger project discovery + try: + creds = resolve_gemini_oauth_runtime_credentials(force_refresh=False) + project_id = creds.get("project_id", "") + if project_id: + print(f" Using GCP project: {project_id}") + else: + print( + " No GCP project configured — free tier will be auto-provisioned on first request." + ) + except Exception as exc: + print(f"Failed to resolve Gemini credentials: {exc}") + return + + models = list(_PROVIDER_MODELS.get("google-gemini-cli") or []) + default = current_model or (models[0] if models else "gemini-3-flash-preview") + selected = _prompt_model_selection(models, current_model=default) + if selected: + _save_model_choice(selected) + _update_config_for_provider( + "google-gemini-cli", DEFAULT_GEMINI_CLOUDCODE_BASE_URL + ) + print( + f"Default model set to: {selected} (via Google Gemini OAuth / Code Assist)" + ) + else: + print("No change.") + def _model_flow_custom(config): """Custom endpoint: collect URL, API key, and model name. @@ -1615,9 +2497,14 @@ def _model_flow_custom(config): print() try: - base_url = input(f"API base URL [{current_url or 'e.g. https://api.example.com/v1'}]: ").strip() + base_url = input( + f"API base URL [{current_url or 'e.g. https://api.example.com/v1'}]: " + ).strip() import getpass - api_key = getpass.getpass(f"API key [{current_key[:8] + '...' if current_key else 'optional'}]: ").strip() + + api_key = getpass.getpass( + f"API key [{current_key[:8] + '...' if current_key else 'optional'}]: " + ).strip() except (KeyboardInterrupt, EOFError): print("\nCancelled.") return @@ -1634,6 +2521,30 @@ def _model_flow_custom(config): effective_key = api_key or current_key + # Hint: most local model servers (Ollama, vLLM, llama.cpp) require /v1 + # in the base URL for OpenAI-compatible chat completions. Prompt the + # user if the URL looks like a local server without /v1. + _url_lower = effective_url.rstrip("/").lower() + _looks_local = any( + h in _url_lower + for h in ("localhost", "127.0.0.1", "0.0.0.0", ":11434", ":8080", ":5000") + ) + if _looks_local and not _url_lower.endswith("/v1"): + print() + print(f" Hint: Did you mean to add /v1 at the end?") + print(f" Most local model servers (Ollama, vLLM, llama.cpp) require it.") + print(f" e.g. {effective_url.rstrip('/')}/v1") + try: + _add_v1 = input(" Add /v1? [Y/n]: ").strip().lower() + except (KeyboardInterrupt, EOFError): + _add_v1 = "n" + if _add_v1 in ("", "y", "yes"): + effective_url = effective_url.rstrip("/") + "/v1" + if base_url: + base_url = effective_url + print(f" Updated URL: {effective_url}") + print() + from hermes_cli.models import probe_api_models probe = probe_api_models(effective_key, effective_url) @@ -1658,7 +2569,9 @@ def _model_flow_custom(config): if probe.get("suggested_base_url"): suggested = probe["suggested_base_url"] if suggested.endswith("/v1"): - print(f" If this server expects /v1 in the path, try base URL: {suggested}") + print( + f" If this server expects /v1 in the path, try base URL: {suggested}" + ) else: print(f" If /v1 should not be in the base URL, try: {suggested}") @@ -1677,7 +2590,9 @@ def _model_flow_custom(config): print(" Available models:") for i, m in enumerate(detected_models, 1): print(f" {i}. {m}") - pick = input(f" Select model [1-{len(detected_models)}] or type name: ").strip() + pick = input( + f" Select model [1-{len(detected_models)}] or type name: " + ).strip() if pick.isdigit() and 1 <= int(pick) <= len(detected_models): model_name = detected_models[int(pick) - 1] elif pick: @@ -1685,7 +2600,13 @@ def _model_flow_custom(config): else: model_name = input("Model name (e.g. gpt-4, llama-3-70b): ").strip() - context_length_str = input("Context length in tokens [leave blank for auto-detect]: ").strip() + context_length_str = input( + "Context length in tokens [leave blank for auto-detect]: " + ).strip() + + # Prompt for a display name — shown in the provider menu on future runs + default_name = _auto_provider_name(effective_url) + display_name = input(f"Display name [{default_name}]: ").strip() or default_name except (KeyboardInterrupt, EOFError): print("\nCancelled.") return @@ -1693,7 +2614,11 @@ def _model_flow_custom(config): context_length = None if context_length_str: try: - context_length = int(context_length_str.replace(",", "").replace("k", "000").replace("K", "000")) + context_length = int( + context_length_str.replace(",", "") + .replace("k", "000") + .replace("K", "000") + ) if context_length <= 0: context_length = None except ValueError: @@ -1741,15 +2666,44 @@ def _model_flow_custom(config): print("Endpoint saved. Use `/model` in chat or `hermes model` to set a model.") # Auto-save to custom_providers so it appears in the menu next time - _save_custom_provider(effective_url, effective_key, model_name or "", context_length=context_length) + _save_custom_provider( + effective_url, + effective_key, + model_name or "", + context_length=context_length, + name=display_name, + ) -def _save_custom_provider(base_url, api_key="", model="", context_length=None): +def _auto_provider_name(base_url: str) -> str: + """Generate a display name from a custom endpoint URL. + + Returns a human-friendly label like "Local (localhost:11434)" or + "RunPod (xyz.runpod.io)". Used as the default when prompting the + user for a display name during custom endpoint setup. + """ + import re + + clean = base_url.replace("https://", "").replace("http://", "").rstrip("/") + clean = re.sub(r"/v1/?$", "", clean) + name = clean.split("/")[0] + if "localhost" in name or "127.0.0.1" in name: + name = f"Local ({name})" + elif "runpod" in name.lower(): + name = f"RunPod ({name})" + else: + name = name.capitalize() + return name + + +def _save_custom_provider( + base_url, api_key="", model="", context_length=None, name=None +): """Save a custom endpoint to custom_providers in config.yaml. Deduplicates by base_url — if the URL already exists, updates the model name and context_length but doesn't add a duplicate entry. - Auto-generates a display name from the URL hostname. + Uses *name* when provided, otherwise auto-generates from the URL. """ from hermes_cli.config import load_config, save_config @@ -1760,7 +2714,9 @@ def _save_custom_provider(base_url, api_key="", model="", context_length=None): # Check if this URL is already saved — update model/context_length if so for entry in providers: - if isinstance(entry, dict) and entry.get("base_url", "").rstrip("/") == base_url.rstrip("/"): + if isinstance(entry, dict) and entry.get("base_url", "").rstrip( + "/" + ) == base_url.rstrip("/"): changed = False if model and entry.get("model") != model: entry["model"] = model @@ -1777,20 +2733,9 @@ def _save_custom_provider(base_url, api_key="", model="", context_length=None): save_config(cfg) return # already saved, updated if needed - # Auto-generate a name from the URL - import re - clean = base_url.replace("https://", "").replace("http://", "").rstrip("/") - # Remove /v1 suffix for cleaner names - clean = re.sub(r"/v1/?$", "", clean) - # Use hostname:port as the name - name = clean.split("/")[0] - # Capitalize for readability - if "localhost" in name or "127.0.0.1" in name: - name = f"Local ({name})" - elif "runpod" in name.lower(): - name = f"RunPod ({name})" - else: - name = name.capitalize() + # Use provided name or auto-generate from URL + if not name: + name = _auto_provider_name(base_url) entry = {"name": name, "base_url": base_url} if api_key: @@ -1803,7 +2748,7 @@ def _save_custom_provider(base_url, api_key="", model="", context_length=None): providers.append(entry) cfg["custom_providers"] = providers save_config(cfg) - print(f" 💾 Saved to custom providers as \"{name}\" (edit in config.yaml)") + print(f' 💾 Saved to custom providers as "{name}" (edit in config.yaml)') def _remove_custom_provider(config): @@ -1831,15 +2776,20 @@ def _remove_custom_provider(config): try: from simple_term_menu import TerminalMenu + menu = TerminalMenu( - [f" {c}" for c in choices], cursor_index=0, - menu_cursor="-> ", menu_cursor_style=("fg_red", "bold"), + [f" {c}" for c in choices], + cursor_index=0, + menu_cursor="-> ", + menu_cursor_style=("fg_red", "bold"), menu_highlight_style=("fg_red",), - cycle_cursor=True, clear_screen=False, + cycle_cursor=True, + clear_screen=False, title="Select provider to remove:", ) idx = menu.show() from hermes_cli.curses_ui import flush_stdin + flush_stdin() print() except (ImportError, NotImplementedError, OSError, subprocess.SubprocessError): @@ -1859,8 +2809,10 @@ def _remove_custom_provider(config): removed = providers.pop(idx) cfg["custom_providers"] = providers save_config(cfg) - removed_name = removed.get("name", "unnamed") if isinstance(removed, dict) else str(removed) - print(f"✅ Removed \"{removed_name}\" from custom providers.") + removed_name = ( + removed.get("name", "unnamed") if isinstance(removed, dict) else str(removed) + ) + print(f'✅ Removed "{removed_name}" from custom providers.') def _model_flow_named_custom(config, provider_info): @@ -1898,19 +2850,23 @@ def _model_flow_named_custom(config, provider_info): print(f"Found {len(models)} model(s):\n") try: from simple_term_menu import TerminalMenu + menu_items = [ - f" {m} (current)" if m == saved_model else f" {m}" - for m in models + f" {m} (current)" if m == saved_model else f" {m}" for m in models ] + [" Cancel"] menu = TerminalMenu( - menu_items, cursor_index=default_idx, - menu_cursor="-> ", menu_cursor_style=("fg_green", "bold"), + menu_items, + cursor_index=default_idx, + menu_cursor="-> ", + menu_cursor_style=("fg_green", "bold"), menu_highlight_style=("fg_green",), - cycle_cursor=True, clear_screen=False, + cycle_cursor=True, + clear_screen=False, title=f"Select model from {name}:", ) idx = menu.show() from hermes_cli.curses_ui import flush_stdin + flush_stdin() print() if idx is None or idx >= len(models): @@ -2023,7 +2979,11 @@ def _set_reasoning_effort(config, effort: str) -> None: def _prompt_reasoning_effort_selection(efforts, current_effort=""): """Prompt for a reasoning effort. Returns effort, 'none', or None to keep current.""" - deduped = list(dict.fromkeys(str(effort).strip().lower() for effort in efforts if str(effort).strip())) + deduped = list( + dict.fromkeys( + str(effort).strip().lower() for effort in efforts if str(effort).strip() + ) + ) canonical_order = ("minimal", "low", "medium", "high", "xhigh") ordered = [effort for effort in canonical_order if effort in deduped] ordered.extend(effort for effort in deduped if effort not in canonical_order) @@ -2065,6 +3025,7 @@ def _label(effort): ) idx = menu.show() from hermes_cli.curses_ui import flush_stdin + flush_stdin() if idx is None: return None @@ -2133,7 +3094,9 @@ def _model_flow_copilot(config, current_model=""): print("No GitHub token configured for GitHub Copilot.") print() print(" Supported token types:") - print(" → OAuth token (gho_*) via `copilot login` or device code flow") + print( + " → OAuth token (gho_*) via `copilot login` or device code flow" + ) print(" → Fine-grained PAT (github_pat_*) with Copilot Requests permission") print(" → GitHub App token (ghu_*) via environment variable") print(" ✗ Classic PAT (ghp_*) NOT supported by Copilot API") @@ -2152,6 +3115,7 @@ def _model_flow_copilot(config, current_model=""): if choice == "1": try: from hermes_cli.copilot_auth import copilot_device_code_login + token = copilot_device_code_login() if token: save_env_value("COPILOT_GITHUB_TOKEN", token) @@ -2166,6 +3130,7 @@ def _model_flow_copilot(config, current_model=""): elif choice == "2": try: import getpass + new_key = getpass.getpass(" Token (COPILOT_GITHUB_TOKEN): ").strip() except (KeyboardInterrupt, EOFError): print() @@ -2176,6 +3141,7 @@ def _model_flow_copilot(config, current_model=""): # Validate token type try: from hermes_cli.copilot_auth import validate_copilot_token + valid, msg = validate_copilot_token(new_key) if not valid: print(f" ✗ {msg}") @@ -2204,23 +3170,34 @@ def _model_flow_copilot(config, current_model=""): effective_base = pconfig.inference_base_url catalog = fetch_github_model_catalog(api_key) - live_models = [item.get("id", "") for item in catalog if item.get("id")] if catalog else fetch_api_models(api_key, effective_base) - normalized_current_model = normalize_copilot_model_id( - current_model, - catalog=catalog, - api_key=api_key, - ) or current_model + live_models = ( + [item.get("id", "") for item in catalog if item.get("id")] + if catalog + else fetch_api_models(api_key, effective_base) + ) + normalized_current_model = ( + normalize_copilot_model_id( + current_model, + catalog=catalog, + api_key=api_key, + ) + or current_model + ) if live_models: model_list = [model_id for model_id in live_models if model_id] print(f" Found {len(model_list)} model(s) from GitHub Copilot") else: model_list = _PROVIDER_MODELS.get(provider_id, []) if model_list: - print(" ⚠ Could not auto-detect models from GitHub Copilot — showing defaults.") + print( + " ⚠ Could not auto-detect models from GitHub Copilot — showing defaults." + ) print(' Use "Enter custom model name" if you do not see your model.') if model_list: - selected = _prompt_model_selection(model_list, current_model=normalized_current_model) + selected = _prompt_model_selection( + model_list, current_model=normalized_current_model + ) else: try: selected = input("Model name: ").strip() @@ -2228,11 +3205,14 @@ def _model_flow_copilot(config, current_model=""): selected = None if selected: - selected = normalize_copilot_model_id( - selected, - catalog=catalog, - api_key=api_key, - ) or selected + selected = ( + normalize_copilot_model_id( + selected, + catalog=catalog, + api_key=api_key, + ) + or selected + ) initial_cfg = load_config() current_effort = _current_reasoning_effort(initial_cfg) reasoning_efforts = github_model_reasoning_efforts( @@ -2299,7 +3279,9 @@ def _model_flow_copilot_acp(config, current_model=""): pconfig = PROVIDER_REGISTRY[provider_id] status = get_external_process_provider_status(provider_id) - resolved_command = status.get("resolved_command") or status.get("command") or "copilot" + resolved_command = ( + status.get("resolved_command") or status.get("command") or "copilot" + ) effective_base = status.get("base_url") or pconfig.inference_base_url print(" GitHub Copilot ACP delegates Hermes turns to `copilot --acp`.") @@ -2313,7 +3295,9 @@ def _model_flow_copilot_acp(config, current_model=""): creds = resolve_external_process_provider_credentials(provider_id) except Exception as exc: print(f" ⚠ {exc}") - print(" Set HERMES_COPILOT_ACP_COMMAND or COPILOT_CLI_PATH if Copilot CLI is installed elsewhere.") + print( + " Set HERMES_COPILOT_ACP_COMMAND or COPILOT_CLI_PATH if Copilot CLI is installed elsewhere." + ) return effective_base = creds.get("base_url") or effective_base @@ -2326,11 +3310,14 @@ def _model_flow_copilot_acp(config, current_model=""): pass catalog = fetch_github_model_catalog(catalog_api_key) - normalized_current_model = normalize_copilot_model_id( - current_model, - catalog=catalog, - api_key=catalog_api_key, - ) or current_model + normalized_current_model = ( + normalize_copilot_model_id( + current_model, + catalog=catalog, + api_key=catalog_api_key, + ) + or current_model + ) if catalog: model_list = [item.get("id", "") for item in catalog if item.get("id")] @@ -2338,7 +3325,9 @@ def _model_flow_copilot_acp(config, current_model=""): else: model_list = _PROVIDER_MODELS.get("copilot", []) if model_list: - print(" ⚠ Could not auto-detect models from GitHub Copilot — showing defaults.") + print( + " ⚠ Could not auto-detect models from GitHub Copilot — showing defaults." + ) print(' Use "Enter custom model name" if you do not see your model.') if model_list: @@ -2356,11 +3345,14 @@ def _model_flow_copilot_acp(config, current_model=""): print("No change.") return - selected = normalize_copilot_model_id( - selected, - catalog=catalog, - api_key=catalog_api_key, - ) or selected + selected = ( + normalize_copilot_model_id( + selected, + catalog=catalog, + api_key=catalog_api_key, + ) + or selected + ) _save_model_choice(selected) cfg = load_config() @@ -2386,10 +3378,18 @@ def _model_flow_kimi(config, current_model=""): No manual base URL prompt — endpoint is determined by key prefix. """ from hermes_cli.auth import ( - PROVIDER_REGISTRY, KIMI_CODE_BASE_URL, _prompt_model_selection, - _save_model_choice, deactivate_provider, + PROVIDER_REGISTRY, + KIMI_CODE_BASE_URL, + _prompt_model_selection, + _save_model_choice, + deactivate_provider, + ) + from hermes_cli.config import ( + get_env_value, + save_env_value, + load_config, + save_config, ) - from hermes_cli.config import get_env_value, save_env_value, load_config, save_config provider_id = "kimi-coding" pconfig = PROVIDER_REGISTRY[provider_id] @@ -2408,6 +3408,7 @@ def _model_flow_kimi(config, current_model=""): if key_env: try: import getpass + new_key = getpass.getpass(f"{key_env} (or Enter to cancel): ").strip() except (KeyboardInterrupt, EOFError): print() @@ -2438,54 +3439,471 @@ def _model_flow_kimi(config, current_model=""): # Step 3: Model selection — show appropriate models for the endpoint if is_coding_plan: - # Coding Plan models (kimi-for-coding first) + # Coding Plan models (kimi-k2.6 first) model_list = [ - "kimi-for-coding", + "kimi-k2.6", "kimi-k2.5", + "kimi-for-coding", "kimi-k2-thinking", "kimi-k2-thinking-turbo", ] else: - # Legacy Moonshot models (excludes Coding Plan-only models) - model_list = _PROVIDER_MODELS.get("moonshot", []) + # Legacy Moonshot models (excludes Coding Plan-only models) + model_list = _PROVIDER_MODELS.get("moonshot", []) + + if model_list: + selected = _prompt_model_selection(model_list, current_model=current_model) + else: + try: + selected = input("Enter model name: ").strip() + except (KeyboardInterrupt, EOFError): + selected = None + + if selected: + _save_model_choice(selected) + + # Update config with provider and base URL + cfg = load_config() + model = cfg.get("model") + if not isinstance(model, dict): + model = {"default": model} if model else {} + cfg["model"] = model + model["provider"] = provider_id + model["base_url"] = effective_base + model.pop("api_mode", None) # let runtime auto-detect from URL + save_config(cfg) + deactivate_provider() + + endpoint_label = "Kimi Coding" if is_coding_plan else "Moonshot" + print(f"Default model set to: {selected} (via {endpoint_label})") + else: + print("No change.") + + +def _infer_stepfun_region(base_url: str) -> str: + """Infer the current StepFun region from the configured endpoint.""" + normalized = (base_url or "").strip().lower() + if "api.stepfun.com" in normalized: + return "china" + return "international" + + +def _stepfun_base_url_for_region(region: str) -> str: + from hermes_cli.auth import ( + STEPFUN_STEP_PLAN_CN_BASE_URL, + STEPFUN_STEP_PLAN_INTL_BASE_URL, + ) + + return ( + STEPFUN_STEP_PLAN_CN_BASE_URL + if region == "china" + else STEPFUN_STEP_PLAN_INTL_BASE_URL + ) + + +def _model_flow_stepfun(config, current_model=""): + """StepFun Step Plan flow with region-specific endpoints.""" + from hermes_cli.auth import ( + PROVIDER_REGISTRY, + _prompt_model_selection, + _save_model_choice, + deactivate_provider, + ) + from hermes_cli.config import get_env_value, save_env_value, load_config, save_config + from hermes_cli.models import fetch_api_models + + provider_id = "stepfun" + pconfig = PROVIDER_REGISTRY[provider_id] + key_env = pconfig.api_key_env_vars[0] if pconfig.api_key_env_vars else "" + base_url_env = pconfig.base_url_env_var or "" + + existing_key = "" + for ev in pconfig.api_key_env_vars: + existing_key = get_env_value(ev) or os.getenv(ev, "") + if existing_key: + break + + if not existing_key: + print(f"No {pconfig.name} API key configured.") + if key_env: + try: + import getpass + new_key = getpass.getpass(f"{key_env} (or Enter to cancel): ").strip() + except (KeyboardInterrupt, EOFError): + print() + return + if not new_key: + print("Cancelled.") + return + save_env_value(key_env, new_key) + existing_key = new_key + print("API key saved.") + print() + else: + print(f" {pconfig.name} API key: {existing_key[:8]}... ✓") + print() + + current_base = "" + if base_url_env: + current_base = get_env_value(base_url_env) or os.getenv(base_url_env, "") + if not current_base: + model_cfg = config.get("model") + if isinstance(model_cfg, dict): + current_base = str(model_cfg.get("base_url") or "").strip() + current_region = _infer_stepfun_region(current_base or pconfig.inference_base_url) + + region_choices = [ + ("international", f"International ({_stepfun_base_url_for_region('international')})"), + ("china", f"China ({_stepfun_base_url_for_region('china')})"), + ] + ordered_regions = [] + for region_key, label in region_choices: + if region_key == current_region: + ordered_regions.insert(0, (region_key, f"{label} ← currently active")) + else: + ordered_regions.append((region_key, label)) + ordered_regions.append(("cancel", "Cancel")) + + region_idx = _prompt_provider_choice([label for _, label in ordered_regions]) + if region_idx is None or ordered_regions[region_idx][0] == "cancel": + print("No change.") + return + + selected_region = ordered_regions[region_idx][0] + effective_base = _stepfun_base_url_for_region(selected_region) + if base_url_env: + save_env_value(base_url_env, effective_base) + + live_models = fetch_api_models(existing_key, effective_base) + if live_models: + model_list = live_models + print(f" Found {len(model_list)} model(s) from {pconfig.name} API") + else: + model_list = _PROVIDER_MODELS.get(provider_id, []) + if model_list: + print( + f" Could not auto-detect models from {pconfig.name} API — " + "showing Step Plan fallback catalog." + ) + + if model_list: + selected = _prompt_model_selection(model_list, current_model=current_model) + else: + try: + selected = input("Model name: ").strip() + except (KeyboardInterrupt, EOFError): + selected = None + + if selected: + _save_model_choice(selected) + + cfg = load_config() + model = cfg.get("model") + if not isinstance(model, dict): + model = {"default": model} if model else {} + cfg["model"] = model + model["provider"] = provider_id + model["base_url"] = effective_base + model.pop("api_mode", None) + save_config(cfg) + deactivate_provider() + + config["model"] = dict(model) + print(f"Default model set to: {selected} (via {pconfig.name})") + else: + print("No change.") + + +def _model_flow_bedrock_api_key(config, region, current_model=""): + """Bedrock API Key mode — uses the OpenAI-compatible bedrock-mantle endpoint. + + For developers who don't have an AWS account but received a Bedrock API Key + from their AWS admin. Works like any OpenAI-compatible endpoint. + """ + from hermes_cli.auth import ( + _prompt_model_selection, + _save_model_choice, + deactivate_provider, + ) + from hermes_cli.config import ( + load_config, + save_config, + get_env_value, + save_env_value, + ) + from hermes_cli.models import _PROVIDER_MODELS + + mantle_base_url = f"https://bedrock-mantle.{region}.api.aws/v1" + + # Prompt for API key + existing_key = get_env_value("AWS_BEARER_TOKEN_BEDROCK") or "" + if existing_key: + print(f" Bedrock API Key: {existing_key[:12]}... ✓") + else: + print(f" Endpoint: {mantle_base_url}") + print() + try: + import getpass + + api_key = getpass.getpass(" Bedrock API Key: ").strip() + except (KeyboardInterrupt, EOFError): + print() + return + if not api_key: + print(" Cancelled.") + return + save_env_value("AWS_BEARER_TOKEN_BEDROCK", api_key) + existing_key = api_key + print(" ✓ API key saved.") + print() + + # Model selection — use static list (mantle doesn't need boto3 for discovery) + model_list = _PROVIDER_MODELS.get("bedrock", []) + print(f" Showing {len(model_list)} curated models") + + if model_list: + selected = _prompt_model_selection(model_list, current_model=current_model) + else: + try: + selected = input(" Model ID: ").strip() + except (KeyboardInterrupt, EOFError): + selected = None + + if selected: + _save_model_choice(selected) + + # Save as custom provider pointing to bedrock-mantle + cfg = load_config() + model = cfg.get("model") + if not isinstance(model, dict): + model = {"default": model} if model else {} + cfg["model"] = model + model["provider"] = "custom" + model["base_url"] = mantle_base_url + model.pop("api_mode", None) # chat_completions is the default + + # Also save region in bedrock config for reference + bedrock_cfg = cfg.get("bedrock", {}) + if not isinstance(bedrock_cfg, dict): + bedrock_cfg = {} + bedrock_cfg["region"] = region + cfg["bedrock"] = bedrock_cfg + + # Save the API key env var name so hermes knows where to find it + save_env_value("OPENAI_API_KEY", existing_key) + save_env_value("OPENAI_BASE_URL", mantle_base_url) + + save_config(cfg) + deactivate_provider() + + print(f" Default model set to: {selected} (via Bedrock API Key, {region})") + print(f" Endpoint: {mantle_base_url}") + else: + print(" No change.") + + +def _model_flow_bedrock(config, current_model=""): + """AWS Bedrock provider: verify credentials, pick region, discover models. + + Uses the native Converse API via boto3 — not the OpenAI-compatible endpoint. + Auth is handled by the AWS SDK default credential chain (env vars, profile, + instance role), so no API key prompt is needed. + """ + from hermes_cli.auth import ( + _prompt_model_selection, + _save_model_choice, + deactivate_provider, + ) + from hermes_cli.config import load_config, save_config + from hermes_cli.models import _PROVIDER_MODELS + + # 1. Check for AWS credentials + try: + from agent.bedrock_adapter import ( + has_aws_credentials, + resolve_aws_auth_env_var, + resolve_bedrock_region, + discover_bedrock_models, + ) + except ImportError: + print(" ✗ boto3 is not installed. Install it with:") + print(" pip install boto3") + print() + return + + if not has_aws_credentials(): + print(" ⚠ No AWS credentials detected via environment variables.") + print(" Bedrock will use boto3's default credential chain (IMDS, SSO, etc.)") + print() + + auth_var = resolve_aws_auth_env_var() + if auth_var: + print(f" AWS credentials: {auth_var} ✓") + else: + print(" AWS credentials: boto3 default chain (instance role / SSO)") + print() + + # 2. Region selection + current_region = resolve_bedrock_region() + try: + region_input = input(f" AWS Region [{current_region}]: ").strip() + except (KeyboardInterrupt, EOFError): + print() + return + region = region_input or current_region + + # 2b. Authentication mode + print(" Choose authentication method:") + print() + print(" 1. IAM credential chain (recommended)") + print(" Works with EC2 instance roles, SSO, env vars, aws configure") + print(" 2. Bedrock API Key") + print(" Enter your Bedrock API Key directly — also supports") + print(" team scenarios where an admin distributes keys") + print() + try: + auth_choice = input(" Choice [1]: ").strip() + except (KeyboardInterrupt, EOFError): + print() + return + + if auth_choice == "2": + _model_flow_bedrock_api_key(config, region, current_model) + return + + # 3. Model discovery — try live API first, fall back to static list + print(f" Discovering models in {region}...") + live_models = discover_bedrock_models(region) + + if live_models: + _EXCLUDE_PREFIXES = ( + "stability.", + "cohere.embed", + "twelvelabs.", + "us.stability.", + "us.cohere.embed", + "us.twelvelabs.", + "global.cohere.embed", + "global.twelvelabs.", + ) + _EXCLUDE_SUBSTRINGS = ("safeguard", "voxtral", "palmyra-vision") + filtered = [] + for m in live_models: + mid = m["id"] + if any(mid.startswith(p) for p in _EXCLUDE_PREFIXES): + continue + if any(s in mid.lower() for s in _EXCLUDE_SUBSTRINGS): + continue + filtered.append(m) + + # Deduplicate: prefer inference profiles (us.*, global.*) over bare + # foundation model IDs. + profile_base_ids = set() + for m in filtered: + mid = m["id"] + if mid.startswith(("us.", "global.")): + base = mid.split(".", 1)[1] if "." in mid[3:] else mid + profile_base_ids.add(base) + + deduped = [] + for m in filtered: + mid = m["id"] + if not mid.startswith(("us.", "global.")) and mid in profile_base_ids: + continue + deduped.append(m) + + _RECOMMENDED = [ + "us.anthropic.claude-sonnet-4-6", + "us.anthropic.claude-opus-4-6", + "us.anthropic.claude-haiku-4-5", + "us.amazon.nova-pro", + "us.amazon.nova-lite", + "us.amazon.nova-micro", + "deepseek.v3", + "us.meta.llama4-maverick", + "us.meta.llama4-scout", + ] + + def _sort_key(m): + mid = m["id"] + for i, rec in enumerate(_RECOMMENDED): + if mid.startswith(rec): + return (0, i, mid) + if mid.startswith("global."): + return (1, 0, mid) + return (2, 0, mid) + + deduped.sort(key=_sort_key) + model_list = [m["id"] for m in deduped] + print( + f" Found {len(model_list)} text model(s) (filtered from {len(live_models)} total)" + ) + else: + model_list = _PROVIDER_MODELS.get("bedrock", []) + if model_list: + print( + f" Using {len(model_list)} curated models (live discovery unavailable)" + ) + else: + print( + " No models found. Check IAM permissions for bedrock:ListFoundationModels." + ) + return + # 4. Model selection if model_list: selected = _prompt_model_selection(model_list, current_model=current_model) else: try: - selected = input("Enter model name: ").strip() + selected = input(" Model ID: ").strip() except (KeyboardInterrupt, EOFError): selected = None if selected: _save_model_choice(selected) - # Update config with provider and base URL cfg = load_config() model = cfg.get("model") if not isinstance(model, dict): model = {"default": model} if model else {} cfg["model"] = model - model["provider"] = provider_id - model["base_url"] = effective_base - model.pop("api_mode", None) # let runtime auto-detect from URL + model["provider"] = "bedrock" + model["base_url"] = f"https://bedrock-runtime.{region}.amazonaws.com" + model.pop("api_mode", None) # bedrock_converse is auto-detected + + bedrock_cfg = cfg.get("bedrock", {}) + if not isinstance(bedrock_cfg, dict): + bedrock_cfg = {} + bedrock_cfg["region"] = region + cfg["bedrock"] = bedrock_cfg + save_config(cfg) deactivate_provider() - endpoint_label = "Kimi Coding" if is_coding_plan else "Moonshot" - print(f"Default model set to: {selected} (via {endpoint_label})") + print(f" Default model set to: {selected} (via AWS Bedrock, {region})") else: - print("No change.") + print(" No change.") def _model_flow_api_key_provider(config, provider_id, current_model=""): """Generic flow for API-key providers (z.ai, MiniMax, OpenCode, etc.).""" from hermes_cli.auth import ( - PROVIDER_REGISTRY, _prompt_model_selection, _save_model_choice, + PROVIDER_REGISTRY, + _prompt_model_selection, + _save_model_choice, deactivate_provider, ) - from hermes_cli.config import get_env_value, save_env_value, load_config, save_config - from hermes_cli.models import fetch_api_models, opencode_model_api_mode, normalize_opencode_model_id + from hermes_cli.config import ( + get_env_value, + save_env_value, + load_config, + save_config, + ) + from hermes_cli.models import ( + fetch_api_models, + opencode_model_api_mode, + normalize_opencode_model_id, + ) pconfig = PROVIDER_REGISTRY[provider_id] key_env = pconfig.api_key_env_vars[0] if pconfig.api_key_env_vars else "" @@ -2503,6 +3921,7 @@ def _model_flow_api_key_provider(config, provider_id, current_model=""): if key_env: try: import getpass + new_key = getpass.getpass(f"{key_env} (or Enter to cancel): ").strip() except (KeyboardInterrupt, EOFError): print() @@ -2530,7 +3949,9 @@ def _model_flow_api_key_provider(config, provider_id, current_model=""): override = "" if override and base_url_env: if not override.startswith(("http://", "https://")): - print(" Invalid URL — must start with http:// or https://. Keeping current value.") + print( + " Invalid URL — must start with http:// or https://. Keeping current value." + ) else: save_env_value(base_url_env, override) effective_base = override @@ -2539,37 +3960,69 @@ def _model_flow_api_key_provider(config, provider_id, current_model=""): # 1. models.dev registry (cached, filtered for agentic/tool-capable models) # 2. Curated static fallback list (offline insurance) # 3. Live /models endpoint probe (small providers without models.dev data) - curated = _PROVIDER_MODELS.get(provider_id, []) - - # Try models.dev first — returns tool-capable models, filtered for noise - mdev_models: list = [] - try: - from agent.models_dev import list_agentic_models - mdev_models = list_agentic_models(provider_id) - except Exception: - pass + # + # Ollama Cloud: dedicated merged discovery (live API + models.dev + disk cache) + if provider_id == "ollama-cloud": + from hermes_cli.models import fetch_ollama_cloud_models - if mdev_models: - model_list = mdev_models - print(f" Found {len(model_list)} model(s) from models.dev registry") - elif curated and len(curated) >= 8: - # Curated list is substantial — use it directly, skip live probe - model_list = curated - print(f" Showing {len(model_list)} curated models — use \"Enter custom model name\" for others.") - else: api_key_for_probe = existing_key or (get_env_value(key_env) if key_env else "") - live_models = fetch_api_models(api_key_for_probe, effective_base) - if live_models and len(live_models) >= len(curated): - model_list = live_models - print(f" Found {len(model_list)} model(s) from {pconfig.name} API") - else: + model_list = fetch_ollama_cloud_models( + api_key=api_key_for_probe, base_url=effective_base + ) + if model_list: + print(f" Found {len(model_list)} model(s) from Ollama Cloud") + else: + curated = _PROVIDER_MODELS.get(provider_id, []) + + # Try models.dev first — returns tool-capable models, filtered for noise + mdev_models: list = [] + try: + from agent.models_dev import list_agentic_models + + mdev_models = list_agentic_models(provider_id) + except Exception: + pass + + if mdev_models: + # Merge models.dev with curated list so newly added models + # (not yet in models.dev) still appear in the picker. + if curated: + seen = {m.lower() for m in mdev_models} + merged = list(mdev_models) + for m in curated: + if m.lower() not in seen: + merged.append(m) + seen.add(m.lower()) + model_list = merged + else: + model_list = mdev_models + print(f" Found {len(model_list)} model(s) from models.dev registry") + elif curated and len(curated) >= 8: + # Curated list is substantial — use it directly, skip live probe model_list = curated - if model_list: - print(f" Showing {len(model_list)} curated models — use \"Enter custom model name\" for others.") - # else: no defaults either, will fall through to raw input + print( + f' Showing {len(model_list)} curated models — use "Enter custom model name" for others.' + ) + else: + api_key_for_probe = existing_key or ( + get_env_value(key_env) if key_env else "" + ) + live_models = fetch_api_models(api_key_for_probe, effective_base) + if live_models and len(live_models) >= len(curated): + model_list = live_models + print(f" Found {len(model_list)} model(s) from {pconfig.name} API") + else: + model_list = curated + if model_list: + print( + f' Showing {len(model_list)} curated models — use "Enter custom model name" for others.' + ) + # else: no defaults either, will fall through to raw input if provider_id in {"opencode-zen", "opencode-go"}: - model_list = [normalize_opencode_model_id(provider_id, mid) for mid in model_list] + model_list = [ + normalize_opencode_model_id(provider_id, mid) for mid in model_list + ] current_model = normalize_opencode_model_id(provider_id, current_model) model_list = list(dict.fromkeys(mid for mid in model_list if mid)) @@ -2625,13 +4078,15 @@ def _activate_claude_code_credentials_if_available() -> bool: except Exception: creds = None if creds and ( - is_claude_code_token_valid(creds) - or bool(creds.get("refreshToken")) + is_claude_code_token_valid(creds) or bool(creds.get("refreshToken")) ): use_anthropic_claude_code_credentials(save_fn=save_env_value) print(" ✓ Claude Code credentials linked.") from hermes_constants import display_hermes_home as _dhh_fn - print(f" Hermes will use Claude's credential store directly instead of copying a setup-token into {_dhh_fn()}/.env.") + + print( + f" Hermes will use Claude's credential store directly instead of copying a setup-token into {_dhh_fn()}/.env." + ) return True return False @@ -2654,7 +4109,10 @@ def _activate_claude_code_credentials_if_available() -> bool: print() try: import getpass - manual_token = getpass.getpass(" Paste setup-token (or Enter to cancel): ").strip() + + manual_token = getpass.getpass( + " Paste setup-token (or Enter to cancel): " + ).strip() except (KeyboardInterrupt, EOFError): print() return False @@ -2682,6 +4140,7 @@ def _activate_claude_code_credentials_if_available() -> bool: print() try: import getpass + token = getpass.getpass(" Setup-token (or Enter to cancel): ").strip() except (KeyboardInterrupt, EOFError): print() @@ -2696,23 +4155,30 @@ def _activate_claude_code_credentials_if_available() -> bool: def _model_flow_anthropic(config, current_model=""): """Flow for Anthropic provider — OAuth subscription, API key, or Claude Code creds.""" - import os from hermes_cli.auth import ( - PROVIDER_REGISTRY, _prompt_model_selection, _save_model_choice, + _prompt_model_selection, + _save_model_choice, deactivate_provider, ) from hermes_cli.config import ( - get_env_value, save_env_value, load_config, save_config, + save_env_value, + load_config, + save_config, save_anthropic_api_key, ) from hermes_cli.models import _PROVIDER_MODELS # Check ALL credential sources from hermes_cli.auth import get_anthropic_key + existing_key = get_anthropic_key() cc_available = False try: - from agent.anthropic_adapter import read_claude_code_credentials, is_claude_code_token_valid + from agent.anthropic_adapter import ( + read_claude_code_credentials, + is_claude_code_token_valid, + ) + cc_creds = read_claude_code_credentials() if cc_creds and is_claude_code_token_valid(cc_creds): cc_available = True @@ -2765,10 +4231,11 @@ def _model_flow_anthropic(config, current_model=""): elif choice == "2": print() - print(" Get an API key at: https://console.anthropic.com/settings/keys") + print(" Get an API key at: https://platform.claude.com/settings/keys") print() try: import getpass + api_key = getpass.getpass(" API key (sk-ant-...): ").strip() except (KeyboardInterrupt, EOFError): print() @@ -2819,60 +4286,76 @@ def _model_flow_anthropic(config, current_model=""): def cmd_login(args): """Authenticate Hermes CLI with a provider.""" from hermes_cli.auth import login_command + login_command(args) def cmd_logout(args): """Clear provider authentication.""" from hermes_cli.auth import logout_command + logout_command(args) def cmd_auth(args): """Manage pooled credentials.""" from hermes_cli.auth_commands import auth_command + auth_command(args) def cmd_status(args): """Show status of all components.""" from hermes_cli.status import show_status + show_status(args) def cmd_cron(args): """Cron job management.""" from hermes_cli.cron import cron_command + cron_command(args) def cmd_webhook(args): """Webhook subscription management.""" from hermes_cli.webhook import webhook_command + webhook_command(args) +def cmd_hooks(args): + """Shell-hook inspection and management.""" + from hermes_cli.hooks import hooks_command + hooks_command(args) + + def cmd_doctor(args): """Check configuration and dependencies.""" from hermes_cli.doctor import run_doctor + run_doctor(args) def cmd_dump(args): """Dump setup summary for support/debugging.""" from hermes_cli.dump import run_dump + run_dump(args) def cmd_debug(args): """Debug tools (share report, etc.).""" from hermes_cli.debug import run_debug + run_debug(args) def cmd_config(args): """Configuration management.""" from hermes_cli.config import config_command + config_command(args) @@ -2880,15 +4363,18 @@ def cmd_backup(args): """Back up Hermes home directory to a zip file.""" if getattr(args, "quick", False): from hermes_cli.backup import run_quick_backup + run_quick_backup(args) else: from hermes_cli.backup import run_backup + run_backup(args) def cmd_import(args): """Restore a Hermes backup from a zip file.""" from hermes_cli.backup import run_import + run_import(args) @@ -2896,13 +4382,14 @@ def cmd_version(args): """Show version.""" print(f"Hermes Agent v{__version__} ({__release_date__})") print(f"Project: {PROJECT_ROOT}") - + # Show Python version print(f"Python: {sys.version.split()[0]}") - + # Check for key dependencies try: import openai + print(f"OpenAI SDK: {openai.__version__}") except ImportError: print("OpenAI SDK: Not installed") @@ -2911,6 +4398,7 @@ def cmd_version(args): try: from hermes_cli.banner import check_for_updates from hermes_cli.config import recommended_update_command + behind = check_for_updates() if behind and behind > 0: commits_word = "commit" if behind == 1 else "commits" @@ -2928,6 +4416,7 @@ def cmd_uninstall(args): """Uninstall Hermes Agent.""" _require_tty("uninstall") from hermes_cli.uninstall import run_uninstall + run_uninstall(args) @@ -2945,13 +4434,13 @@ def _clear_bytecode_cache(root: Path) -> int: for dirpath, dirnames, _ in os.walk(root): # Skip venv / node_modules / .git entirely dirnames[:] = [ - d for d in dirnames + d + for d in dirnames if d not in ("venv", ".venv", "node_modules", ".git", ".worktrees") ] if os.path.basename(dirpath) == "__pycache__": try: - import shutil as _shutil - _shutil.rmtree(dirpath) + shutil.rmtree(dirpath) removed += 1 except OSError: pass @@ -2990,7 +4479,6 @@ def _gateway_prompt(prompt_text: str, default: str = "", timeout: float = 300.0) tmp.replace(prompt_path) # Poll for response - import time as _time deadline = _time.monotonic() + timeout while _time.monotonic() < deadline: if response_path.exists(): @@ -3022,7 +4510,7 @@ def _build_web_ui(web_dir: Path, *, fatal: bool = False) -> bool: """ if not (web_dir / "package.json").exists(): return True - import shutil + npm = shutil.which("npm") if not npm: if fatal: @@ -3032,15 +4520,19 @@ def _build_web_ui(web_dir: Path, *, fatal: bool = False) -> bool: print("→ Building web UI...") r1 = subprocess.run([npm, "install", "--silent"], cwd=web_dir, capture_output=True) if r1.returncode != 0: - print(f" {'✗' if fatal else '⚠'} Web UI npm install failed" - + ("" if fatal else " (hermes web will not be available)")) + print( + f" {'✗' if fatal else '⚠'} Web UI npm install failed" + + ("" if fatal else " (hermes web will not be available)") + ) if fatal: print(" Run manually: cd web && npm install && npm run build") return False r2 = subprocess.run([npm, "run", "build"], cwd=web_dir, capture_output=True) if r2.returncode != 0: - print(f" {'✗' if fatal else '⚠'} Web UI build failed" - + ("" if fatal else " (hermes web will not be available)")) + print( + f" {'✗' if fatal else '⚠'} Web UI build failed" + + ("" if fatal else " (hermes web will not be available)") + ) if fatal: print(" Run manually: cd web && npm install && npm run build") return False @@ -3050,34 +4542,40 @@ def _build_web_ui(web_dir: Path, *, fatal: bool = False) -> bool: def _update_via_zip(args): """Update Hermes Agent by downloading a ZIP archive. - - Used on Windows when git file I/O is broken (antivirus, NTFS filter + + Used on Windows when git file I/O is broken (antivirus, NTFS filter drivers causing 'Invalid argument' errors on file creation). """ - import shutil import tempfile import zipfile from urllib.request import urlretrieve - + branch = "main" - zip_url = f"https://github.com/NousResearch/hermes-agent/archive/refs/heads/{branch}.zip" - + zip_url = ( + f"https://github.com/NousResearch/hermes-agent/archive/refs/heads/{branch}.zip" + ) + print("→ Downloading latest version...") try: tmp_dir = tempfile.mkdtemp(prefix="hermes-update-") zip_path = os.path.join(tmp_dir, f"hermes-agent-{branch}.zip") urlretrieve(zip_url, zip_path) - + print("→ Extracting...") - with zipfile.ZipFile(zip_path, 'r') as zf: + with zipfile.ZipFile(zip_path, "r") as zf: # Validate paths to prevent zip-slip (path traversal) tmp_dir_real = os.path.realpath(tmp_dir) for member in zf.infolist(): member_path = os.path.realpath(os.path.join(tmp_dir, member.filename)) - if not member_path.startswith(tmp_dir_real + os.sep) and member_path != tmp_dir_real: - raise ValueError(f"Zip-slip detected: {member.filename} escapes extraction directory") + if ( + not member_path.startswith(tmp_dir_real + os.sep) + and member_path != tmp_dir_real + ): + raise ValueError( + f"Zip-slip detected: {member.filename} escapes extraction directory" + ) zf.extractall(tmp_dir) - + # GitHub ZIPs extract to hermes-agent-/ extracted = os.path.join(tmp_dir, f"hermes-agent-{branch}") if not os.path.isdir(extracted): @@ -3087,9 +4585,9 @@ def _update_via_zip(args): if os.path.isdir(candidate) and d != "__MACOSX": extracted = candidate break - + # Copy updated files over existing installation, preserving venv/node_modules/.git - preserve = {'venv', 'node_modules', '.git', '.env'} + preserve = {"venv", "node_modules", ".git", ".env"} update_count = 0 for item in os.listdir(extracted): if item in preserve: @@ -3103,12 +4601,12 @@ def _update_via_zip(args): else: shutil.copy2(src, dst) update_count += 1 - + print(f"✓ Updated {update_count} items from ZIP") - + # Cleanup shutil.rmtree(tmp_dir, ignore_errors=True) - + except Exception as e: print(f"✗ ZIP update failed: {e}") sys.exit(1) @@ -3116,13 +4614,15 @@ def _update_via_zip(args): # Clear stale bytecode after ZIP extraction removed = _clear_bytecode_cache(PROJECT_ROOT) if removed: - print(f" ✓ Cleared {removed} stale __pycache__ director{'y' if removed == 1 else 'ies'}") - + print( + f" ✓ Cleared {removed} stale __pycache__ director{'y' if removed == 1 else 'ies'}" + ) + # Reinstall Python dependencies. Prefer .[all], but if one optional extra # breaks on this machine, keep base deps and reinstall the remaining extras # individually so update does not silently strip working capabilities. print("→ Updating Python dependencies...") - import subprocess + uv_bin = shutil.which("uv") if uv_bin: uv_env = {**os.environ, "VIRTUAL_ENV": str(PROJECT_ROOT / "venv")} @@ -3134,7 +4634,12 @@ def _update_via_zip(args): # ensurepip before trying the editable install. pip_cmd = [sys.executable, "-m", "pip"] try: - subprocess.run(pip_cmd + ["--version"], cwd=PROJECT_ROOT, check=True, capture_output=True) + subprocess.run( + pip_cmd + ["--version"], + cwd=PROJECT_ROOT, + check=True, + capture_output=True, + ) except subprocess.CalledProcessError: subprocess.run( [sys.executable, "-m", "ensurepip", "--upgrade", "--default-pip"], @@ -3143,18 +4648,21 @@ def _update_via_zip(args): ) _install_python_dependencies_with_optional_fallback(pip_cmd) - # Build web UI frontend (optional — requires npm) + _update_node_dependencies() _build_web_ui(PROJECT_ROOT / "web") # Sync skills try: from tools.skills_sync import sync_skills + print("→ Syncing bundled skills...") result = sync_skills(quiet=True) if result["copied"]: print(f" + {len(result['copied'])} new: {', '.join(result['copied'])}") if result.get("updated"): - print(f" ↑ {len(result['updated'])} updated: {', '.join(result['updated'])}") + print( + f" ↑ {len(result['updated'])} updated: {', '.join(result['updated'])}" + ) if result.get("user_modified"): print(f" ~ {len(result['user_modified'])} user-modified (kept)") if result.get("cleaned"): @@ -3163,7 +4671,7 @@ def _update_via_zip(args): print(" ✓ Skills are up to date") except Exception: pass - + print() print("✓ Update complete!") @@ -3195,7 +4703,9 @@ def _stash_local_changes_if_needed(git_cmd: list[str], cwd: Path) -> Optional[st from datetime import datetime, timezone - stash_name = datetime.now(timezone.utc).strftime("hermes-update-autostash-%Y%m%d-%H%M%S") + stash_name = datetime.now(timezone.utc).strftime( + "hermes-update-autostash-%Y%m%d-%H%M%S" + ) print("→ Local changes detected — stashing before update...") subprocess.run( git_cmd + ["stash", "push", "--include-untracked", "-m", stash_name], @@ -3212,8 +4722,9 @@ def _stash_local_changes_if_needed(git_cmd: list[str], cwd: Path) -> Optional[st return stash_ref - -def _resolve_stash_selector(git_cmd: list[str], cwd: Path, stash_ref: str) -> Optional[str]: +def _resolve_stash_selector( + git_cmd: list[str], cwd: Path, stash_ref: str +) -> Optional[str]: stash_list = subprocess.run( git_cmd + ["stash", "list", "--format=%gd %H"], cwd=cwd, @@ -3228,15 +4739,19 @@ def _resolve_stash_selector(git_cmd: list[str], cwd: Path, stash_ref: str) -> Op return None - -def _print_stash_cleanup_guidance(stash_ref: str, stash_selector: Optional[str] = None) -> None: - print(" Check `git status` first so you don't accidentally reapply the same change twice.") +def _print_stash_cleanup_guidance( + stash_ref: str, stash_selector: Optional[str] = None +) -> None: + print( + " Check `git status` first so you don't accidentally reapply the same change twice." + ) print(" Find the saved entry with: git stash list --format='%gd %H %s'") if stash_selector: print(f" Remove it with: git stash drop {stash_selector}") else: - print(f" Look for commit {stash_ref}, then drop its selector with: git stash drop stash@{{N}}") - + print( + f" Look for commit {stash_ref}, then drop its selector with: git stash drop stash@{{N}}" + ) def _restore_stashed_changes( @@ -3249,7 +4764,9 @@ def _restore_stashed_changes( if prompt_user: print() print("⚠ Local changes were stashed before updating.") - print(" Restoring them may reapply local customizations onto the updated codebase.") + print( + " Restoring them may reapply local customizations onto the updated codebase." + ) print(" Review the result afterward if Hermes behaves unexpectedly.") print("Restore local changes now? [Y/n]") if input_fn is not None: @@ -3313,8 +4830,12 @@ def _restore_stashed_changes( stash_selector = _resolve_stash_selector(git_cmd, cwd, stash_ref) if stash_selector is None: - print("⚠ Local changes were restored, but Hermes couldn't find the stash entry to drop.") - print(" The stash was left in place. You can remove it manually after checking the result.") + print( + "⚠ Local changes were restored, but Hermes couldn't find the stash entry to drop." + ) + print( + " The stash was left in place. You can remove it manually after checking the result." + ) _print_stash_cleanup_guidance(stash_ref) else: drop = subprocess.run( @@ -3324,18 +4845,23 @@ def _restore_stashed_changes( text=True, ) if drop.returncode != 0: - print("⚠ Local changes were restored, but Hermes couldn't drop the saved stash entry.") + print( + "⚠ Local changes were restored, but Hermes couldn't drop the saved stash entry." + ) if drop.stdout.strip(): print(drop.stdout.strip()) if drop.stderr.strip(): print(drop.stderr.strip()) - print(" The stash was left in place. You can remove it manually after checking the result.") + print( + " The stash was left in place. You can remove it manually after checking the result." + ) _print_stash_cleanup_guidance(stash_ref, stash_selector) print("⚠ Local changes were restored on top of the updated codebase.") print(" Review `git diff` / `git status` if Hermes behaves unexpectedly.") return True + # ========================================================================= # Fork detection and upstream management for `hermes update` # ========================================================================= @@ -3430,6 +4956,7 @@ def _count_commits_between(git_cmd: list[str], cwd: Path, base: str, head: str) def _should_skip_upstream_prompt() -> bool: """Check if user previously declined to add upstream.""" from hermes_constants import get_hermes_home + return (get_hermes_home() / SKIP_UPSTREAM_PROMPT_FILE).exists() @@ -3437,6 +4964,7 @@ def _mark_skip_upstream_prompt(): """Create marker file to skip future upstream prompts.""" try: from hermes_constants import get_hermes_home + (get_hermes_home() / SKIP_UPSTREAM_PROMPT_FILE).touch() except Exception: pass @@ -3481,7 +5009,9 @@ def _sync_with_upstream_if_needed(git_cmd: list[str], cwd: Path) -> None: print(" This means you may miss updates from NousResearch/hermes-agent.") print() try: - response = input("Add official repo as 'upstream' remote? [Y/n]: ").strip().lower() + response = ( + input("Add official repo as 'upstream' remote? [Y/n]: ").strip().lower() + ) except (EOFError, KeyboardInterrupt): print() response = "n" @@ -3489,13 +5019,17 @@ def _sync_with_upstream_if_needed(git_cmd: list[str], cwd: Path) -> None: if response in ("", "y", "yes"): print("→ Adding upstream remote...") if _add_upstream_remote(git_cmd, cwd): - print(" ✓ Added upstream: https://github.com/NousResearch/hermes-agent.git") + print( + " ✓ Added upstream: https://github.com/NousResearch/hermes-agent.git" + ) has_upstream = True else: print(" ✗ Failed to add upstream remote. Skipping upstream sync.") return else: - print(" Skipped. Run 'git remote add upstream https://github.com/NousResearch/hermes-agent.git' to add later.") + print( + " Skipped. Run 'git remote add upstream https://github.com/NousResearch/hermes-agent.git' to add later." + ) _mark_skip_upstream_prompt() return @@ -3515,7 +5049,9 @@ def _sync_with_upstream_if_needed(git_cmd: list[str], cwd: Path) -> None: # Compare origin/main with upstream/main origin_ahead = _count_commits_between(git_cmd, cwd, "upstream/main", "origin/main") - upstream_ahead = _count_commits_between(git_cmd, cwd, "origin/main", "upstream/main") + upstream_ahead = _count_commits_between( + git_cmd, cwd, "origin/main", "upstream/main" + ) if origin_ahead < 0 or upstream_ahead < 0: print(" ✗ Could not compare branches. Skipping upstream sync.") @@ -3547,7 +5083,9 @@ def _sync_with_upstream_if_needed(git_cmd: list[str], cwd: Path) -> None: check=True, ) except subprocess.CalledProcessError: - print(" ✗ Failed to pull from upstream. You may need to resolve conflicts manually.") + print( + " ✗ Failed to pull from upstream. You may need to resolve conflicts manually." + ) return print(" ✓ Updated from upstream") @@ -3557,7 +5095,9 @@ def _sync_with_upstream_if_needed(git_cmd: list[str], cwd: Path) -> None: if _sync_fork_with_upstream(git_cmd, cwd): print(" ✓ Fork synced with upstream") else: - print(" ℹ Got updates from upstream but couldn't push to fork (no write access?)") + print( + " ℹ Got updates from upstream but couldn't push to fork (no write access?)" + ) print(" Your local repo is updated, but your fork on GitHub may be behind.") @@ -3571,6 +5111,7 @@ def _invalidate_update_cache(): homes = [] # Default profile home (Docker-aware — uses /opt/data in Docker) from hermes_constants import get_default_hermes_root + default_home = get_default_hermes_root() homes.append(default_home) # Named profiles under /profiles/ @@ -3598,6 +5139,7 @@ def _load_installable_optional_extras() -> list[str]: """ try: import tomllib + with (PROJECT_ROOT / "pyproject.toml").open("rb") as handle: project = tomllib.load(handle).get("project", {}) except Exception: @@ -3620,7 +5162,6 @@ def _load_installable_optional_extras() -> list[str]: return referenced - def _install_python_dependencies_with_optional_fallback( install_cmd_prefix: list[str], *, @@ -3636,7 +5177,9 @@ def _install_python_dependencies_with_optional_fallback( ) return except subprocess.CalledProcessError: - print(" ⚠ Optional extras failed, reinstalling base dependencies and retrying extras individually...") + print( + " ⚠ Optional extras failed, reinstalling base dependencies and retrying extras individually..." + ) subprocess.run( install_cmd_prefix + ["install", "-e", ".", "--quiet"], @@ -3660,14 +5203,232 @@ def _install_python_dependencies_with_optional_fallback( failed_extras.append(extra) if installed_extras: - print(f" ✓ Reinstalled optional extras individually: {', '.join(installed_extras)}") + print( + f" ✓ Reinstalled optional extras individually: {', '.join(installed_extras)}" + ) if failed_extras: - print(f" ⚠ Skipped optional extras that still failed: {', '.join(failed_extras)}") + print( + f" ⚠ Skipped optional extras that still failed: {', '.join(failed_extras)}" + ) + + +def _update_node_dependencies() -> None: + npm = shutil.which("npm") + if not npm: + return + + paths = ( + ("repo root", PROJECT_ROOT), + ("ui-tui", PROJECT_ROOT / "ui-tui"), + ) + if not any((path / "package.json").exists() for _, path in paths): + return + + print("→ Updating Node.js dependencies...") + for label, path in paths: + if not (path / "package.json").exists(): + continue + + result = subprocess.run( + [npm, "install", "--silent", "--no-fund", "--no-audit", "--progress=false"], + cwd=path, + capture_output=True, + text=True, + check=False, + ) + if result.returncode == 0: + print(f" ✓ {label}") + continue + + print(f" ⚠ npm install failed in {label}") + stderr = (result.stderr or "").strip() + if stderr: + print(f" {stderr.splitlines()[-1]}") + + +class _UpdateOutputStream: + """Stream wrapper used during ``hermes update`` to survive terminal loss. + + Wraps the process's original stdout/stderr so that: + + * Every write is also mirrored to an append-only log file + (``~/.hermes/logs/update.log``) that users can inspect after the + terminal disconnects. + * Writes to the original stream that fail with ``BrokenPipeError`` / + ``OSError`` / ``ValueError`` (closed file) no longer cascade into + process exit — the update keeps going, only the on-screen output + stops. + + Combined with ``SIGHUP -> SIG_IGN`` installed by + ``_install_hangup_protection``, this makes ``hermes update`` safe to + run in a plain SSH session that might disconnect mid-install. + """ + + def __init__(self, original, log_file): + self._original = original + self._log = log_file + self._original_broken = False + + def write(self, data): + # Mirror to the log file first — it's the most reliable destination. + if self._log is not None: + try: + self._log.write(data) + except Exception: + # Log errors should never abort the update. + pass + + if self._original_broken: + return len(data) if isinstance(data, (str, bytes)) else 0 + + try: + return self._original.write(data) + except (BrokenPipeError, OSError, ValueError): + # Terminal vanished (SSH disconnect, shell close). Stop trying + # to write to it, but keep the update running. + self._original_broken = True + return len(data) if isinstance(data, (str, bytes)) else 0 + + def flush(self): + if self._log is not None: + try: + self._log.flush() + except Exception: + pass + if self._original_broken: + return + try: + self._original.flush() + except (BrokenPipeError, OSError, ValueError): + self._original_broken = True + + def isatty(self): + if self._original_broken: + return False + try: + return self._original.isatty() + except Exception: + return False + + def fileno(self): + # Some tools probe fileno(); defer to the underlying stream and let + # callers handle failures (same behaviour as the unwrapped stream). + return self._original.fileno() + + def __getattr__(self, name): + return getattr(self._original, name) + + +def _install_hangup_protection(gateway_mode: bool = False): + """Protect ``cmd_update`` from SIGHUP and broken terminal pipes. + + Users commonly run ``hermes update`` in an SSH session or a terminal + that may close mid-install. Without protection, ``SIGHUP`` from the + terminal kills the Python process during ``pip install`` and leaves + the venv half-installed; the documented workaround ("use screen / + tmux") shouldn't be required for something as routine as an update. + + Protections installed: + + 1. ``SIGHUP`` is set to ``SIG_IGN``. POSIX preserves ``SIG_IGN`` + across ``exec()``, so pip and git subprocesses also stop dying on + hangup. + 2. ``sys.stdout`` / ``sys.stderr`` are wrapped to mirror output to + ``~/.hermes/logs/update.log`` and to silently absorb + ``BrokenPipeError`` when the terminal vanishes. + + ``SIGINT`` (Ctrl-C) and ``SIGTERM`` (systemd shutdown) are + **intentionally left alone** — those are legitimate cancellation + signals the user or OS sent on purpose. + + In gateway mode (``hermes update --gateway``) the update is already + spawned detached from a terminal, so this function is a no-op. + + Returns a dict that ``cmd_update`` can pass to + ``_finalize_update_output`` on exit. Returning a dict rather than a + tuple keeps the call site forward-compatible with future additions. + """ + state = { + "prev_stdout": sys.stdout, + "prev_stderr": sys.stderr, + "log_file": None, + "installed": False, + } + + if gateway_mode: + return state + + import signal as _signal + + # (1) Ignore SIGHUP for the remainder of this process. + if hasattr(_signal, "SIGHUP"): + try: + _signal.signal(_signal.SIGHUP, _signal.SIG_IGN) + except (ValueError, OSError): + # Called from a non-main thread — not fatal. The update still + # runs, just without hangup protection. + pass + + # (2) Mirror output to update.log and wrap stdio for broken-pipe + # tolerance. Any failure here is non-fatal; we just skip the wrap. + try: + # Late-bound import so tests can monkeypatch + # hermes_cli.config.get_hermes_home to simulate setup failure. + from hermes_cli.config import get_hermes_home as _get_hermes_home + + logs_dir = _get_hermes_home() / "logs" + logs_dir.mkdir(parents=True, exist_ok=True) + log_path = logs_dir / "update.log" + log_file = open(log_path, "a", buffering=1, encoding="utf-8") + + import datetime as _dt + + log_file.write( + f"\n=== hermes update started " + f"{_dt.datetime.now().isoformat(timespec='seconds')} ===\n" + ) + + state["log_file"] = log_file + sys.stdout = _UpdateOutputStream(state["prev_stdout"], log_file) + sys.stderr = _UpdateOutputStream(state["prev_stderr"], log_file) + state["installed"] = True + except Exception: + # Leave stdio untouched on any setup failure. Update continues + # without mirroring. + state["log_file"] = None + + return state + + +def _finalize_update_output(state): + """Restore stdio and close the update.log handle opened by ``_install_hangup_protection``.""" + if not state: + return + if state.get("installed"): + try: + sys.stdout = state.get("prev_stdout", sys.stdout) + except Exception: + pass + try: + sys.stderr = state.get("prev_stderr", sys.stderr) + except Exception: + pass + log_file = state.get("log_file") + if log_file is not None: + try: + log_file.flush() + log_file.close() + except Exception: + pass def cmd_update(args): - """Update Hermes Agent to the latest version.""" - import shutil + """Update Hermes Agent to the latest version. + + Thin wrapper around ``_cmd_update_impl``: installs hangup protection, + runs the update, then restores stdio on the way out (even on + ``sys.exit`` or unhandled exceptions). + """ from hermes_cli.config import is_managed, managed_error if is_managed(): @@ -3675,31 +5436,60 @@ def cmd_update(args): return gateway_mode = getattr(args, "gateway", False) + + # Protect against mid-update terminal disconnects (SIGHUP) and tolerate + # writes to a closed stdout. No-op in gateway mode. See + # _install_hangup_protection for rationale. + _update_io_state = _install_hangup_protection(gateway_mode=gateway_mode) + try: + _cmd_update_impl(args, gateway_mode=gateway_mode) + finally: + _finalize_update_output(_update_io_state) + + +def _cmd_update_impl(args, gateway_mode: bool): + """Body of ``cmd_update`` — kept separate so the wrapper can always + restore stdio even on ``sys.exit``.""" # In gateway mode, use file-based IPC for prompts instead of stdin - gw_input_fn = (lambda prompt, default="": _gateway_prompt(prompt, default)) if gateway_mode else None - + gw_input_fn = ( + (lambda prompt, default="": _gateway_prompt(prompt, default)) + if gateway_mode + else None + ) + print("⚕ Updating Hermes Agent...") print() - + # Try git-based update first, fall back to ZIP download on Windows # when git file I/O is broken (antivirus, NTFS filter drivers, etc.) use_zip_update = False - git_dir = PROJECT_ROOT / '.git' - + git_dir = PROJECT_ROOT / ".git" + if not git_dir.exists(): if sys.platform == "win32": use_zip_update = True else: print("✗ Not a git repository. Please reinstall:") - print(" curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash") + print( + " curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash" + ) sys.exit(1) - + # On Windows, git can fail with "unable to write loose object file: Invalid argument" # due to filesystem atomicity issues. Set the recommended workaround. if sys.platform == "win32" and git_dir.exists(): subprocess.run( - ["git", "-c", "windows.appendAtomically=false", "config", "windows.appendAtomically", "false"], - cwd=PROJECT_ROOT, check=False, capture_output=True + [ + "git", + "-c", + "windows.appendAtomically=false", + "config", + "windows.appendAtomically", + "false", + ], + cwd=PROJECT_ROOT, + check=False, + capture_output=True, ) # Build git command once — reused for fork detection and the update itself. @@ -3736,8 +5526,12 @@ def cmd_update(args): if "Could not resolve host" in stderr or "unable to access" in stderr: print("✗ Network error — cannot reach the remote repository.") print(f" {stderr.splitlines()[0]}" if stderr else "") - elif "Authentication failed" in stderr or "could not read Username" in stderr: - print("✗ Authentication failed — check your git credentials or SSH key.") + elif ( + "Authentication failed" in stderr or "could not read Username" in stderr + ): + print( + "✗ Authentication failed — check your git credentials or SSH key." + ) else: print(f"✗ Failed to fetch updates from origin.") if stderr: @@ -3759,7 +5553,11 @@ def cmd_update(args): # If user is on a non-main branch or detached HEAD, switch to main if current_branch != "main": - label = "detached HEAD" if current_branch == "HEAD" else f"branch '{current_branch}'" + label = ( + "detached HEAD" + if current_branch == "HEAD" + else f"branch '{current_branch}'" + ) print(f" ⚠ Currently on {label} — switching to main for update...") # Stash before checkout so uncommitted work isn't lost auto_stash_ref = _stash_local_changes_if_needed(git_cmd, PROJECT_ROOT) @@ -3792,14 +5590,19 @@ def cmd_update(args): # Restore stash and switch back to original branch if we moved if auto_stash_ref is not None: _restore_stashed_changes( - git_cmd, PROJECT_ROOT, auto_stash_ref, + git_cmd, + PROJECT_ROOT, + auto_stash_ref, prompt_user=prompt_for_restore, input_fn=gw_input_fn, ) if current_branch not in ("main", "HEAD"): subprocess.run( git_cmd + ["checkout", current_branch], - cwd=PROJECT_ROOT, capture_output=True, text=True, check=False, + cwd=PROJECT_ROOT, + capture_output=True, + text=True, + check=False, ) print("✓ Already up to date!") return @@ -3819,7 +5622,9 @@ def cmd_update(args): # ff-only failed — local and remote have diverged (e.g. upstream # force-pushed or rebase). Since local changes are already # stashed, reset to match the remote exactly. - print(" ⚠ Fast-forward not possible (history diverged), resetting to match remote...") + print( + " ⚠ Fast-forward not possible (history diverged), resetting to match remote..." + ) reset_result = subprocess.run( git_cmd + ["reset", "--hard", f"origin/{branch}"], cwd=PROJECT_ROOT, @@ -3830,7 +5635,9 @@ def cmd_update(args): print(f"✗ Failed to reset to origin/{branch}.") if reset_result.stderr.strip(): print(f" {reset_result.stderr.strip()}") - print(" Try manually: git fetch origin && git reset --hard origin/main") + print( + " Try manually: git fetch origin && git reset --hard origin/main" + ) sys.exit(1) update_succeeded = True finally: @@ -3838,7 +5645,9 @@ def cmd_update(args): # Don't attempt stash restore if the code update itself failed — # working tree is in an unknown state. if not update_succeeded: - print(f" ℹ️ Local changes preserved in stash (ref: {auto_stash_ref})") + print( + f" ℹ️ Local changes preserved in stash (ref: {auto_stash_ref})" + ) print(f" Restore manually with: git stash apply") else: _restore_stashed_changes( @@ -3848,7 +5657,7 @@ def cmd_update(args): prompt_user=prompt_for_restore, input_fn=gw_input_fn, ) - + _invalidate_update_cache() # Clear stale .pyc bytecode cache — prevents ImportError on gateway @@ -3856,12 +5665,14 @@ def cmd_update(args): # the old bytecode (e.g. get_hermes_home added to hermes_constants). removed = _clear_bytecode_cache(PROJECT_ROOT) if removed: - print(f" ✓ Cleared {removed} stale __pycache__ director{'y' if removed == 1 else 'ies'}") + print( + f" ✓ Cleared {removed} stale __pycache__ director{'y' if removed == 1 else 'ies'}" + ) # Fork upstream sync logic (only for main branch on forks) if is_fork and branch == "main": _sync_with_upstream_if_needed(git_cmd, PROJECT_ROOT) - + # Reinstall Python dependencies. Prefer .[all], but if one optional extra # breaks on this machine, keep base deps and reinstall the remaining extras # individually so update does not silently strip working capabilities. @@ -3869,7 +5680,9 @@ def cmd_update(args): uv_bin = shutil.which("uv") if uv_bin: uv_env = {**os.environ, "VIRTUAL_ENV": str(PROJECT_ROOT / "venv")} - _install_python_dependencies_with_optional_fallback([uv_bin, "pip"], env=uv_env) + _install_python_dependencies_with_optional_fallback( + [uv_bin, "pip"], env=uv_env + ) else: # Use sys.executable to explicitly call the venv's pip module, # avoiding PEP 668 'externally-managed-environment' errors on Debian/Ubuntu. @@ -3877,7 +5690,12 @@ def cmd_update(args): # ensurepip before trying the editable install. pip_cmd = [sys.executable, "-m", "pip"] try: - subprocess.run(pip_cmd + ["--version"], cwd=PROJECT_ROOT, check=True, capture_output=True) + subprocess.run( + pip_cmd + ["--version"], + cwd=PROJECT_ROOT, + check=True, + capture_output=True, + ) except subprocess.CalledProcessError: subprocess.run( [sys.executable, "-m", "ensurepip", "--upgrade", "--default-pip"], @@ -3885,20 +5703,13 @@ def cmd_update(args): check=True, ) _install_python_dependencies_with_optional_fallback(pip_cmd) - - # Check for Node.js deps - if (PROJECT_ROOT / "package.json").exists(): - import shutil - if shutil.which("npm"): - print("→ Updating Node.js dependencies...") - subprocess.run(["npm", "install", "--silent"], cwd=PROJECT_ROOT, check=False) - - # Build web UI frontend (optional — requires npm) + + _update_node_dependencies() _build_web_ui(PROJECT_ROOT / "web") print() print("✓ Code updated!") - + # After git pull, source files on disk are newer than cached Python # modules in this process. Reload hermes_constants so that any lazy # import executed below (skills sync, gateway restart) sees new @@ -3906,20 +5717,24 @@ def cmd_update(args): try: import importlib import hermes_constants as _hc + importlib.reload(_hc) except Exception: pass # non-fatal — worst case a lazy import fails gracefully - + # Sync bundled skills (copies new, updates changed, respects user deletions) try: from tools.skills_sync import sync_skills + print() print("→ Syncing bundled skills...") result = sync_skills(quiet=True) if result["copied"]: print(f" + {len(result['copied'])} new: {', '.join(result['copied'])}") if result.get("updated"): - print(f" ↑ {len(result['updated'])} updated: {', '.join(result['updated'])}") + print( + f" ↑ {len(result['updated'])} updated: {', '.join(result['updated'])}" + ) if result.get("user_modified"): print(f" ~ {len(result['user_modified'])} user-modified (kept)") if result.get("cleaned"): @@ -3931,7 +5746,12 @@ def cmd_update(args): # Sync bundled skills to all other profiles try: - from hermes_cli.profiles import list_profiles, get_active_profile_name, seed_profile_skills + from hermes_cli.profiles import ( + list_profiles, + get_active_profile_name, + seed_profile_skills, + ) + active = get_active_profile_name() other_profiles = [p for p in list_profiles() if p.name != active] if other_profiles: @@ -3945,9 +5765,12 @@ def cmd_update(args): updated = len(r.get("updated", [])) modified = len(r.get("user_modified", [])) parts = [] - if copied: parts.append(f"+{copied} new") - if updated: parts.append(f"↑{updated} updated") - if modified: parts.append(f"~{modified} user-modified") + if copied: + parts.append(f"+{copied} new") + if updated: + parts.append(f"↑{updated} updated") + if modified: + parts.append(f"~{modified} user-modified") status = ", ".join(parts) if parts else "up to date" else: status = "sync failed" @@ -3960,6 +5783,7 @@ def cmd_update(args): # Sync Honcho host blocks to all profiles try: from plugins.memory.honcho.cli import sync_honcho_profiles_quiet + synced = sync_honcho_profiles_quiet() if synced: print(f"\n-> Honcho: synced {synced} profile(s)") @@ -3969,46 +5793,60 @@ def cmd_update(args): # Check for config migrations print() print("→ Checking configuration for new options...") - + from hermes_cli.config import ( - get_missing_env_vars, get_missing_config_fields, - check_config_version, migrate_config + get_missing_env_vars, + get_missing_config_fields, + check_config_version, + migrate_config, ) - + missing_env = get_missing_env_vars(required_only=True) missing_config = get_missing_config_fields() current_ver, latest_ver = check_config_version() - + needs_migration = missing_env or missing_config or current_ver < latest_ver - + if needs_migration: print() if missing_env: - print(f" ⚠️ {len(missing_env)} new required setting(s) need configuration") + print( + f" ⚠️ {len(missing_env)} new required setting(s) need configuration" + ) if missing_config: print(f" ℹ️ {len(missing_config)} new config option(s) available") - + print() if gateway_mode: - response = _gateway_prompt( - "Would you like to configure new options now? [Y/n]", "n" - ).strip().lower() + response = ( + _gateway_prompt( + "Would you like to configure new options now? [Y/n]", "n" + ) + .strip() + .lower() + ) elif not (sys.stdin.isatty() and sys.stdout.isatty()): print(" ℹ Non-interactive session — skipping config migration prompt.") - print(" Run 'hermes config migrate' later to apply any new config/env options.") + print( + " Run 'hermes config migrate' later to apply any new config/env options." + ) response = "n" else: try: - response = input("Would you like to configure them now? [Y/n]: ").strip().lower() + response = ( + input("Would you like to configure them now? [Y/n]: ") + .strip() + .lower() + ) except EOFError: response = "n" - - if response in ('', 'y', 'yes'): + + if response in ("", "y", "yes"): print() # In gateway mode, run auto-migrations only (no input() prompts # for API keys which would hang the detached process). results = migrate_config(interactive=not gateway_mode, quiet=False) - + if results["env_added"] or results["config_added"]: print() print("✓ Configuration updated!") @@ -4019,19 +5857,22 @@ def cmd_update(args): print("Skipped. Run 'hermes config migrate' later to configure.") else: print(" ✓ Configuration is up to date") - + print() print("✓ Update complete!") - + # Write exit code *before* the gateway restart attempt. # When running as ``hermes update --gateway`` (spawned by the gateway's # /update command), this process lives inside the gateway's systemd - # cgroup. ``systemctl restart hermes-gateway`` kills everything in the - # cgroup (KillMode=mixed → SIGKILL to remaining processes), including - # us and the wrapping bash shell. The shell never reaches its - # ``printf $status > .update_exit_code`` epilogue, so the exit-code - # marker file is never created. The new gateway's update watcher then - # polls for 30 minutes and sends a spurious timeout message. + # cgroup. A graceful SIGUSR1 restart keeps the drain loop alive long + # enough for the exit-code marker to be written below, but the + # fallback ``systemctl restart`` path (see below) kills everything in + # the cgroup (KillMode=mixed → SIGKILL to remaining processes), + # including us and the wrapping bash shell. The shell never reaches + # its ``printf $status > .update_exit_code`` epilogue, so the + # exit-code marker file would never be created. The new gateway's + # update watcher would then poll for 30 minutes and send a spurious + # timeout message. # # Writing the marker here — after git pull + pip install succeed but # before we attempt the restart — ensures the new gateway sees it @@ -4042,18 +5883,48 @@ def cmd_update(args): _exit_code_path.write_text("0") except OSError: pass - + # Auto-restart ALL gateways after update. # The code update (git pull) is shared across all profiles, so every # running gateway needs restarting to pick up the new code. try: from hermes_cli.gateway import ( - is_macos, supports_systemd_services, _ensure_user_systemd_env, + is_macos, + supports_systemd_services, + _ensure_user_systemd_env, find_gateway_pids, _get_service_pids, + _graceful_restart_via_sigusr1, ) import signal as _signal + # Drain budget for graceful SIGUSR1 restarts. The gateway drains + # for up to ``agent.restart_drain_timeout`` (default 60s) before + # exiting with code 75; we wait slightly longer so the drain + # completes before we fall back to a hard restart. On older + # systemd units without SIGUSR1 wiring this wait just times out + # and we fall back to ``systemctl restart`` (the old behaviour). + try: + from hermes_constants import ( + DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT as _DEFAULT_DRAIN, + ) + except Exception: + _DEFAULT_DRAIN = 60.0 + _cfg_drain = None + try: + from hermes_cli.config import load_config + _cfg_agent = (load_config().get("agent") or {}) + _cfg_drain = _cfg_agent.get("restart_drain_timeout") + except Exception: + pass + try: + _drain_budget = float(_cfg_drain) if _cfg_drain is not None else float(_DEFAULT_DRAIN) + except (TypeError, ValueError): + _drain_budget = float(_DEFAULT_DRAIN) + # Add a 15s margin so the drain loop + final exit finish before + # we escalate to ``systemctl restart`` / SIGTERM. + _drain_budget = max(_drain_budget, 30.0) + 15.0 + restarted_services = [] killed_pids = set() @@ -4065,46 +5936,168 @@ def cmd_update(args): except Exception: pass - for scope, scope_cmd in [("user", ["systemctl", "--user"]), ("system", ["systemctl"])]: + for scope, scope_cmd in [ + ("user", ["systemctl", "--user"]), + ("system", ["systemctl"]), + ]: try: result = subprocess.run( - scope_cmd + ["list-units", "hermes-gateway*", "--plain", "--no-legend", "--no-pager"], - capture_output=True, text=True, timeout=10, + scope_cmd + + [ + "list-units", + "hermes-gateway*", + "--plain", + "--no-legend", + "--no-pager", + ], + capture_output=True, + text=True, + timeout=10, ) for line in result.stdout.strip().splitlines(): parts = line.split() if not parts: continue - unit = parts[0] # e.g. hermes-gateway.service or hermes-gateway-coder.service + unit = parts[ + 0 + ] # e.g. hermes-gateway.service or hermes-gateway-coder.service if not unit.endswith(".service"): continue svc_name = unit.removesuffix(".service") # Check if active check = subprocess.run( scope_cmd + ["is-active", svc_name], - capture_output=True, text=True, timeout=5, + capture_output=True, + text=True, + timeout=5, + ) + if check.stdout.strip() != "active": + continue + + # Prefer a graceful SIGUSR1 restart so in-flight + # agent runs drain instead of being SIGKILLed. + # The gateway's SIGUSR1 handler calls + # request_restart(via_service=True) → drain → + # exit(75); systemd's Restart=on-failure (and + # RestartForceExitStatus=75) respawns the unit. + _main_pid = 0 + try: + _show = subprocess.run( + scope_cmd + [ + "show", svc_name, + "--property=MainPID", "--value", + ], + capture_output=True, text=True, timeout=5, + ) + _main_pid = int((_show.stdout or "").strip() or 0) + except (ValueError, subprocess.TimeoutExpired, FileNotFoundError): + _main_pid = 0 + + _graceful_ok = False + if _main_pid > 0: + print( + f" → {svc_name}: draining (up to {int(_drain_budget)}s)..." + ) + _graceful_ok = _graceful_restart_via_sigusr1( + _main_pid, drain_timeout=_drain_budget, + ) + + if _graceful_ok: + # Gateway exited 75; systemd should relaunch + # via Restart=on-failure. Verify the new + # process came up. + _time.sleep(3) + verify = subprocess.run( + scope_cmd + ["is-active", svc_name], + capture_output=True, text=True, timeout=5, + ) + if verify.stdout.strip() == "active": + restarted_services.append(svc_name) + continue + # Process exited but wasn't respawned (older + # unit without Restart=on-failure or + # RestartForceExitStatus=75). Fall through + # to systemctl start/restart. + print( + f" ⚠ {svc_name} drained but didn't relaunch — forcing restart" + ) + + # Fallback: blunt systemctl restart. This is + # what the old code always did; we get here only + # when the graceful path failed (unit missing + # SIGUSR1 wiring, drain exceeded the budget, + # restart-policy mismatch). + restart = subprocess.run( + scope_cmd + ["restart", svc_name], + capture_output=True, + text=True, + timeout=15, ) - if check.stdout.strip() == "active": - restart = subprocess.run( - scope_cmd + ["restart", svc_name], - capture_output=True, text=True, timeout=15, + if restart.returncode == 0: + # Verify the service actually survived the + # restart. systemctl restart returns 0 even + # if the new process crashes immediately. + _time.sleep(3) + verify = subprocess.run( + scope_cmd + ["is-active", svc_name], + capture_output=True, + text=True, + timeout=5, ) - if restart.returncode == 0: + if verify.stdout.strip() == "active": restarted_services.append(svc_name) else: - print(f" ⚠ Failed to restart {svc_name}: {restart.stderr.strip()}") + # Retry once — transient startup failures + # (stale module cache, import race) often + # resolve on the second attempt. + print( + f" ⚠ {svc_name} died after restart, retrying..." + ) + retry = subprocess.run( + scope_cmd + ["restart", svc_name], + capture_output=True, + text=True, + timeout=15, + ) + _time.sleep(3) + verify2 = subprocess.run( + scope_cmd + ["is-active", svc_name], + capture_output=True, + text=True, + timeout=5, + ) + if verify2.stdout.strip() == "active": + restarted_services.append(svc_name) + print(f" ✓ {svc_name} recovered on retry") + else: + print( + f" ✗ {svc_name} failed to stay running after restart.\n" + f" Check logs: journalctl --user -u {svc_name} --since '2 min ago'\n" + f" Restart manually: systemctl {'--user ' if scope == 'user' else ''}restart {svc_name}" + ) + else: + print( + f" ⚠ Failed to restart {svc_name}: {restart.stderr.strip()}" + ) except (FileNotFoundError, subprocess.TimeoutExpired): pass # --- Launchd services (macOS) --- if is_macos(): try: - from hermes_cli.gateway import launchd_restart, get_launchd_label, get_launchd_plist_path + from hermes_cli.gateway import ( + launchd_restart, + get_launchd_label, + get_launchd_plist_path, + ) + plist_path = get_launchd_plist_path() if plist_path.exists(): check = subprocess.run( ["launchctl", "list", get_launchd_label()], - capture_output=True, text=True, timeout=5, + capture_output=True, + text=True, + timeout=5, ) if check.returncode == 0: try: @@ -4121,7 +6114,9 @@ def cmd_update(args): # Exclude PIDs that belong to just-restarted services so we don't # immediately kill the process that systemd/launchd just spawned. service_pids = _get_service_pids() - manual_pids = find_gateway_pids(exclude_pids=service_pids, all_profiles=True) + manual_pids = find_gateway_pids( + exclude_pids=service_pids, all_profiles=True + ) for pid in manual_pids: try: os.kill(pid, _signal.SIGTERM) @@ -4138,7 +6133,9 @@ def cmd_update(args): print(" Restart manually: hermes gateway run") # Also restart for each profile if needed if len(killed_pids) > 1: - print(" (or: hermes -p gateway run for each profile)") + print( + " (or: hermes -p gateway run for each profile)" + ) if not restarted_services and not killed_pids: # No gateways were running — nothing to do @@ -4146,11 +6143,40 @@ def cmd_update(args): except Exception as e: logger.debug("Gateway restart during update failed: %s", e) - + + # Warn if legacy Hermes gateway unit files are still installed. + # When both hermes.service (from a pre-rename install) and the + # current hermes-gateway.service are enabled, they SIGTERM-fight + # for the same bot token (see PR #11909). Flagging here means + # every `hermes update` surfaces the issue until the user migrates. + try: + from hermes_cli.gateway import ( + has_legacy_hermes_units, + _find_legacy_hermes_units, + supports_systemd_services, + ) + + if supports_systemd_services() and has_legacy_hermes_units(): + print() + print("⚠ Legacy Hermes gateway unit(s) detected:") + for name, path, is_sys in _find_legacy_hermes_units(): + scope = "system" if is_sys else "user" + print(f" {path} ({scope} scope)") + print() + print(" These pre-rename units (hermes.service) fight the current") + print(" hermes-gateway.service for the bot token and cause SIGTERM") + print(" flap loops. Remove them with:") + print() + print(" hermes gateway migrate-legacy") + print() + print(" (add `sudo` if any are in system scope)") + except Exception as e: + logger.debug("Legacy unit check during update failed: %s", e) + print() print("Tip: You can now select a provider and model:") print(" hermes model # Select provider and model") - + except subprocess.CalledProcessError as e: if sys.platform == "win32": print(f"⚠ Git update failed: {e}") @@ -4174,10 +6200,41 @@ def _coalesce_session_name_args(argv: list) -> list: or a known top-level subcommand. """ _SUBCOMMANDS = { - "chat", "model", "gateway", "setup", "whatsapp", "login", "logout", "auth", - "status", "cron", "doctor", "config", "pairing", "skills", "tools", - "mcp", "sessions", "insights", "version", "update", "uninstall", - "profile", "dashboard", + "chat", + "model", + "gateway", + "setup", + "whatsapp", + "login", + "logout", + "auth", + "status", + "cron", + "doctor", + "config", + "pairing", + "skills", + "tools", + "mcp", + "sessions", + "insights", + "version", + "update", + "uninstall", + "profile", + "dashboard", + "honcho", + "claw", + "plugins", + "acp", + "webhook", + "memory", + "dump", + "debug", + "backup", + "import", + "completion", + "logs", } _SESSION_FLAGS = {"-c", "--continue", "-r", "--resume"} @@ -4190,7 +6247,11 @@ def _coalesce_session_name_args(argv: list) -> list: i += 1 # Collect subsequent non-flag, non-subcommand tokens as one name parts: list = [] - while i < len(argv) and not argv[i].startswith("-") and argv[i] not in _SUBCOMMANDS: + while ( + i < len(argv) + and not argv[i].startswith("-") + and argv[i] not in _SUBCOMMANDS + ): parts.append(argv[i]) i += 1 if parts: @@ -4204,10 +6265,17 @@ def _coalesce_session_name_args(argv: list) -> list: def cmd_profile(args): """Profile management — create, delete, list, switch, alias.""" from hermes_cli.profiles import ( - list_profiles, create_profile, delete_profile, seed_profile_skills, - set_active_profile, get_active_profile_name, - check_alias_collision, create_wrapper_script, remove_wrapper_script, - _is_wrapper_dir_in_path, _get_wrapper_dir, + list_profiles, + create_profile, + delete_profile, + seed_profile_skills, + set_active_profile, + get_active_profile_name, + check_alias_collision, + create_wrapper_script, + remove_wrapper_script, + _is_wrapper_dir_in_path, + _get_wrapper_dir, ) from hermes_constants import display_hermes_home @@ -4224,8 +6292,13 @@ def cmd_profile(args): for p in profiles: if p.name == profile_name or (profile_name == "default" and p.is_default): if p.model: - print(f"Model: {p.model}" + (f" ({p.provider})" if p.provider else "")) - print(f"Gateway: {'running' if p.gateway_running else 'stopped'}") + print( + f"Model: {p.model}" + + (f" ({p.provider})" if p.provider else "") + ) + print( + f"Gateway: {'running' if p.gateway_running else 'stopped'}" + ) print(f"Skills: {p.skill_count} installed") if p.alias_path: print(f"Alias: {p.name} → hermes -p {p.name}") @@ -4246,7 +6319,11 @@ def cmd_profile(args): print(f" {'─' * 15} {'─' * 27} {'─' * 11} {'─' * 12}") for p in profiles: - marker = " ◆" if (p.name == active or (active == "default" and p.is_default)) else " " + marker = ( + " ◆" + if (p.name == active or (active == "default" and p.is_default)) + else " " + ) name = p.name model = (p.model or "—")[:26] gw = "running" if p.gateway_running else "stopped" @@ -4287,7 +6364,9 @@ def cmd_profile(args): print(f"\nProfile '{name}' created at {profile_dir}") if clone or clone_all: - source_label = getattr(args, "clone_from", None) or get_active_profile_name() + source_label = ( + getattr(args, "clone_from", None) or get_active_profile_name() + ) if clone_all: print(f"Full copy from {source_label}.") else: @@ -4297,6 +6376,7 @@ def cmd_profile(args): if clone or clone_all: try: from plugins.memory.honcho.cli import clone_honcho_for_profile + if clone_honcho_for_profile(name): print(f"Honcho config cloned (peer: {name})") except Exception: @@ -4309,14 +6389,20 @@ def cmd_profile(args): copied = len(result.get("copied", [])) print(f"{copied} bundled skills synced.") else: - print("⚠ Skills could not be seeded. Run `{} update` to retry.".format(name)) + print( + "⚠ Skills could not be seeded. Run `{} update` to retry.".format( + name + ) + ) # Create wrapper alias if not no_alias: collision = check_alias_collision(name) if collision: print(f"\n⚠ Cannot create alias '{name}' — {collision}") - print(f" Choose a custom alias: hermes profile alias {name} --name ") + print( + f" Choose a custom alias: hermes profile alias {name} --name " + ) print(f" Or access via flag: hermes -p {name} chat") else: wrapper_path = create_wrapper_script(name) @@ -4324,7 +6410,9 @@ def cmd_profile(args): print(f"Wrapper created: {wrapper_path}") if not _is_wrapper_dir_in_path(): print(f"\n⚠ {_get_wrapper_dir()} is not in your PATH.") - print(f' Add to your shell config (~/.bashrc or ~/.zshrc):') + print( + f" Add to your shell config (~/.bashrc or ~/.zshrc):" + ) print(f' export PATH="$HOME/.local/bin:$PATH"') # Profile dir for display @@ -4342,7 +6430,9 @@ def cmd_profile(args): print(f"\n Edit {profile_dir_display}/.env for different API keys") print(f" Edit {profile_dir_display}/SOUL.md for different personality") else: - print(f"\n ⚠ This profile has no API keys yet. Run '{name} setup' first,") + print( + f"\n ⚠ This profile has no API keys yet. Run '{name} setup' first," + ) print(f" or it will inherit keys from your shell environment.") print(f" Edit {profile_dir_display}/SOUL.md to customize personality") print() @@ -4362,7 +6452,14 @@ def cmd_profile(args): elif action == "show": name = args.profile_name - from hermes_cli.profiles import get_profile_dir, profile_exists, _read_config_model, _check_gateway_running, _count_skills + from hermes_cli.profiles import ( + get_profile_dir, + profile_exists, + _read_config_model, + _check_gateway_running, + _count_skills, + ) + if not profile_exists(name): print(f"Error: Profile '{name}' does not exist.") sys.exit(1) @@ -4378,8 +6475,12 @@ def cmd_profile(args): print(f"Model: {model}" + (f" ({provider})" if provider else "")) print(f"Gateway: {'running' if gw else 'stopped'}") print(f"Skills: {skills}") - print(f".env: {'exists' if (profile_dir / '.env').exists() else 'not configured'}") - print(f"SOUL.md: {'exists' if (profile_dir / 'SOUL.md').exists() else 'not configured'}") + print( + f".env: {'exists' if (profile_dir / '.env').exists() else 'not configured'}" + ) + print( + f"SOUL.md: {'exists' if (profile_dir / 'SOUL.md').exists() else 'not configured'}" + ) if wrapper.exists(): print(f"Alias: {wrapper}") print() @@ -4390,6 +6491,7 @@ def cmd_profile(args): custom_name = getattr(args, "alias_name", None) from hermes_cli.profiles import profile_exists + if not profile_exists(name): print(f"Error: Profile '{name}' does not exist.") sys.exit(1) @@ -4417,6 +6519,7 @@ def cmd_profile(args): elif action == "rename": from hermes_cli.profiles import rename_profile + try: new_dir = rename_profile(args.old_name, args.new_name) print(f"\nProfile renamed: {args.old_name} → {args.new_name}") @@ -4427,6 +6530,7 @@ def cmd_profile(args): elif action == "export": from hermes_cli.profiles import export_profile + name = args.profile_name output = args.output or f"{name}.tar.gz" try: @@ -4438,8 +6542,11 @@ def cmd_profile(args): elif action == "import": from hermes_cli.profiles import import_profile + try: - profile_dir = import_profile(args.archive, name=getattr(args, "import_name", None)) + profile_dir = import_profile( + args.archive, name=getattr(args, "import_name", None) + ) name = profile_dir.name print(f"✓ Imported profile '{name}' at {profile_dir}") @@ -4462,28 +6569,34 @@ def cmd_dashboard(args): import uvicorn # noqa: F401 except ImportError: print("Web UI dependencies not installed.") - print("Install them with: pip install hermes-agent[web]") + print(f"Install them with: {sys.executable} -m pip install 'fastapi' 'uvicorn[standard]'") sys.exit(1) - if not _build_web_ui(PROJECT_ROOT / "web", fatal=True): - sys.exit(1) + if "HERMES_WEB_DIST" not in os.environ: + if not _build_web_ui(PROJECT_ROOT / "web", fatal=True): + sys.exit(1) from hermes_cli.web_server import start_server + start_server( host=args.host, port=args.port, open_browser=not args.no_open, + allow_public=getattr(args, "insecure", False), ) -def cmd_completion(args): +def cmd_completion(args, parser=None): """Print shell completion script.""" - from hermes_cli.profiles import generate_bash_completion, generate_zsh_completion + from hermes_cli.completion import generate_bash, generate_zsh, generate_fish + shell = getattr(args, "shell", "bash") if shell == "zsh": - print(generate_zsh_completion()) + print(generate_zsh(parser)) + elif shell == "fish": + print(generate_fish(parser)) else: - print(generate_bash_completion()) + print(generate_bash(parser)) def cmd_logs(args): @@ -4546,152 +6659,246 @@ def main(): For more help on a command: hermes --help -""" +""", ) - + parser.add_argument( - "--version", "-V", - action="store_true", - help="Show version and exit" + "--version", "-V", action="store_true", help="Show version and exit" ) parser.add_argument( - "--resume", "-r", + "--resume", + "-r", metavar="SESSION", default=None, - help="Resume a previous session by ID or title" + help="Resume a previous session by ID or title", ) parser.add_argument( - "--continue", "-c", + "--continue", + "-c", dest="continue_last", nargs="?", const=True, default=None, metavar="SESSION_NAME", - help="Resume a session by name, or the most recent if no name given" + help="Resume a session by name, or the most recent if no name given", + ) + parser.add_argument( + "--worktree", + "-w", + action="store_true", + default=False, + help="Run in an isolated git worktree (for parallel agents)", ) parser.add_argument( - "--worktree", "-w", + "--accept-hooks", action="store_true", default=False, - help="Run in an isolated git worktree (for parallel agents)" + help=( + "Auto-approve any unseen shell hooks declared in config.yaml " + "without a TTY prompt. Equivalent to HERMES_ACCEPT_HOOKS=1 or " + "hooks_auto_accept: true in config.yaml. Use on CI / headless " + "runs that can't prompt." + ), ) parser.add_argument( - "--skills", "-s", + "--skills", + "-s", action="append", default=None, - help="Preload one or more skills for the session (repeat flag or comma-separate)" + help="Preload one or more skills for the session (repeat flag or comma-separate)", ) parser.add_argument( "--yolo", action="store_true", default=False, - help="Bypass all dangerous command approval prompts (use at your own risk)" + help="Bypass all dangerous command approval prompts (use at your own risk)", ) parser.add_argument( "--pass-session-id", action="store_true", default=False, - help="Include the session ID in the agent's system prompt" + help="Include the session ID in the agent's system prompt", + ) + parser.add_argument( + "--ignore-user-config", + action="store_true", + default=False, + help="Ignore ~/.hermes/config.yaml and fall back to built-in defaults (credentials in .env are still loaded)", ) - + parser.add_argument( + "--ignore-rules", + action="store_true", + default=False, + help="Skip auto-injection of AGENTS.md, SOUL.md, .cursorrules, memory, and preloaded skills", + ) + parser.add_argument( + "--tui", + action="store_true", + default=False, + help="Launch the modern TUI instead of the classic REPL", + ) + parser.add_argument( + "--dev", + dest="tui_dev", + action="store_true", + default=False, + help="With --tui: run TypeScript sources via tsx (skip dist build)", + ) + subparsers = parser.add_subparsers(dest="command", help="Command to run") - + # ========================================================================= # chat command # ========================================================================= chat_parser = subparsers.add_parser( "chat", help="Interactive chat with the agent", - description="Start an interactive chat session with Hermes Agent" + description="Start an interactive chat session with Hermes Agent", ) chat_parser.add_argument( - "-q", "--query", - help="Single query (non-interactive mode)" + "-q", "--query", help="Single query (non-interactive mode)" ) chat_parser.add_argument( - "--image", - help="Optional local image path to attach to a single query" + "--image", help="Optional local image path to attach to a single query" ) chat_parser.add_argument( - "-m", "--model", - help="Model to use (e.g., anthropic/claude-sonnet-4)" + "-m", "--model", help="Model to use (e.g., anthropic/claude-sonnet-4)" ) chat_parser.add_argument( - "-t", "--toolsets", - help="Comma-separated toolsets to enable" + "-t", "--toolsets", help="Comma-separated toolsets to enable" ) chat_parser.add_argument( - "-s", "--skills", + "-s", + "--skills", action="append", default=argparse.SUPPRESS, - help="Preload one or more skills for the session (repeat flag or comma-separate)" + help="Preload one or more skills for the session (repeat flag or comma-separate)", ) chat_parser.add_argument( "--provider", - choices=["auto", "openrouter", "nous", "openai-codex", "copilot-acp", "copilot", "anthropic", "gemini", "huggingface", "zai", "kimi-coding", "kimi-coding-cn", "minimax", "minimax-cn", "kilocode", "xiaomi"], + choices=[ + "auto", + "openrouter", + "nous", + "openai-codex", + "copilot-acp", + "copilot", + "anthropic", + "gemini", + "xai", + "ollama-cloud", + "huggingface", + "zai", + "kimi-coding", + "kimi-coding-cn", + "stepfun", + "minimax", + "minimax-cn", + "kilocode", + "xiaomi", + "arcee", + "nvidia", + ], default=None, - help="Inference provider (default: auto)" + help="Inference provider (default: auto)", ) chat_parser.add_argument( - "-v", "--verbose", - action="store_true", - help="Verbose output" + "-v", "--verbose", action="store_true", help="Verbose output" ) chat_parser.add_argument( - "-Q", "--quiet", + "-Q", + "--quiet", action="store_true", - help="Quiet mode for programmatic use: suppress banner, spinner, and tool previews. Only output the final response and session info." + help="Quiet mode for programmatic use: suppress banner, spinner, and tool previews. Only output the final response and session info.", ) chat_parser.add_argument( - "--resume", "-r", + "--resume", + "-r", metavar="SESSION_ID", default=argparse.SUPPRESS, - help="Resume a previous session by ID (shown on exit)" + help="Resume a previous session by ID (shown on exit)", ) chat_parser.add_argument( - "--continue", "-c", + "--continue", + "-c", dest="continue_last", nargs="?", const=True, default=argparse.SUPPRESS, metavar="SESSION_NAME", - help="Resume a session by name, or the most recent if no name given" + help="Resume a session by name, or the most recent if no name given", + ) + chat_parser.add_argument( + "--worktree", + "-w", + action="store_true", + default=argparse.SUPPRESS, + help="Run in an isolated git worktree (for parallel agents on the same repo)", ) chat_parser.add_argument( - "--worktree", "-w", + "--accept-hooks", action="store_true", default=argparse.SUPPRESS, - help="Run in an isolated git worktree (for parallel agents on the same repo)" + help=( + "Auto-approve any unseen shell hooks declared in config.yaml " + "without a TTY prompt (see also HERMES_ACCEPT_HOOKS env var and " + "hooks_auto_accept: in config.yaml)." + ), ) chat_parser.add_argument( "--checkpoints", action="store_true", default=False, - help="Enable filesystem checkpoints before destructive file operations (use /rollback to restore)" + help="Enable filesystem checkpoints before destructive file operations (use /rollback to restore)", ) chat_parser.add_argument( "--max-turns", type=int, default=None, metavar="N", - help="Maximum tool-calling iterations per conversation turn (default: 90, or agent.max_turns in config)" + help="Maximum tool-calling iterations per conversation turn (default: 90, or agent.max_turns in config)", ) chat_parser.add_argument( "--yolo", action="store_true", default=argparse.SUPPRESS, - help="Bypass all dangerous command approval prompts (use at your own risk)" + help="Bypass all dangerous command approval prompts (use at your own risk)", ) chat_parser.add_argument( "--pass-session-id", action="store_true", default=argparse.SUPPRESS, - help="Include the session ID in the agent's system prompt" + help="Include the session ID in the agent's system prompt", + ) + chat_parser.add_argument( + "--ignore-user-config", + action="store_true", + default=argparse.SUPPRESS, + help="Ignore ~/.hermes/config.yaml and fall back to built-in defaults (credentials in .env are still loaded). Useful for isolated CI runs, reproduction, and third-party integrations.", + ) + chat_parser.add_argument( + "--ignore-rules", + action="store_true", + default=argparse.SUPPRESS, + help="Skip auto-injection of AGENTS.md, SOUL.md, .cursorrules, memory, and preloaded skills. Combine with --ignore-user-config for a fully isolated run.", ) chat_parser.add_argument( "--source", default=None, - help="Session source tag for filtering (default: cli). Use 'tool' for third-party integrations that should not appear in user session lists." + help="Session source tag for filtering (default: cli). Use 'tool' for third-party integrations that should not appear in user session lists.", + ) + chat_parser.add_argument( + "--tui", + action="store_true", + default=False, + help="Launch the modern TUI instead of the classic REPL", + ) + chat_parser.add_argument( + "--dev", + dest="tui_dev", + action="store_true", + default=False, + help="With --tui: run TypeScript sources via tsx (skip dist build)", ) chat_parser.set_defaults(func=cmd_chat) @@ -4701,45 +6908,42 @@ def main(): model_parser = subparsers.add_parser( "model", help="Select default model and provider", - description="Interactively select your inference provider and default model" + description="Interactively select your inference provider and default model", ) model_parser.add_argument( "--portal-url", - help="Portal base URL for Nous login (default: production portal)" + help="Portal base URL for Nous login (default: production portal)", ) model_parser.add_argument( "--inference-url", - help="Inference API base URL for Nous login (default: production inference API)" + help="Inference API base URL for Nous login (default: production inference API)", ) model_parser.add_argument( "--client-id", default=None, - help="OAuth client id to use for Nous login (default: hermes-cli)" + help="OAuth client id to use for Nous login (default: hermes-cli)", ) model_parser.add_argument( - "--scope", - default=None, - help="OAuth scope to request for Nous login" + "--scope", default=None, help="OAuth scope to request for Nous login" ) model_parser.add_argument( "--no-browser", action="store_true", - help="Do not attempt to open the browser automatically during Nous login" + help="Do not attempt to open the browser automatically during Nous login", ) model_parser.add_argument( "--timeout", type=float, default=15.0, - help="HTTP request timeout in seconds for Nous login (default: 15)" + help="HTTP request timeout in seconds for Nous login (default: 15)", ) model_parser.add_argument( - "--ca-bundle", - help="Path to CA bundle PEM file for Nous TLS verification" + "--ca-bundle", help="Path to CA bundle PEM file for Nous TLS verification" ) model_parser.add_argument( "--insecure", action="store_true", - help="Disable TLS verification for Nous login (testing only)" + help="Disable TLS verification for Nous login (testing only)", ) model_parser.set_defaults(func=cmd_model) @@ -4749,52 +6953,146 @@ def main(): gateway_parser = subparsers.add_parser( "gateway", help="Messaging gateway management", - description="Manage the messaging gateway (Telegram, Discord, WhatsApp)" + description="Manage the messaging gateway (Telegram, Discord, WhatsApp)", ) gateway_subparsers = gateway_parser.add_subparsers(dest="gateway_command") - + # gateway run (default) - gateway_run = gateway_subparsers.add_parser("run", help="Run gateway in foreground (recommended for WSL, Docker, Termux)") - gateway_run.add_argument("-v", "--verbose", action="count", default=0, - help="Increase stderr log verbosity (-v=INFO, -vv=DEBUG)") - gateway_run.add_argument("-q", "--quiet", action="store_true", - help="Suppress all stderr log output") - gateway_run.add_argument("--replace", action="store_true", - help="Replace any existing gateway instance (useful for systemd)") - + gateway_run = gateway_subparsers.add_parser( + "run", help="Run gateway in foreground (recommended for WSL, Docker, Termux)" + ) + gateway_run.add_argument( + "-v", + "--verbose", + action="count", + default=0, + help="Increase stderr log verbosity (-v=INFO, -vv=DEBUG)", + ) + gateway_run.add_argument( + "-q", "--quiet", action="store_true", help="Suppress all stderr log output" + ) + gateway_run.add_argument( + "--replace", + action="store_true", + help="Replace any existing gateway instance (useful for systemd)", + ) + _add_accept_hooks_flag(gateway_run) + _add_accept_hooks_flag(gateway_parser) + # gateway start - gateway_start = gateway_subparsers.add_parser("start", help="Start the installed systemd/launchd background service") - gateway_start.add_argument("--system", action="store_true", help="Target the Linux system-level gateway service") - + gateway_start = gateway_subparsers.add_parser( + "start", help="Start the installed systemd/launchd background service" + ) + gateway_start.add_argument( + "--system", + action="store_true", + help="Target the Linux system-level gateway service", + ) + gateway_start.add_argument( + "--all", + action="store_true", + help="Kill ALL stale gateway processes across all profiles before starting", + ) + # gateway stop gateway_stop = gateway_subparsers.add_parser("stop", help="Stop gateway service") - gateway_stop.add_argument("--system", action="store_true", help="Target the Linux system-level gateway service") - gateway_stop.add_argument("--all", action="store_true", help="Stop ALL gateway processes across all profiles") - + gateway_stop.add_argument( + "--system", + action="store_true", + help="Target the Linux system-level gateway service", + ) + gateway_stop.add_argument( + "--all", + action="store_true", + help="Stop ALL gateway processes across all profiles", + ) + # gateway restart - gateway_restart = gateway_subparsers.add_parser("restart", help="Restart gateway service") - gateway_restart.add_argument("--system", action="store_true", help="Target the Linux system-level gateway service") - + gateway_restart = gateway_subparsers.add_parser( + "restart", help="Restart gateway service" + ) + gateway_restart.add_argument( + "--system", + action="store_true", + help="Target the Linux system-level gateway service", + ) + gateway_restart.add_argument( + "--all", + action="store_true", + help="Kill ALL gateway processes across all profiles before restarting", + ) + # gateway status gateway_status = gateway_subparsers.add_parser("status", help="Show gateway status") gateway_status.add_argument("--deep", action="store_true", help="Deep status check") - gateway_status.add_argument("--system", action="store_true", help="Target the Linux system-level gateway service") - + gateway_status.add_argument( + "-l", + "--full", + action="store_true", + help="Show full, untruncated service/log output where supported", + ) + gateway_status.add_argument( + "--system", + action="store_true", + help="Target the Linux system-level gateway service", + ) + # gateway install - gateway_install = gateway_subparsers.add_parser("install", help="Install gateway as a systemd/launchd background service") + gateway_install = gateway_subparsers.add_parser( + "install", help="Install gateway as a systemd/launchd background service" + ) gateway_install.add_argument("--force", action="store_true", help="Force reinstall") - gateway_install.add_argument("--system", action="store_true", help="Install as a Linux system-level service (starts at boot)") - gateway_install.add_argument("--run-as-user", dest="run_as_user", help="User account the Linux system service should run as") - + gateway_install.add_argument( + "--system", + action="store_true", + help="Install as a Linux system-level service (starts at boot)", + ) + gateway_install.add_argument( + "--run-as-user", + dest="run_as_user", + help="User account the Linux system service should run as", + ) + # gateway uninstall - gateway_uninstall = gateway_subparsers.add_parser("uninstall", help="Uninstall gateway service") - gateway_uninstall.add_argument("--system", action="store_true", help="Target the Linux system-level gateway service") + gateway_uninstall = gateway_subparsers.add_parser( + "uninstall", help="Uninstall gateway service" + ) + gateway_uninstall.add_argument( + "--system", + action="store_true", + help="Target the Linux system-level gateway service", + ) # gateway setup gateway_subparsers.add_parser("setup", help="Configure messaging platforms") + # gateway migrate-legacy + gateway_migrate_legacy = gateway_subparsers.add_parser( + "migrate-legacy", + help="Remove legacy hermes.service units from pre-rename installs", + description=( + "Stop, disable, and remove legacy Hermes gateway unit files " + "(e.g. hermes.service) left over from older installs. Profile " + "units (hermes-gateway-.service) and unrelated " + "third-party services are never touched." + ), + ) + gateway_migrate_legacy.add_argument( + "--dry-run", + dest="dry_run", + action="store_true", + help="List what would be removed without doing it", + ) + gateway_migrate_legacy.add_argument( + "-y", + "--yes", + dest="yes", + action="store_true", + help="Skip the confirmation prompt", + ) + gateway_parser.set_defaults(func=cmd_gateway) - + # ========================================================================= # setup command # ========================================================================= @@ -4802,24 +7100,22 @@ def main(): "setup", help="Interactive setup wizard", description="Configure Hermes Agent with an interactive wizard. " - "Run a specific section: hermes setup model|tts|terminal|gateway|tools|agent" + "Run a specific section: hermes setup model|tts|terminal|gateway|tools|agent", ) setup_parser.add_argument( "section", nargs="?", choices=["model", "tts", "terminal", "gateway", "tools", "agent"], default=None, - help="Run a specific setup section instead of the full wizard" + help="Run a specific setup section instead of the full wizard", ) setup_parser.add_argument( "--non-interactive", action="store_true", - help="Non-interactive mode (use defaults/env vars)" + help="Non-interactive mode (use defaults/env vars)", ) setup_parser.add_argument( - "--reset", - action="store_true", - help="Reset configuration to defaults" + "--reset", action="store_true", help="Reset configuration to defaults" ) setup_parser.set_defaults(func=cmd_setup) @@ -4829,7 +7125,7 @@ def main(): whatsapp_parser = subparsers.add_parser( "whatsapp", help="Set up WhatsApp integration", - description="Configure WhatsApp and pair via QR code" + description="Configure WhatsApp and pair via QR code", ) whatsapp_parser.set_defaults(func=cmd_whatsapp) @@ -4839,51 +7135,43 @@ def main(): login_parser = subparsers.add_parser( "login", help="Authenticate with an inference provider", - description="Run OAuth device authorization flow for Hermes CLI" + description="Run OAuth device authorization flow for Hermes CLI", ) login_parser.add_argument( "--provider", choices=["nous", "openai-codex"], default=None, - help="Provider to authenticate with (default: nous)" + help="Provider to authenticate with (default: nous)", ) login_parser.add_argument( - "--portal-url", - help="Portal base URL (default: production portal)" + "--portal-url", help="Portal base URL (default: production portal)" ) login_parser.add_argument( "--inference-url", - help="Inference API base URL (default: production inference API)" - ) - login_parser.add_argument( - "--client-id", - default=None, - help="OAuth client id to use (default: hermes-cli)" + help="Inference API base URL (default: production inference API)", ) login_parser.add_argument( - "--scope", - default=None, - help="OAuth scope to request" + "--client-id", default=None, help="OAuth client id to use (default: hermes-cli)" ) + login_parser.add_argument("--scope", default=None, help="OAuth scope to request") login_parser.add_argument( "--no-browser", action="store_true", - help="Do not attempt to open the browser automatically" + help="Do not attempt to open the browser automatically", ) login_parser.add_argument( "--timeout", type=float, default=15.0, - help="HTTP request timeout in seconds (default: 15)" + help="HTTP request timeout in seconds (default: 15)", ) login_parser.add_argument( - "--ca-bundle", - help="Path to CA bundle PEM file for TLS verification" + "--ca-bundle", help="Path to CA bundle PEM file for TLS verification" ) login_parser.add_argument( "--insecure", action="store_true", - help="Disable TLS verification (testing only)" + help="Disable TLS verification (testing only)", ) login_parser.set_defaults(func=cmd_login) @@ -4893,13 +7181,13 @@ def main(): logout_parser = subparsers.add_parser( "logout", help="Clear authentication for an inference provider", - description="Remove stored credentials and reset provider config" + description="Remove stored credentials and reset provider config", ) logout_parser.add_argument( "--provider", choices=["nous", "openai-codex"], default=None, - help="Provider to log out from (default: active provider)" + help="Provider to log out from (default: active provider)", ) logout_parser.set_defaults(func=cmd_logout) @@ -4909,24 +7197,50 @@ def main(): ) auth_subparsers = auth_parser.add_subparsers(dest="auth_action") auth_add = auth_subparsers.add_parser("add", help="Add a pooled credential") - auth_add.add_argument("provider", help="Provider id (for example: anthropic, openai-codex, openrouter)") - auth_add.add_argument("--type", dest="auth_type", choices=["oauth", "api-key", "api_key"], help="Credential type to add") + auth_add.add_argument( + "provider", + help="Provider id (for example: anthropic, openai-codex, openrouter)", + ) + auth_add.add_argument( + "--type", + dest="auth_type", + choices=["oauth", "api-key", "api_key"], + help="Credential type to add", + ) auth_add.add_argument("--label", help="Optional display label") - auth_add.add_argument("--api-key", help="API key value (otherwise prompted securely)") + auth_add.add_argument( + "--api-key", help="API key value (otherwise prompted securely)" + ) auth_add.add_argument("--portal-url", help="Nous portal base URL") auth_add.add_argument("--inference-url", help="Nous inference base URL") auth_add.add_argument("--client-id", help="OAuth client id") auth_add.add_argument("--scope", help="OAuth scope override") - auth_add.add_argument("--no-browser", action="store_true", help="Do not auto-open a browser for OAuth login") - auth_add.add_argument("--timeout", type=float, help="OAuth/network timeout in seconds") - auth_add.add_argument("--insecure", action="store_true", help="Disable TLS verification for OAuth login") + auth_add.add_argument( + "--no-browser", + action="store_true", + help="Do not auto-open a browser for OAuth login", + ) + auth_add.add_argument( + "--timeout", type=float, help="OAuth/network timeout in seconds" + ) + auth_add.add_argument( + "--insecure", + action="store_true", + help="Disable TLS verification for OAuth login", + ) auth_add.add_argument("--ca-bundle", help="Custom CA bundle for OAuth login") auth_list = auth_subparsers.add_parser("list", help="List pooled credentials") auth_list.add_argument("provider", nargs="?", help="Optional provider filter") - auth_remove = auth_subparsers.add_parser("remove", help="Remove a pooled credential by index, id, or label") + auth_remove = auth_subparsers.add_parser( + "remove", help="Remove a pooled credential by index, id, or label" + ) auth_remove.add_argument("provider", help="Provider id") - auth_remove.add_argument("target", help="Credential index, entry id, or exact label") - auth_reset = auth_subparsers.add_parser("reset", help="Clear exhaustion status for all credentials for a provider") + auth_remove.add_argument( + "target", help="Credential index, entry id, or exact label" + ) + auth_reset = auth_subparsers.add_parser( + "reset", help="Clear exhaustion status for all credentials for a provider" + ) auth_reset.add_argument("provider", help="Provider id") auth_parser.set_defaults(func=cmd_auth) @@ -4936,57 +7250,92 @@ def main(): status_parser = subparsers.add_parser( "status", help="Show status of all components", - description="Display status of Hermes Agent components" + description="Display status of Hermes Agent components", ) status_parser.add_argument( - "--all", - action="store_true", - help="Show all details (redacted for sharing)" + "--all", action="store_true", help="Show all details (redacted for sharing)" ) status_parser.add_argument( - "--deep", - action="store_true", - help="Run deep checks (may take longer)" + "--deep", action="store_true", help="Run deep checks (may take longer)" ) status_parser.set_defaults(func=cmd_status) - + # ========================================================================= # cron command # ========================================================================= cron_parser = subparsers.add_parser( - "cron", - help="Cron job management", - description="Manage scheduled tasks" + "cron", help="Cron job management", description="Manage scheduled tasks" ) cron_subparsers = cron_parser.add_subparsers(dest="cron_command") - + # cron list cron_list = cron_subparsers.add_parser("list", help="List scheduled jobs") cron_list.add_argument("--all", action="store_true", help="Include disabled jobs") # cron create/add - cron_create = cron_subparsers.add_parser("create", aliases=["add"], help="Create a scheduled job") - cron_create.add_argument("schedule", help="Schedule like '30m', 'every 2h', or '0 9 * * *'") - cron_create.add_argument("prompt", nargs="?", help="Optional self-contained prompt or task instruction") + cron_create = cron_subparsers.add_parser( + "create", aliases=["add"], help="Create a scheduled job" + ) + cron_create.add_argument( + "schedule", help="Schedule like '30m', 'every 2h', or '0 9 * * *'" + ) + cron_create.add_argument( + "prompt", nargs="?", help="Optional self-contained prompt or task instruction" + ) cron_create.add_argument("--name", help="Optional human-friendly job name") - cron_create.add_argument("--deliver", help="Delivery target: origin, local, telegram, discord, signal, or platform:chat_id") + cron_create.add_argument( + "--deliver", + help="Delivery target: origin, local, telegram, discord, signal, or platform:chat_id", + ) cron_create.add_argument("--repeat", type=int, help="Optional repeat count") - cron_create.add_argument("--skill", dest="skills", action="append", help="Attach a skill. Repeat to add multiple skills.") - cron_create.add_argument("--script", help="Path to a Python script whose stdout is injected into the prompt each run") + cron_create.add_argument( + "--skill", + dest="skills", + action="append", + help="Attach a skill. Repeat to add multiple skills.", + ) + cron_create.add_argument( + "--script", + help="Path to a Python script whose stdout is injected into the prompt each run", + ) # cron edit - cron_edit = cron_subparsers.add_parser("edit", help="Edit an existing scheduled job") + cron_edit = cron_subparsers.add_parser( + "edit", help="Edit an existing scheduled job" + ) cron_edit.add_argument("job_id", help="Job ID to edit") cron_edit.add_argument("--schedule", help="New schedule") cron_edit.add_argument("--prompt", help="New prompt/task instruction") cron_edit.add_argument("--name", help="New job name") cron_edit.add_argument("--deliver", help="New delivery target") cron_edit.add_argument("--repeat", type=int, help="New repeat count") - cron_edit.add_argument("--skill", dest="skills", action="append", help="Replace the job's skills with this set. Repeat to attach multiple skills.") - cron_edit.add_argument("--add-skill", dest="add_skills", action="append", help="Append a skill without replacing the existing list. Repeatable.") - cron_edit.add_argument("--remove-skill", dest="remove_skills", action="append", help="Remove a specific attached skill. Repeatable.") - cron_edit.add_argument("--clear-skills", action="store_true", help="Remove all attached skills from the job") - cron_edit.add_argument("--script", help="Path to a Python script whose stdout is injected into the prompt each run. Pass empty string to clear.") + cron_edit.add_argument( + "--skill", + dest="skills", + action="append", + help="Replace the job's skills with this set. Repeat to attach multiple skills.", + ) + cron_edit.add_argument( + "--add-skill", + dest="add_skills", + action="append", + help="Append a skill without replacing the existing list. Repeatable.", + ) + cron_edit.add_argument( + "--remove-skill", + dest="remove_skills", + action="append", + help="Remove a specific attached skill. Repeatable.", + ) + cron_edit.add_argument( + "--clear-skills", + action="store_true", + help="Remove all attached skills from the job", + ) + cron_edit.add_argument( + "--script", + help="Path to a Python script whose stdout is injected into the prompt each run. Pass empty string to clear.", + ) # lifecycle actions cron_pause = cron_subparsers.add_parser("pause", help="Pause a scheduled job") @@ -4995,18 +7344,24 @@ def main(): cron_resume = cron_subparsers.add_parser("resume", help="Resume a paused job") cron_resume.add_argument("job_id", help="Job ID to resume") - cron_run = cron_subparsers.add_parser("run", help="Run a job on the next scheduler tick") + cron_run = cron_subparsers.add_parser( + "run", help="Run a job on the next scheduler tick" + ) cron_run.add_argument("job_id", help="Job ID to trigger") + _add_accept_hooks_flag(cron_run) - cron_remove = cron_subparsers.add_parser("remove", aliases=["rm", "delete"], help="Remove a scheduled job") + cron_remove = cron_subparsers.add_parser( + "remove", aliases=["rm", "delete"], help="Remove a scheduled job" + ) cron_remove.add_argument("job_id", help="Job ID to remove") # cron status cron_subparsers.add_parser("status", help="Check if cron scheduler is running") # cron tick (mostly for debugging) - cron_subparsers.add_parser("tick", help="Run due jobs once and exit") - + cron_tick = cron_subparsers.add_parser("tick", help="Run due jobs once and exit") + _add_accept_hooks_flag(cron_tick) + _add_accept_hooks_flag(cron_parser) cron_parser.set_defaults(func=cmd_cron) # ========================================================================= @@ -5019,39 +7374,131 @@ def main(): ) webhook_subparsers = webhook_parser.add_subparsers(dest="webhook_action") - wh_sub = webhook_subparsers.add_parser("subscribe", aliases=["add"], help="Create a webhook subscription") + wh_sub = webhook_subparsers.add_parser( + "subscribe", aliases=["add"], help="Create a webhook subscription" + ) wh_sub.add_argument("name", help="Route name (used in URL: /webhooks/)") - wh_sub.add_argument("--prompt", default="", help="Prompt template with {dot.notation} payload refs") - wh_sub.add_argument("--events", default="", help="Comma-separated event types to accept") + wh_sub.add_argument( + "--prompt", default="", help="Prompt template with {dot.notation} payload refs" + ) + wh_sub.add_argument( + "--events", default="", help="Comma-separated event types to accept" + ) wh_sub.add_argument("--description", default="", help="What this subscription does") - wh_sub.add_argument("--skills", default="", help="Comma-separated skill names to load") - wh_sub.add_argument("--deliver", default="log", help="Delivery target: log, telegram, discord, slack, etc.") - wh_sub.add_argument("--deliver-chat-id", default="", help="Target chat ID for cross-platform delivery") - wh_sub.add_argument("--secret", default="", help="HMAC secret (auto-generated if omitted)") + wh_sub.add_argument( + "--skills", default="", help="Comma-separated skill names to load" + ) + wh_sub.add_argument( + "--deliver", + default="log", + help="Delivery target: log, telegram, discord, slack, etc.", + ) + wh_sub.add_argument( + "--deliver-chat-id", + default="", + help="Target chat ID for cross-platform delivery", + ) + wh_sub.add_argument( + "--secret", default="", help="HMAC secret (auto-generated if omitted)" + ) + wh_sub.add_argument( + "--deliver-only", + action="store_true", + help="Skip the agent — deliver the rendered prompt directly as the " + "message. Zero LLM cost. Requires --deliver to be a real target " + "(not 'log').", + ) - webhook_subparsers.add_parser("list", aliases=["ls"], help="List all dynamic subscriptions") + webhook_subparsers.add_parser( + "list", aliases=["ls"], help="List all dynamic subscriptions" + ) - wh_rm = webhook_subparsers.add_parser("remove", aliases=["rm"], help="Remove a subscription") + wh_rm = webhook_subparsers.add_parser( + "remove", aliases=["rm"], help="Remove a subscription" + ) wh_rm.add_argument("name", help="Subscription name to remove") - wh_test = webhook_subparsers.add_parser("test", help="Send a test POST to a webhook route") + wh_test = webhook_subparsers.add_parser( + "test", help="Send a test POST to a webhook route" + ) wh_test.add_argument("name", help="Subscription name to test") - wh_test.add_argument("--payload", default="", help="JSON payload to send (default: test payload)") + wh_test.add_argument( + "--payload", default="", help="JSON payload to send (default: test payload)" + ) webhook_parser.set_defaults(func=cmd_webhook) + # ========================================================================= + # hooks command — shell-hook inspection and management + # ========================================================================= + hooks_parser = subparsers.add_parser( + "hooks", + help="Inspect and manage shell-script hooks", + description=( + "Inspect shell-script hooks declared in ~/.hermes/config.yaml, " + "test them against synthetic payloads, and manage the first-use " + "consent allowlist at ~/.hermes/shell-hooks-allowlist.json." + ), + ) + hooks_subparsers = hooks_parser.add_subparsers(dest="hooks_action") + + hooks_subparsers.add_parser( + "list", aliases=["ls"], + help="List configured hooks with matcher, timeout, and consent status", + ) + + _hk_test = hooks_subparsers.add_parser( + "test", + help="Fire every hook matching against a synthetic payload", + ) + _hk_test.add_argument( + "event", + help="Hook event name (e.g. pre_tool_call, pre_llm_call, subagent_stop)", + ) + _hk_test.add_argument( + "--for-tool", dest="for_tool", default=None, + help=( + "Only fire hooks whose matcher matches this tool name " + "(used for pre_tool_call / post_tool_call)" + ), + ) + _hk_test.add_argument( + "--payload-file", dest="payload_file", default=None, + help=( + "Path to a JSON file whose contents are merged into the " + "synthetic payload before execution" + ), + ) + + _hk_revoke = hooks_subparsers.add_parser( + "revoke", aliases=["remove", "rm"], + help="Remove a command's allowlist entries (takes effect on next restart)", + ) + _hk_revoke.add_argument( + "command", + help="The exact command string to revoke (as declared in config.yaml)", + ) + + hooks_subparsers.add_parser( + "doctor", + help=( + "Check each configured hook: exec bit, allowlist, mtime drift, " + "JSON validity, and synthetic run timing" + ), + ) + + hooks_parser.set_defaults(func=cmd_hooks) + # ========================================================================= # doctor command # ========================================================================= doctor_parser = subparsers.add_parser( "doctor", help="Check configuration and dependencies", - description="Diagnose issues with Hermes Agent setup" + description="Diagnose issues with Hermes Agent setup", ) doctor_parser.add_argument( - "--fix", - action="store_true", - help="Attempt to fix issues automatically" + "--fix", action="store_true", help="Attempt to fix issues automatically" ) doctor_parser.set_defaults(func=cmd_doctor) @@ -5062,12 +7509,12 @@ def main(): "dump", help="Dump setup summary for support/debugging", description="Output a compact, plain-text summary of your Hermes setup " - "that can be copy-pasted into Discord/GitHub for support context" + "that can be copy-pasted into Discord/GitHub for support context", ) dump_parser.add_argument( "--show-keys", action="store_true", - help="Show redacted API key prefixes (first/last 4 chars) instead of just set/not set" + help="Show redacted API key prefixes (first/last 4 chars) instead of just set/not set", ) dump_parser.set_defaults(func=cmd_dump) @@ -5078,8 +7525,8 @@ def main(): "debug", help="Debug tools — upload logs and system info for support", description="Debug utilities for Hermes Agent. Use 'hermes debug share' to " - "upload a debug report (system info + recent logs) to a paste " - "service and get a shareable URL.", + "upload a debug report (system info + recent logs) to a paste " + "service and get a shareable URL.", formatter_class=argparse.RawDescriptionHelpFormatter, epilog="""\ Examples: @@ -5087,6 +7534,7 @@ def main(): hermes debug share --lines 500 Include more log lines hermes debug share --expire 30 Keep paste for 30 days hermes debug share --local Print report locally (no upload) + hermes debug delete Delete a previously uploaded paste """, ) debug_sub = debug_parser.add_subparsers(dest="debug_command") @@ -5095,17 +7543,32 @@ def main(): help="Upload debug report to a paste service and print a shareable URL", ) share_parser.add_argument( - "--lines", type=int, default=200, + "--lines", + type=int, + default=200, help="Number of log lines to include per log file (default: 200)", ) share_parser.add_argument( - "--expire", type=int, default=7, + "--expire", + type=int, + default=7, help="Paste expiry in days (default: 7)", ) share_parser.add_argument( - "--local", action="store_true", + "--local", + action="store_true", help="Print the report locally instead of uploading", ) + delete_parser = debug_sub.add_parser( + "delete", + help="Delete a paste uploaded by 'hermes debug share'", + ) + delete_parser.add_argument( + "urls", + nargs="*", + default=[], + help="One or more paste URLs to delete (e.g. https://paste.rs/abc123)", + ) debug_parser.set_defaults(func=cmd_debug) # ========================================================================= @@ -5115,21 +7578,22 @@ def main(): "backup", help="Back up Hermes home directory to a zip file", description="Create a zip archive of your entire Hermes configuration, " - "skills, sessions, and data (excludes the hermes-agent codebase). " - "Use --quick for a fast snapshot of just critical state files." + "skills, sessions, and data (excludes the hermes-agent codebase). " + "Use --quick for a fast snapshot of just critical state files.", ) backup_parser.add_argument( - "-o", "--output", - help="Output path for the zip file (default: ~/hermes-backup-.zip)" + "-o", + "--output", + help="Output path for the zip file (default: ~/hermes-backup-.zip)", ) backup_parser.add_argument( - "-q", "--quick", + "-q", + "--quick", action="store_true", - help="Quick snapshot: only critical state files (config, state.db, .env, auth, cron)" + help="Quick snapshot: only critical state files (config, state.db, .env, auth, cron)", ) backup_parser.add_argument( - "-l", "--label", - help="Label for the snapshot (only used with --quick)" + "-l", "--label", help="Label for the snapshot (only used with --quick)" ) backup_parser.set_defaults(func=cmd_backup) @@ -5140,17 +7604,15 @@ def main(): "import", help="Restore a Hermes backup from a zip file", description="Extract a previously created Hermes backup into your " - "Hermes home directory, restoring configuration, skills, " - "sessions, and data" + "Hermes home directory, restoring configuration, skills, " + "sessions, and data", ) + import_parser.add_argument("zipfile", help="Path to the backup zip file") import_parser.add_argument( - "zipfile", - help="Path to the backup zip file" - ) - import_parser.add_argument( - "--force", "-f", + "--force", + "-f", action="store_true", - help="Overwrite existing files without confirmation" + help="Overwrite existing files without confirmation", ) import_parser.set_defaults(func=cmd_import) @@ -5160,49 +7622,55 @@ def main(): config_parser = subparsers.add_parser( "config", help="View and edit configuration", - description="Manage Hermes Agent configuration" + description="Manage Hermes Agent configuration", ) config_subparsers = config_parser.add_subparsers(dest="config_command") - + # config show (default) config_subparsers.add_parser("show", help="Show current configuration") - + # config edit config_subparsers.add_parser("edit", help="Open config file in editor") - + # config set config_set = config_subparsers.add_parser("set", help="Set a configuration value") - config_set.add_argument("key", nargs="?", help="Configuration key (e.g., model, terminal.backend)") + config_set.add_argument( + "key", nargs="?", help="Configuration key (e.g., model, terminal.backend)" + ) config_set.add_argument("value", nargs="?", help="Value to set") - + # config path config_subparsers.add_parser("path", help="Print config file path") - + # config env-path config_subparsers.add_parser("env-path", help="Print .env file path") - + # config check config_subparsers.add_parser("check", help="Check for missing/outdated config") - + # config migrate config_subparsers.add_parser("migrate", help="Update config with new options") - + config_parser.set_defaults(func=cmd_config) - + # ========================================================================= # pairing command # ========================================================================= pairing_parser = subparsers.add_parser( "pairing", help="Manage DM pairing codes for user authorization", - description="Approve or revoke user access via pairing codes" + description="Approve or revoke user access via pairing codes", ) pairing_sub = pairing_parser.add_subparsers(dest="pairing_action") pairing_sub.add_parser("list", help="Show pending + approved users") - pairing_approve_parser = pairing_sub.add_parser("approve", help="Approve a pairing code") - pairing_approve_parser.add_argument("platform", help="Platform name (telegram, discord, slack, whatsapp)") + pairing_approve_parser = pairing_sub.add_parser( + "approve", help="Approve a pairing code" + ) + pairing_approve_parser.add_argument( + "platform", help="Platform name (telegram, discord, slack, whatsapp)" + ) pairing_approve_parser.add_argument("code", help="Pairing code to approve") pairing_revoke_parser = pairing_sub.add_parser("revoke", help="Revoke user access") @@ -5213,6 +7681,7 @@ def main(): def cmd_pairing(args): from hermes_cli.pairing import pairing_command + pairing_command(args) pairing_parser.set_defaults(func=cmd_pairing) @@ -5223,58 +7692,158 @@ def cmd_pairing(args): skills_parser = subparsers.add_parser( "skills", help="Search, install, configure, and manage skills", - description="Search, install, inspect, audit, configure, and manage skills from skills.sh, well-known agent skill endpoints, GitHub, ClawHub, and other registries." + description="Search, install, inspect, audit, configure, and manage skills from skills.sh, well-known agent skill endpoints, GitHub, ClawHub, and other registries.", ) skills_subparsers = skills_parser.add_subparsers(dest="skills_action") - skills_browse = skills_subparsers.add_parser("browse", help="Browse all available skills (paginated)") - skills_browse.add_argument("--page", type=int, default=1, help="Page number (default: 1)") - skills_browse.add_argument("--size", type=int, default=20, help="Results per page (default: 20)") - skills_browse.add_argument("--source", default="all", - choices=["all", "official", "skills-sh", "well-known", "github", "clawhub", "lobehub"], - help="Filter by source (default: all)") - - skills_search = skills_subparsers.add_parser("search", help="Search skill registries") + skills_browse = skills_subparsers.add_parser( + "browse", help="Browse all available skills (paginated)" + ) + skills_browse.add_argument( + "--page", type=int, default=1, help="Page number (default: 1)" + ) + skills_browse.add_argument( + "--size", type=int, default=20, help="Results per page (default: 20)" + ) + skills_browse.add_argument( + "--source", + default="all", + choices=[ + "all", + "official", + "skills-sh", + "well-known", + "github", + "clawhub", + "lobehub", + ], + help="Filter by source (default: all)", + ) + + skills_search = skills_subparsers.add_parser( + "search", help="Search skill registries" + ) skills_search.add_argument("query", help="Search query") - skills_search.add_argument("--source", default="all", choices=["all", "official", "skills-sh", "well-known", "github", "clawhub", "lobehub"]) + skills_search.add_argument( + "--source", + default="all", + choices=[ + "all", + "official", + "skills-sh", + "well-known", + "github", + "clawhub", + "lobehub", + ], + ) skills_search.add_argument("--limit", type=int, default=10, help="Max results") skills_install = skills_subparsers.add_parser("install", help="Install a skill") - skills_install.add_argument("identifier", help="Skill identifier (e.g. openai/skills/skill-creator)") - skills_install.add_argument("--category", default="", help="Category folder to install into") - skills_install.add_argument("--force", action="store_true", help="Install despite blocked scan verdict") - skills_install.add_argument("--yes", "-y", action="store_true", help="Skip confirmation prompt (needed in TUI mode)") + skills_install.add_argument( + "identifier", help="Skill identifier (e.g. openai/skills/skill-creator)" + ) + skills_install.add_argument( + "--category", default="", help="Category folder to install into" + ) + skills_install.add_argument( + "--force", action="store_true", help="Install despite blocked scan verdict" + ) + skills_install.add_argument( + "--yes", + "-y", + action="store_true", + help="Skip confirmation prompt (needed in TUI mode)", + ) - skills_inspect = skills_subparsers.add_parser("inspect", help="Preview a skill without installing") + skills_inspect = skills_subparsers.add_parser( + "inspect", help="Preview a skill without installing" + ) skills_inspect.add_argument("identifier", help="Skill identifier") skills_list = skills_subparsers.add_parser("list", help="List installed skills") - skills_list.add_argument("--source", default="all", choices=["all", "hub", "builtin", "local"]) + skills_list.add_argument( + "--source", default="all", choices=["all", "hub", "builtin", "local"] + ) - skills_check = skills_subparsers.add_parser("check", help="Check installed hub skills for updates") - skills_check.add_argument("name", nargs="?", help="Specific skill to check (default: all)") + skills_check = skills_subparsers.add_parser( + "check", help="Check installed hub skills for updates" + ) + skills_check.add_argument( + "name", nargs="?", help="Specific skill to check (default: all)" + ) - skills_update = skills_subparsers.add_parser("update", help="Update installed hub skills") - skills_update.add_argument("name", nargs="?", help="Specific skill to update (default: all outdated skills)") + skills_update = skills_subparsers.add_parser( + "update", help="Update installed hub skills" + ) + skills_update.add_argument( + "name", + nargs="?", + help="Specific skill to update (default: all outdated skills)", + ) - skills_audit = skills_subparsers.add_parser("audit", help="Re-scan installed hub skills") - skills_audit.add_argument("name", nargs="?", help="Specific skill to audit (default: all)") + skills_audit = skills_subparsers.add_parser( + "audit", help="Re-scan installed hub skills" + ) + skills_audit.add_argument( + "name", nargs="?", help="Specific skill to audit (default: all)" + ) - skills_uninstall = skills_subparsers.add_parser("uninstall", help="Remove a hub-installed skill") + skills_uninstall = skills_subparsers.add_parser( + "uninstall", help="Remove a hub-installed skill" + ) skills_uninstall.add_argument("name", help="Skill name to remove") - skills_publish = skills_subparsers.add_parser("publish", help="Publish a skill to a registry") + skills_reset = skills_subparsers.add_parser( + "reset", + help="Reset a bundled skill — clears 'user-modified' tracking so updates work again", + description=( + "Clear a bundled skill's entry from the sync manifest (~/.hermes/skills/.bundled_manifest) " + "so future 'hermes update' runs stop marking it as user-modified. Pass --restore to also " + "replace the current copy with the bundled version." + ), + ) + skills_reset.add_argument( + "name", help="Skill name to reset (e.g. google-workspace)" + ) + skills_reset.add_argument( + "--restore", + action="store_true", + help="Also delete the current copy and re-copy the bundled version", + ) + skills_reset.add_argument( + "--yes", + "-y", + action="store_true", + help="Skip confirmation prompt when using --restore", + ) + + skills_publish = skills_subparsers.add_parser( + "publish", help="Publish a skill to a registry" + ) skills_publish.add_argument("skill_path", help="Path to skill directory") - skills_publish.add_argument("--to", default="github", choices=["github", "clawhub"], help="Target registry") - skills_publish.add_argument("--repo", default="", help="Target GitHub repo (e.g. openai/skills)") + skills_publish.add_argument( + "--to", default="github", choices=["github", "clawhub"], help="Target registry" + ) + skills_publish.add_argument( + "--repo", default="", help="Target GitHub repo (e.g. openai/skills)" + ) - skills_snapshot = skills_subparsers.add_parser("snapshot", help="Export/import skill configurations") + skills_snapshot = skills_subparsers.add_parser( + "snapshot", help="Export/import skill configurations" + ) snapshot_subparsers = skills_snapshot.add_subparsers(dest="snapshot_action") - snap_export = snapshot_subparsers.add_parser("export", help="Export installed skills to a file") + snap_export = snapshot_subparsers.add_parser( + "export", help="Export installed skills to a file" + ) snap_export.add_argument("output", help="Output JSON file path (use - for stdout)") - snap_import = snapshot_subparsers.add_parser("import", help="Import and install skills from a file") + snap_import = snapshot_subparsers.add_parser( + "import", help="Import and install skills from a file" + ) snap_import.add_argument("input", help="Input JSON file path") - snap_import.add_argument("--force", action="store_true", help="Force install despite caution verdict") + snap_import.add_argument( + "--force", action="store_true", help="Force install despite caution verdict" + ) skills_tap = skills_subparsers.add_parser("tap", help="Manage skill sources") tap_subparsers = skills_tap.add_subparsers(dest="tap_action") @@ -5285,16 +7854,21 @@ def cmd_pairing(args): tap_rm.add_argument("name", help="Tap name to remove") # config sub-action: interactive enable/disable - skills_subparsers.add_parser("config", help="Interactive skill configuration — enable/disable individual skills") + skills_subparsers.add_parser( + "config", + help="Interactive skill configuration — enable/disable individual skills", + ) def cmd_skills(args): # Route 'config' action to skills_config module - if getattr(args, 'skills_action', None) == 'config': + if getattr(args, "skills_action", None) == "config": _require_tty("skills config") from hermes_cli.skills_config import skills_command as skills_config_command + skills_config_command(args) else: from hermes_cli.skills_hub import skills_command + skills_command(args) skills_parser.set_defaults(func=cmd_skills) @@ -5317,9 +7891,22 @@ def cmd_skills(args): help="Git URL or owner/repo shorthand (e.g. anpicasso/hermes-plugin-chrome-profiles)", ) plugins_install.add_argument( - "--force", "-f", action="store_true", + "--force", + "-f", + action="store_true", help="Remove existing plugin and reinstall", ) + _install_enable_group = plugins_install.add_mutually_exclusive_group() + _install_enable_group.add_argument( + "--enable", + action="store_true", + help="Auto-enable the plugin after install (skip confirmation prompt)", + ) + _install_enable_group.add_argument( + "--no-enable", + action="store_true", + help="Install disabled (skip confirmation prompt); enable later with `hermes plugins enable `", + ) plugins_update = plugins_subparsers.add_parser( "update", help="Pull latest changes for an installed plugin" @@ -5345,6 +7932,7 @@ def cmd_skills(args): def cmd_plugins(args): from hermes_cli.plugins_cmd import plugins_command + plugins_command(args) plugins_parser.set_defaults(func=cmd_plugins) @@ -5356,6 +7944,7 @@ def cmd_plugins(args): # ========================================================================= try: from plugins.memory import discover_plugin_cli_commands + for cmd_info in discover_plugin_cli_commands(): plugin_parser = subparsers.add_parser( cmd_info["name"], @@ -5365,8 +7954,7 @@ def cmd_plugins(args): ) cmd_info["setup_fn"](plugin_parser) except Exception as _exc: - import logging as _log - _log.getLogger(__name__).debug("Plugin CLI discovery failed: %s", _exc) + logging.getLogger(__name__).debug("Plugin CLI discovery failed: %s", _exc) # ========================================================================= # memory command @@ -5383,14 +7971,33 @@ def cmd_plugins(args): ), ) memory_sub = memory_parser.add_subparsers(dest="memory_command") - memory_sub.add_parser("setup", help="Interactive provider selection and configuration") + memory_sub.add_parser( + "setup", help="Interactive provider selection and configuration" + ) memory_sub.add_parser("status", help="Show current memory provider config") memory_sub.add_parser("off", help="Disable external provider (built-in only)") + _reset_parser = memory_sub.add_parser( + "reset", + help="Erase all built-in memory (MEMORY.md and USER.md)", + ) + _reset_parser.add_argument( + "--yes", + "-y", + action="store_true", + help="Skip confirmation prompt", + ) + _reset_parser.add_argument( + "--target", + choices=["all", "memory", "user"], + default="all", + help="Which store to reset: 'all' (default), 'memory', or 'user'", + ) def cmd_memory(args): sub = getattr(args, "memory_command", None) if sub == "off": from hermes_cli.config import load_config, save_config + config = load_config() if not isinstance(config.get("memory"), dict): config["memory"] = {} @@ -5398,8 +8005,54 @@ def cmd_memory(args): save_config(config) print("\n ✓ Memory provider: built-in only") print(" Saved to config.yaml\n") + elif sub == "reset": + from hermes_constants import get_hermes_home, display_hermes_home + + mem_dir = get_hermes_home() / "memories" + target = getattr(args, "target", "all") + files_to_reset = [] + if target in ("all", "memory"): + files_to_reset.append(("MEMORY.md", "agent notes")) + if target in ("all", "user"): + files_to_reset.append(("USER.md", "user profile")) + + # Check what exists + existing = [ + (f, desc) for f, desc in files_to_reset if (mem_dir / f).exists() + ] + if not existing: + print( + f"\n Nothing to reset — no memory files found in {display_hermes_home()}/memories/\n" + ) + return + + print(f"\n This will permanently erase the following memory files:") + for f, desc in existing: + path = mem_dir / f + size = path.stat().st_size + print(f" ◆ {f} ({desc}) — {size:,} bytes") + + if not getattr(args, "yes", False): + try: + answer = input("\n Type 'yes' to confirm: ").strip().lower() + except (EOFError, KeyboardInterrupt): + print("\n Cancelled.\n") + return + if answer != "yes": + print(" Cancelled.\n") + return + + for f, desc in existing: + (mem_dir / f).unlink() + print(f" ✓ Deleted {f} ({desc})") + + print( + f"\n Memory reset complete. New sessions will start with a blank slate." + ) + print(f" Files were in: {display_hermes_home()}/memories/\n") else: from hermes_cli.memory_setup import memory_command + memory_command(args) memory_parser.set_defaults(func=cmd_memory) @@ -5420,7 +8073,7 @@ def cmd_memory(args): tools_parser.add_argument( "--summary", action="store_true", - help="Print a summary of enabled tools per platform and exit" + help="Print a summary of enabled tools per platform and exit", ) tools_sub = tools_parser.add_subparsers(dest="tools_action") @@ -5430,7 +8083,8 @@ def cmd_memory(args): help="Show all tools and their enabled/disabled status", ) tools_list_p.add_argument( - "--platform", default="cli", + "--platform", + default="cli", help="Platform to show (default: cli)", ) @@ -5440,11 +8094,14 @@ def cmd_memory(args): help="Disable toolsets or MCP tools", ) tools_disable_p.add_argument( - "names", nargs="+", metavar="NAME", + "names", + nargs="+", + metavar="NAME", help="Toolset name (e.g. web) or MCP tool in server:tool form", ) tools_disable_p.add_argument( - "--platform", default="cli", + "--platform", + default="cli", help="Platform to apply to (default: cli)", ) @@ -5454,11 +8111,14 @@ def cmd_memory(args): help="Enable toolsets or MCP tools", ) tools_enable_p.add_argument( - "names", nargs="+", metavar="NAME", + "names", + nargs="+", + metavar="NAME", help="Toolset name or MCP tool in server:tool form", ) tools_enable_p.add_argument( - "--platform", default="cli", + "--platform", + default="cli", help="Platform to apply to (default: cli)", ) @@ -5466,10 +8126,12 @@ def cmd_tools(args): action = getattr(args, "tools_action", None) if action in ("list", "disable", "enable"): from hermes_cli.tools_config import tools_disable_enable_command + tools_disable_enable_command(args) else: _require_tty("tools") from hermes_cli.tools_config import tools_command + tools_command(args) tools_parser.set_defaults(func=cmd_tools) @@ -5493,18 +8155,30 @@ def cmd_tools(args): help="Run Hermes as an MCP server (expose conversations to other agents)", ) mcp_serve_p.add_argument( - "-v", "--verbose", action="store_true", + "-v", + "--verbose", + action="store_true", help="Enable verbose logging on stderr", ) + _add_accept_hooks_flag(mcp_serve_p) - mcp_add_p = mcp_sub.add_parser("add", help="Add an MCP server (discovery-first install)") + mcp_add_p = mcp_sub.add_parser( + "add", help="Add an MCP server (discovery-first install)" + ) mcp_add_p.add_argument("name", help="Server name (used as config key)") mcp_add_p.add_argument("--url", help="HTTP/SSE endpoint URL") mcp_add_p.add_argument("--command", help="Stdio command (e.g. npx)") - mcp_add_p.add_argument("--args", nargs="*", default=[], help="Arguments for stdio command") + mcp_add_p.add_argument( + "--args", nargs="*", default=[], help="Arguments for stdio command" + ) mcp_add_p.add_argument("--auth", choices=["oauth", "header"], help="Auth method") mcp_add_p.add_argument("--preset", help="Known MCP preset name") - mcp_add_p.add_argument("--env", nargs="*", default=[], help="Environment variables for stdio servers (KEY=VALUE)") + mcp_add_p.add_argument( + "--env", + nargs="*", + default=[], + help="Environment variables for stdio servers (KEY=VALUE)", + ) mcp_rm_p = mcp_sub.add_parser("remove", aliases=["rm"], help="Remove an MCP server") mcp_rm_p.add_argument("name", help="Server name to remove") @@ -5514,11 +8188,22 @@ def cmd_tools(args): mcp_test_p = mcp_sub.add_parser("test", help="Test MCP server connection") mcp_test_p.add_argument("name", help="Server name to test") - mcp_cfg_p = mcp_sub.add_parser("configure", aliases=["config"], help="Toggle tool selection") + mcp_cfg_p = mcp_sub.add_parser( + "configure", aliases=["config"], help="Toggle tool selection" + ) mcp_cfg_p.add_argument("name", help="Server name to configure") + mcp_login_p = mcp_sub.add_parser( + "login", + help="Force re-authentication for an OAuth-based MCP server", + ) + mcp_login_p.add_argument("name", help="Server name to re-authenticate") + + _add_accept_hooks_flag(mcp_parser) + def cmd_mcp(args): from hermes_cli.mcp_config import mcp_command + mcp_command(args) mcp_parser.set_defaults(func=cmd_mcp) @@ -5529,31 +8214,52 @@ def cmd_mcp(args): sessions_parser = subparsers.add_parser( "sessions", help="Manage session history (list, rename, export, prune, delete)", - description="View and manage the SQLite session store" + description="View and manage the SQLite session store", ) sessions_subparsers = sessions_parser.add_subparsers(dest="sessions_action") sessions_list = sessions_subparsers.add_parser("list", help="List recent sessions") - sessions_list.add_argument("--source", help="Filter by source (cli, telegram, discord, etc.)") - sessions_list.add_argument("--limit", type=int, default=20, help="Max sessions to show") + sessions_list.add_argument( + "--source", help="Filter by source (cli, telegram, discord, etc.)" + ) + sessions_list.add_argument( + "--limit", type=int, default=20, help="Max sessions to show" + ) - sessions_export = sessions_subparsers.add_parser("export", help="Export sessions to a JSONL file") - sessions_export.add_argument("output", help="Output JSONL file path (use - for stdout)") + sessions_export = sessions_subparsers.add_parser( + "export", help="Export sessions to a JSONL file" + ) + sessions_export.add_argument( + "output", help="Output JSONL file path (use - for stdout)" + ) sessions_export.add_argument("--source", help="Filter by source") sessions_export.add_argument("--session-id", help="Export a specific session") - sessions_delete = sessions_subparsers.add_parser("delete", help="Delete a specific session") + sessions_delete = sessions_subparsers.add_parser( + "delete", help="Delete a specific session" + ) sessions_delete.add_argument("session_id", help="Session ID to delete") - sessions_delete.add_argument("--yes", "-y", action="store_true", help="Skip confirmation") + sessions_delete.add_argument( + "--yes", "-y", action="store_true", help="Skip confirmation" + ) sessions_prune = sessions_subparsers.add_parser("prune", help="Delete old sessions") - sessions_prune.add_argument("--older-than", type=int, default=90, help="Delete sessions older than N days (default: 90)") + sessions_prune.add_argument( + "--older-than", + type=int, + default=90, + help="Delete sessions older than N days (default: 90)", + ) sessions_prune.add_argument("--source", help="Only prune sessions from this source") - sessions_prune.add_argument("--yes", "-y", action="store_true", help="Skip confirmation") + sessions_prune.add_argument( + "--yes", "-y", action="store_true", help="Skip confirmation" + ) sessions_subparsers.add_parser("stats", help="Show session store statistics") - sessions_rename = sessions_subparsers.add_parser("rename", help="Set or change a session's title") + sessions_rename = sessions_subparsers.add_parser( + "rename", help="Set or change a session's title" + ) sessions_rename.add_argument("session_id", help="Session ID to rename") sessions_rename.add_argument("title", nargs="+", help="New title for the session") @@ -5561,8 +8267,12 @@ def cmd_mcp(args): "browse", help="Interactive session picker — browse, search, and resume sessions", ) - sessions_browse.add_argument("--source", help="Filter by source (cli, telegram, discord, etc.)") - sessions_browse.add_argument("--limit", type=int, default=50, help="Max sessions to load (default: 50)") + sessions_browse.add_argument( + "--source", help="Filter by source (cli, telegram, discord, etc.)" + ) + sessions_browse.add_argument( + "--limit", type=int, default=50, help="Max sessions to load (default: 50)" + ) def _confirm_prompt(prompt: str) -> bool: """Prompt for y/N confirmation, safe against non-TTY environments.""" @@ -5573,8 +8283,10 @@ def _confirm_prompt(prompt: str) -> bool: def cmd_sessions(args): import json as _json + try: from hermes_state import SessionDB + db = SessionDB() except Exception as e: print(f"Error: Could not open session database: {e}") @@ -5587,7 +8299,9 @@ def cmd_sessions(args): _exclude = None if _source else ["tool"] if action == "list": - sessions = db.list_sessions_rich(source=args.source, exclude_sources=_exclude, limit=args.limit) + sessions = db.list_sessions_rich( + source=args.source, exclude_sources=_exclude, limit=args.limit + ) if not sessions: print("No sessions found.") return @@ -5600,7 +8314,11 @@ def cmd_sessions(args): print("─" * 95) for s in sessions: last_active = _relative_time(s.get("last_active")) - preview = s.get("preview", "")[:38] if has_titles else s.get("preview", "")[:48] + preview = ( + s.get("preview", "")[:38] + if has_titles + else s.get("preview", "")[:48] + ) if has_titles: title = (s.get("title") or "—")[:30] sid = s["id"] @@ -5621,7 +8339,7 @@ def cmd_sessions(args): return line = _json.dumps(data, ensure_ascii=False) + "\n" if args.output == "-": - import sys + sys.stdout.write(line) else: with open(args.output, "w", encoding="utf-8") as f: @@ -5630,7 +8348,7 @@ def cmd_sessions(args): else: sessions = db.export_all(source=args.source) if args.output == "-": - import sys + for s in sessions: sys.stdout.write(_json.dumps(s, ensure_ascii=False) + "\n") else: @@ -5645,7 +8363,9 @@ def cmd_sessions(args): print(f"Session '{args.session_id}' not found.") return if not args.yes: - if not _confirm_prompt(f"Delete session '{resolved_session_id}' and all its messages? [y/N] "): + if not _confirm_prompt( + f"Delete session '{resolved_session_id}' and all its messages? [y/N] " + ): print("Cancelled.") return if db.delete_session(resolved_session_id): @@ -5657,7 +8377,9 @@ def cmd_sessions(args): days = args.older_than source_msg = f" from '{args.source}'" if args.source else "" if not args.yes: - if not _confirm_prompt(f"Delete all ended sessions older than {days} days{source_msg}? [y/N] "): + if not _confirm_prompt( + f"Delete all ended sessions older than {days} days{source_msg}? [y/N] " + ): print("Cancelled.") return count = db.prune_sessions(older_than_days=days, source=args.source) @@ -5681,7 +8403,9 @@ def cmd_sessions(args): limit = getattr(args, "limit", 50) or 50 source = getattr(args, "source", None) _browse_exclude = None if source else ["tool"] - sessions = db.list_sessions_rich(source=source, exclude_sources=_browse_exclude, limit=limit) + sessions = db.list_sessions_rich( + source=source, exclude_sources=_browse_exclude, limit=limit + ) db.close() if not sessions: print("No sessions found.") @@ -5694,7 +8418,6 @@ def cmd_sessions(args): # Launch hermes --resume by replacing the current process print(f"Resuming session: {selected_id}") - import shutil hermes_bin = shutil.which("hermes") if hermes_bin: os.execvp(hermes_bin, ["hermes", "--resume", selected_id]) @@ -5733,10 +8456,14 @@ def cmd_sessions(args): insights_parser = subparsers.add_parser( "insights", help="Show usage insights and analytics", - description="Analyze session history to show token usage, costs, tool patterns, and activity trends" + description="Analyze session history to show token usage, costs, tool patterns, and activity trends", + ) + insights_parser.add_argument( + "--days", type=int, default=30, help="Number of days to analyze (default: 30)" + ) + insights_parser.add_argument( + "--source", help="Filter by platform (cli, telegram, discord, etc.)" ) - insights_parser.add_argument("--days", type=int, default=30, help="Number of days to analyze (default: 30)") - insights_parser.add_argument("--source", help="Filter by platform (cli, telegram, discord, etc.)") def cmd_insights(args): try: @@ -5759,7 +8486,7 @@ def cmd_insights(args): claw_parser = subparsers.add_parser( "claw", help="OpenClaw migration tools", - description="Migrate settings, memories, skills, and API keys from OpenClaw to Hermes" + description="Migrate settings, memories, skills, and API keys from OpenClaw to Hermes", ) claw_subparsers = claw_parser.add_subparsers(dest="claw_action") @@ -5768,47 +8495,43 @@ def cmd_insights(args): "migrate", help="Migrate from OpenClaw to Hermes", description="Import settings, memories, skills, and API keys from an OpenClaw installation. " - "Always shows a preview before making changes." + "Always shows a preview before making changes.", ) claw_migrate.add_argument( - "--source", - help="Path to OpenClaw directory (default: ~/.openclaw)" + "--source", help="Path to OpenClaw directory (default: ~/.openclaw)" ) claw_migrate.add_argument( "--dry-run", action="store_true", - help="Preview only — stop after showing what would be migrated" + help="Preview only — stop after showing what would be migrated", ) claw_migrate.add_argument( "--preset", choices=["user-data", "full"], default="full", - help="Migration preset (default: full). 'user-data' excludes secrets" + help="Migration preset (default: full). 'user-data' excludes secrets", ) claw_migrate.add_argument( "--overwrite", action="store_true", - help="Overwrite existing files (default: skip conflicts)" + help="Overwrite existing files (default: skip conflicts)", ) claw_migrate.add_argument( "--migrate-secrets", action="store_true", - help="Include allowlisted secrets (TELEGRAM_BOT_TOKEN, API keys, etc.)" + help="Include allowlisted secrets (TELEGRAM_BOT_TOKEN, API keys, etc.)", ) claw_migrate.add_argument( - "--workspace-target", - help="Absolute path to copy workspace instructions into" + "--workspace-target", help="Absolute path to copy workspace instructions into" ) claw_migrate.add_argument( "--skill-conflict", choices=["skip", "overwrite", "rename"], default="skip", - help="How to handle skill name conflicts (default: skip)" + help="How to handle skill name conflicts (default: skip)", ) claw_migrate.add_argument( - "--yes", "-y", - action="store_true", - help="Skip confirmation prompts" + "--yes", "-y", action="store_true", help="Skip confirmation prompts" ) # claw cleanup @@ -5816,25 +8539,23 @@ def cmd_insights(args): "cleanup", aliases=["clean"], help="Archive leftover OpenClaw directories after migration", - description="Scan for and archive leftover OpenClaw directories to prevent state fragmentation" + description="Scan for and archive leftover OpenClaw directories to prevent state fragmentation", ) claw_cleanup.add_argument( - "--source", - help="Path to a specific OpenClaw directory to clean up" + "--source", help="Path to a specific OpenClaw directory to clean up" ) claw_cleanup.add_argument( "--dry-run", action="store_true", - help="Preview what would be archived without making changes" + help="Preview what would be archived without making changes", ) claw_cleanup.add_argument( - "--yes", "-y", - action="store_true", - help="Skip confirmation prompts" + "--yes", "-y", action="store_true", help="Skip confirmation prompts" ) def cmd_claw(args): from hermes_cli.claw import claw_command + claw_command(args) claw_parser.set_defaults(func=cmd_claw) @@ -5842,43 +8563,40 @@ def cmd_claw(args): # ========================================================================= # version command # ========================================================================= - version_parser = subparsers.add_parser( - "version", - help="Show version information" - ) + version_parser = subparsers.add_parser("version", help="Show version information") version_parser.set_defaults(func=cmd_version) - + # ========================================================================= # update command # ========================================================================= update_parser = subparsers.add_parser( "update", help="Update Hermes Agent to the latest version", - description="Pull the latest changes from git and reinstall dependencies" + description="Pull the latest changes from git and reinstall dependencies", ) update_parser.add_argument( - "--gateway", action="store_true", default=False, - help="Gateway mode: use file-based IPC for prompts instead of stdin (used internally by /update)" + "--gateway", + action="store_true", + default=False, + help="Gateway mode: use file-based IPC for prompts instead of stdin (used internally by /update)", ) update_parser.set_defaults(func=cmd_update) - + # ========================================================================= # uninstall command # ========================================================================= uninstall_parser = subparsers.add_parser( "uninstall", help="Uninstall Hermes Agent", - description="Remove Hermes Agent from your system. Can keep configs/data for reinstall." + description="Remove Hermes Agent from your system. Can keep configs/data for reinstall.", ) uninstall_parser.add_argument( "--full", action="store_true", - help="Full uninstall - remove everything including configs and data" + help="Full uninstall - remove everything including configs and data", ) uninstall_parser.add_argument( - "--yes", "-y", - action="store_true", - help="Skip confirmation prompts" + "--yes", "-y", action="store_true", help="Skip confirmation prompts" ) uninstall_parser.set_defaults(func=cmd_uninstall) @@ -5890,11 +8608,13 @@ def cmd_claw(args): help="Run Hermes Agent as an ACP (Agent Client Protocol) server", description="Start Hermes Agent in ACP mode for editor integration (VS Code, Zed, JetBrains)", ) + _add_accept_hooks_flag(acp_parser) def cmd_acp(args): """Launch Hermes Agent as an ACP server.""" try: from acp_adapter.entry import main as acp_main + acp_main() except ImportError: print("ACP dependencies not installed.") @@ -5913,48 +8633,81 @@ def cmd_acp(args): profile_subparsers = profile_parser.add_subparsers(dest="profile_action") profile_subparsers.add_parser("list", help="List all profiles") - profile_use = profile_subparsers.add_parser("use", help="Set sticky default profile") + profile_use = profile_subparsers.add_parser( + "use", help="Set sticky default profile" + ) profile_use.add_argument("profile_name", help="Profile name (or 'default')") - profile_create = profile_subparsers.add_parser("create", help="Create a new profile") - profile_create.add_argument("profile_name", help="Profile name (lowercase, alphanumeric)") - profile_create.add_argument("--clone", action="store_true", - help="Copy config.yaml, .env, SOUL.md from active profile") - profile_create.add_argument("--clone-all", action="store_true", - help="Full copy of active profile (all state)") - profile_create.add_argument("--clone-from", metavar="SOURCE", - help="Source profile to clone from (default: active)") - profile_create.add_argument("--no-alias", action="store_true", - help="Skip wrapper script creation") + profile_create = profile_subparsers.add_parser( + "create", help="Create a new profile" + ) + profile_create.add_argument( + "profile_name", help="Profile name (lowercase, alphanumeric)" + ) + profile_create.add_argument( + "--clone", + action="store_true", + help="Copy config.yaml, .env, SOUL.md from active profile", + ) + profile_create.add_argument( + "--clone-all", + action="store_true", + help="Full copy of active profile (all state)", + ) + profile_create.add_argument( + "--clone-from", + metavar="SOURCE", + help="Source profile to clone from (default: active)", + ) + profile_create.add_argument( + "--no-alias", action="store_true", help="Skip wrapper script creation" + ) profile_delete = profile_subparsers.add_parser("delete", help="Delete a profile") profile_delete.add_argument("profile_name", help="Profile to delete") - profile_delete.add_argument("-y", "--yes", action="store_true", - help="Skip confirmation prompt") + profile_delete.add_argument( + "-y", "--yes", action="store_true", help="Skip confirmation prompt" + ) profile_show = profile_subparsers.add_parser("show", help="Show profile details") profile_show.add_argument("profile_name", help="Profile to show") - profile_alias = profile_subparsers.add_parser("alias", help="Manage wrapper scripts") + profile_alias = profile_subparsers.add_parser( + "alias", help="Manage wrapper scripts" + ) profile_alias.add_argument("profile_name", help="Profile name") - profile_alias.add_argument("--remove", action="store_true", - help="Remove the wrapper script") - profile_alias.add_argument("--name", dest="alias_name", metavar="NAME", - help="Custom alias name (default: profile name)") + profile_alias.add_argument( + "--remove", action="store_true", help="Remove the wrapper script" + ) + profile_alias.add_argument( + "--name", + dest="alias_name", + metavar="NAME", + help="Custom alias name (default: profile name)", + ) profile_rename = profile_subparsers.add_parser("rename", help="Rename a profile") profile_rename.add_argument("old_name", help="Current profile name") profile_rename.add_argument("new_name", help="New profile name") - profile_export = profile_subparsers.add_parser("export", help="Export a profile to archive") + profile_export = profile_subparsers.add_parser( + "export", help="Export a profile to archive" + ) profile_export.add_argument("profile_name", help="Profile to export") - profile_export.add_argument("-o", "--output", default=None, - help="Output file (default: .tar.gz)") + profile_export.add_argument( + "-o", "--output", default=None, help="Output file (default: .tar.gz)" + ) - profile_import = profile_subparsers.add_parser("import", help="Import a profile from archive") + profile_import = profile_subparsers.add_parser( + "import", help="Import a profile from archive" + ) profile_import.add_argument("archive", help="Path to .tar.gz archive") - profile_import.add_argument("--name", dest="import_name", metavar="NAME", - help="Profile name (default: inferred from archive)") + profile_import.add_argument( + "--name", + dest="import_name", + metavar="NAME", + help="Profile name (default: inferred from archive)", + ) profile_parser.set_defaults(func=cmd_profile) @@ -5963,13 +8716,16 @@ def cmd_acp(args): # ========================================================================= completion_parser = subparsers.add_parser( "completion", - help="Print shell completion script (bash or zsh)", + help="Print shell completion script (bash, zsh, or fish)", ) completion_parser.add_argument( - "shell", nargs="?", default="bash", choices=["bash", "zsh"], + "shell", + nargs="?", + default="bash", + choices=["bash", "zsh", "fish"], help="Shell type (default: bash)", ) - completion_parser.set_defaults(func=cmd_completion) + completion_parser.set_defaults(func=lambda args: cmd_completion(args, parser)) # ========================================================================= # dashboard command @@ -5979,9 +8735,20 @@ def cmd_acp(args): help="Start the web UI dashboard", description="Launch the Hermes Agent web dashboard for managing config, API keys, and sessions", ) - dashboard_parser.add_argument("--port", type=int, default=9119, help="Port (default 9119)") - dashboard_parser.add_argument("--host", default="127.0.0.1", help="Host (default 127.0.0.1)") - dashboard_parser.add_argument("--no-open", action="store_true", help="Don't open browser automatically") + dashboard_parser.add_argument( + "--port", type=int, default=9119, help="Port (default 9119)" + ) + dashboard_parser.add_argument( + "--host", default="127.0.0.1", help="Host (default 127.0.0.1)" + ) + dashboard_parser.add_argument( + "--no-open", action="store_true", help="Don't open browser automatically" + ) + dashboard_parser.add_argument( + "--insecure", + action="store_true", + help="Allow binding to non-localhost (DANGEROUS: exposes API keys on the network)", + ) dashboard_parser.set_defaults(func=cmd_dashboard) # ========================================================================= @@ -6007,31 +8774,42 @@ def cmd_acp(args): """, ) logs_parser.add_argument( - "log_name", nargs="?", default="agent", + "log_name", + nargs="?", + default="agent", help="Log to view: agent (default), errors, gateway, or 'list' to show available files", ) logs_parser.add_argument( - "-n", "--lines", type=int, default=50, + "-n", + "--lines", + type=int, + default=50, help="Number of lines to show (default: 50)", ) logs_parser.add_argument( - "-f", "--follow", action="store_true", + "-f", + "--follow", + action="store_true", help="Follow the log in real time (like tail -f)", ) logs_parser.add_argument( - "--level", metavar="LEVEL", + "--level", + metavar="LEVEL", help="Minimum log level to show (DEBUG, INFO, WARNING, ERROR)", ) logs_parser.add_argument( - "--session", metavar="ID", + "--session", + metavar="ID", help="Filter lines containing this session ID substring", ) logs_parser.add_argument( - "--since", metavar="TIME", + "--since", + metavar="TIME", help="Show lines since TIME ago (e.g. 1h, 30m, 2d)", ) logs_parser.add_argument( - "--component", metavar="NAME", + "--component", + metavar="NAME", help="Filter by component: gateway, agent, tools, cli, cron", ) logs_parser.set_defaults(func=cmd_logs) @@ -6048,6 +8826,7 @@ def cmd_acp(args): # --help, unrecognised flags, and every subcommand are forwarded # transparently instead of being intercepted by argparse on the host. from hermes_cli.config import get_container_exec_info + container_info = get_container_exec_info() if container_info: _exec_in_container(container_info, sys.argv[1:]) @@ -6056,42 +8835,124 @@ def cmd_acp(args): sys.exit(1) _processed_argv = _coalesce_session_name_args(sys.argv[1:]) - args = parser.parse_args(_processed_argv) + + # ── Defensive subparser routing (bpo-9338 workaround) ─────────── + # On some Python versions (notably <3.11), argparse fails to route + # subcommand tokens when the parent parser has nargs='?' optional + # arguments (--continue). The symptom: "unrecognized arguments: model" + # even though 'model' is a registered subcommand. + # + # Fix: when argv contains a token matching a known subcommand, set + # subparsers.required=True to force deterministic routing. If that + # fails (e.g. 'hermes -c model' where 'model' is consumed as the + # session name for --continue), fall back to the default behaviour. + import io as _io + + _known_cmds = ( + set(subparsers.choices.keys()) if hasattr(subparsers, "choices") else set() + ) + _has_cmd_token = any( + t in _known_cmds for t in _processed_argv if not t.startswith("-") + ) + + if _has_cmd_token: + subparsers.required = True + _saved_stderr = sys.stderr + try: + sys.stderr = _io.StringIO() + args = parser.parse_args(_processed_argv) + sys.stderr = _saved_stderr + except SystemExit as exc: + sys.stderr = _saved_stderr + # Help/version flags (exit code 0) already printed output — + # re-raise immediately to avoid a second parse_args printing + # the same help text again (#10230). + if exc.code == 0: + raise + # Subcommand name was consumed as a flag value (e.g. -c model). + # Fall back to optional subparsers so argparse handles it normally. + subparsers.required = False + args = parser.parse_args(_processed_argv) + else: + subparsers.required = False + args = parser.parse_args(_processed_argv) # Handle --version flag if args.version: cmd_version(args) return - + + # Discover Python plugins and register shell hooks once, before any + # command that can fire lifecycle hooks. Both are idempotent; gated + # so introspection/management commands (hermes hooks list, cron + # list, gateway status, mcp add, ...) don't pay discovery cost or + # trigger consent prompts for hooks the user is still inspecting. + # Groups with mixed admin/CRUD vs. agent-running entries narrow via + # the nested subcommand (dest varies by parser). + _AGENT_COMMANDS = {None, "chat", "acp", "rl"} + _AGENT_SUBCOMMANDS = { + "cron": ("cron_command", {"run", "tick"}), + "gateway": ("gateway_command", {"run"}), + "mcp": ("mcp_action", {"serve"}), + } + _sub_attr, _sub_set = _AGENT_SUBCOMMANDS.get(args.command, (None, None)) + if ( + args.command in _AGENT_COMMANDS + or (_sub_attr and getattr(args, _sub_attr, None) in _sub_set) + ): + _accept_hooks = bool(getattr(args, "accept_hooks", False)) + try: + from hermes_cli.plugins import discover_plugins + discover_plugins() + except Exception: + logger.debug( + "plugin discovery failed at CLI startup", exc_info=True, + ) + try: + from hermes_cli.config import load_config + from agent.shell_hooks import register_from_config + register_from_config(load_config(), accept_hooks=_accept_hooks) + except Exception: + logger.debug( + "shell-hook registration failed at CLI startup", + exc_info=True, + ) + # Handle top-level --resume / --continue as shortcut to chat if (args.resume or args.continue_last) and args.command is None: args.command = "chat" - args.query = None - args.model = None - args.provider = None - args.toolsets = None - args.verbose = False - if not hasattr(args, "worktree"): - args.worktree = False + for attr, default in [ + ("query", None), + ("model", None), + ("provider", None), + ("toolsets", None), + ("verbose", False), + ("worktree", False), + ]: + if not hasattr(args, attr): + setattr(args, attr, default) cmd_chat(args) return - + # Default to chat if no command specified if args.command is None: - args.query = None - args.model = None - args.provider = None - args.toolsets = None - args.verbose = False - args.resume = None - args.continue_last = None - if not hasattr(args, "worktree"): - args.worktree = False + for attr, default in [ + ("query", None), + ("model", None), + ("provider", None), + ("toolsets", None), + ("verbose", False), + ("resume", None), + ("continue_last", None), + ("worktree", False), + ]: + if not hasattr(args, attr): + setattr(args, attr, default) cmd_chat(args) return - + # Execute the command - if hasattr(args, 'func'): + if hasattr(args, "func"): args.func(args) else: parser.print_help() diff --git a/hermes_cli/mcp_config.py b/hermes_cli/mcp_config.py index b21234ce0a62..ae845b069bac 100644 --- a/hermes_cli/mcp_config.py +++ b/hermes_cli/mcp_config.py @@ -279,8 +279,8 @@ def cmd_mcp_add(args): _info(f"Starting OAuth flow for '{name}'...") oauth_ok = False try: - from tools.mcp_oauth import build_oauth_auth - oauth_auth = build_oauth_auth(name, url) + from tools.mcp_oauth_manager import get_manager + oauth_auth = get_manager().get_or_build_provider(name, url, None) if oauth_auth: server_config["auth"] = "oauth" _success("OAuth configured (tokens will be acquired on first connection)") @@ -428,10 +428,12 @@ def cmd_mcp_remove(args): _remove_mcp_server(name) _success(f"Removed '{name}' from config") - # Clean up OAuth tokens if they exist + # Clean up OAuth tokens if they exist — route through MCPOAuthManager so + # any provider instance cached in the current process (e.g. from an + # earlier `hermes mcp test` in the same session) is evicted too. try: - from tools.mcp_oauth import remove_oauth_tokens - remove_oauth_tokens(name) + from tools.mcp_oauth_manager import get_manager + get_manager().remove(name) _success("Cleaned up OAuth tokens") except Exception: pass @@ -577,6 +579,63 @@ def _replace(m): return re.sub(r"\$\{(\w+)\}", _replace, value) +# ─── hermes mcp login ──────────────────────────────────────────────────────── + +def cmd_mcp_login(args): + """Force re-authentication for an OAuth-based MCP server. + + Deletes cached tokens (both on disk and in the running process's + MCPOAuthManager cache) and triggers a fresh OAuth flow via the + existing probe path. + + Use this when: + - Tokens are stuck in a bad state (server revoked, refresh token + consumed by an external process, etc.) + - You want to re-authenticate to change scopes or account + - A tool call returned ``needs_reauth: true`` + """ + name = args.name + servers = _get_mcp_servers() + + if name not in servers: + _error(f"Server '{name}' not found in config.") + if servers: + _info(f"Available servers: {', '.join(servers)}") + return + + server_config = servers[name] + url = server_config.get("url") + if not url: + _error(f"Server '{name}' has no URL — not an OAuth-capable server") + return + if server_config.get("auth") != "oauth": + _error(f"Server '{name}' is not configured for OAuth (auth={server_config.get('auth')})") + _info("Use `hermes mcp remove` + `hermes mcp add` to reconfigure auth.") + return + + # Wipe both disk and in-memory cache so the next probe forces a fresh + # OAuth flow. + try: + from tools.mcp_oauth_manager import get_manager + mgr = get_manager() + mgr.remove(name) + except Exception as exc: + _warning(f"Could not clear existing OAuth state: {exc}") + + print() + _info(f"Starting OAuth flow for '{name}'...") + + # Probe triggers the OAuth flow (browser redirect + callback capture). + try: + tools = _probe_single_server(name, server_config) + if tools: + _success(f"Authenticated — {len(tools)} tool(s) available") + else: + _success("Authenticated (server reported no tools)") + except Exception as exc: + _error(f"Authentication failed: {exc}") + + # ─── hermes mcp configure ──────────────────────────────────────────────────── def cmd_mcp_configure(args): @@ -696,6 +755,7 @@ def mcp_command(args): "test": cmd_mcp_test, "configure": cmd_mcp_configure, "config": cmd_mcp_configure, + "login": cmd_mcp_login, } handler = handlers.get(action) @@ -713,4 +773,5 @@ def mcp_command(args): _info("hermes mcp list List servers") _info("hermes mcp test Test connection") _info("hermes mcp configure Toggle tools") + _info("hermes mcp login Re-authenticate OAuth") print() diff --git a/hermes_cli/memory_setup.py b/hermes_cli/memory_setup.py index 1aa43136765b..88186b8ec662 100644 --- a/hermes_cli/memory_setup.py +++ b/hermes_cli/memory_setup.py @@ -58,9 +58,11 @@ def _prompt(label: str, default: str | None = None, secret: bool = False) -> str def _install_dependencies(provider_name: str) -> None: """Install pip dependencies declared in plugin.yaml.""" import subprocess - from pathlib import Path as _Path + from plugins.memory import find_provider_dir - plugin_dir = _Path(__file__).parent.parent / "plugins" / "memory" / provider_name + plugin_dir = find_provider_dir(provider_name) + if not plugin_dir: + return yaml_path = plugin_dir / "plugin.yaml" if not yaml_path.exists(): return @@ -324,6 +326,9 @@ def cmd_setup(args) -> None: val = _prompt(desc, default=str(effective_default) if effective_default else None) if val: provider_config[key] = val + # Also write to .env if this field has an env_var + if env_var and env_var not in env_writes: + env_writes[env_var] = val # Write activation key to config.yaml config["memory"]["provider"] = name @@ -409,12 +414,13 @@ def cmd_status(args) -> None: else: print(f" Status: not available ✗") schema = p.get_config_schema() if hasattr(p, "get_config_schema") else [] - secrets = [f for f in schema if f.get("secret")] - if secrets: + # Check all fields that have env_var (both secret and non-secret) + required_fields = [f for f in schema if f.get("env_var")] + if required_fields: print(f" Missing:") - for s in secrets: - env_var = s.get("env_var", "") - url = s.get("url", "") + for f in required_fields: + env_var = f.get("env_var", "") + url = f.get("url", "") is_set = bool(os.environ.get(env_var)) mark = "✓" if is_set else "✗" line = f" {mark} {env_var}" diff --git a/hermes_cli/model_normalize.py b/hermes_cli/model_normalize.py index c391b0715c98..76dace065a3a 100644 --- a/hermes_cli/model_normalize.py +++ b/hermes_cli/model_normalize.py @@ -51,6 +51,7 @@ "grok": "x-ai", "qwen": "qwen", "mimo": "xiaomi", + "trinity": "arcee-ai", "nemotron": "nvidia", "llama": "meta-llama", "step": "stepfun", @@ -94,6 +95,8 @@ "alibaba", "qwen-oauth", "xiaomi", + "arcee", + "ollama-cloud", "custom", }) @@ -371,7 +374,26 @@ def normalize_model_for_provider(model_input: str, target_provider: str) -> str: return bare return _dots_to_hyphens(bare) - # --- Copilot: strip matching provider prefix, keep dots --- + # --- Copilot / Copilot ACP: delegate to the Copilot-specific + # normalizer. It knows about the alias table (vendor-prefix + # stripping for Anthropic/OpenAI, dash-to-dot repair for Claude) + # and live-catalog lookups. Without this, vendor-prefixed or + # dash-notation Claude IDs survive to the Copilot API and hit + # HTTP 400 "model_not_supported". See issue #6879. + if provider in {"copilot", "copilot-acp"}: + try: + from hermes_cli.models import normalize_copilot_model_id + + normalized = normalize_copilot_model_id(name) + if normalized: + return normalized + except Exception: + # Fall through to the generic strip-vendor behaviour below + # if the Copilot-specific path is unavailable for any reason. + pass + + # --- Copilot / Copilot ACP / openai-codex fallback: + # strip matching provider prefix, keep dots --- if provider in _STRIP_VENDOR_ONLY_PROVIDERS: stripped = _strip_matching_provider_prefix(name, provider) if stripped == name and name.startswith("openai/"): diff --git a/hermes_cli/model_switch.py b/hermes_cli/model_switch.py index 443321b8c257..41fbe36deb8b 100644 --- a/hermes_cli/model_switch.py +++ b/hermes_cli/model_switch.py @@ -41,7 +41,6 @@ get_model_capabilities, get_model_info, list_provider_models, - search_models_dev, ) logger = logging.getLogger(__name__) @@ -144,7 +143,7 @@ class ModelIdentity(NamedTuple): # Z.AI / GLM "glm": ModelIdentity("z-ai", "glm"), - # StepFun + # Step Plan (StepFun) "step": ModelIdentity("stepfun", "step"), # Xiaomi @@ -275,6 +274,11 @@ def parse_model_flags(raw_args: str) -> tuple[str, str, bool]: is_global = False explicit_provider = "" + # Normalize Unicode dashes (Telegram/iOS auto-converts -- to em/en dash) + # A single Unicode dash before a flag keyword becomes "--" + import re as _re + raw_args = _re.sub(r'[\u2012\u2013\u2014\u2015](provider|global)', r'--\1', raw_args) + # Extract --global if "--global" in raw_args: is_global = True @@ -300,6 +304,113 @@ def parse_model_flags(raw_args: str) -> tuple[str, str, bool]: # Alias resolution # --------------------------------------------------------------------------- +def _model_sort_key(model_id: str, prefix: str) -> tuple: + """Sort key for model version preference. + + Extracts version numbers after the family prefix and returns a sort key + that prefers higher versions. Suffix tokens (``pro``, ``omni``, etc.) + are used as tiebreakers, with common quality indicators ranked. + + Examples (with prefix ``"mimo"``):: + + mimo-v2.5-pro → (-2.5, 0, 'pro') # highest version wins + mimo-v2.5 → (-2.5, 1, '') # no suffix = lower than pro + mimo-v2-pro → (-2.0, 0, 'pro') + mimo-v2-omni → (-2.0, 1, 'omni') + mimo-v2-flash → (-2.0, 1, 'flash') + """ + # Strip the prefix (and optional "/" separator for aggregator slugs) + rest = model_id[len(prefix):] + if rest.startswith("/"): + rest = rest[1:] + rest = rest.lstrip("-").strip() + + # Parse version and suffix from the remainder. + # "v2.5-pro" → version [2.5], suffix "pro" + # "-omni" → version [], suffix "omni" + # State machine: start → in_version → between → in_suffix + nums: list[float] = [] + suffix_buf = "" + state = "start" + num_buf = "" + + for ch in rest: + if state == "start": + if ch in "vV": + state = "in_version" + elif ch.isdigit(): + state = "in_version" + num_buf += ch + elif ch in "-_.": + pass # skip separators before any content + else: + state = "in_suffix" + suffix_buf += ch + elif state == "in_version": + if ch.isdigit(): + num_buf += ch + elif ch == ".": + if "." in num_buf: + # Second dot — flush current number, start new component + try: + nums.append(float(num_buf.rstrip("."))) + except ValueError: + pass + num_buf = "" + else: + num_buf += ch + elif ch in "-_.": + if num_buf: + try: + nums.append(float(num_buf.rstrip("."))) + except ValueError: + pass + num_buf = "" + state = "between" + else: + if num_buf: + try: + nums.append(float(num_buf.rstrip("."))) + except ValueError: + pass + num_buf = "" + state = "in_suffix" + suffix_buf += ch + elif state == "between": + if ch.isdigit(): + state = "in_version" + num_buf = ch + elif ch in "vV": + state = "in_version" + elif ch in "-_.": + pass + else: + state = "in_suffix" + suffix_buf += ch + elif state == "in_suffix": + suffix_buf += ch + + # Flush remaining buffer (strip trailing dots — "5.4." → "5.4") + if num_buf and state == "in_version": + try: + nums.append(float(num_buf.rstrip("."))) + except ValueError: + pass + + suffix = suffix_buf.lower().strip("-_.") + suffix = suffix.strip() + + # Negate versions so higher → sorts first + version_key = tuple(-n for n in nums) + + # Suffix quality ranking: pro/max > (no suffix) > omni/flash/mini/lite + # Lower number = preferred + _SUFFIX_RANK = {"pro": 0, "max": 0, "plus": 0, "turbo": 0} + suffix_rank = _SUFFIX_RANK.get(suffix, 1) + + return version_key + (suffix_rank, suffix) + + def resolve_alias( raw_input: str, current_provider: str, @@ -307,9 +418,9 @@ def resolve_alias( """Resolve a short alias against the current provider's catalog. Looks up *raw_input* in :data:`MODEL_ALIASES`, then searches the - current provider's models.dev catalog for the first model whose ID - starts with ``vendor/family`` (or just ``family`` for non-aggregator - providers). + current provider's models.dev catalog for the model whose ID starts + with ``vendor/family`` (or just ``family`` for non-aggregator + providers) and has the **highest version**. Returns: ``(provider, resolved_model_id, alias_name)`` if a match is @@ -337,28 +448,44 @@ def resolve_alias( vendor, family = identity - # Search the provider's catalog from models.dev + # Build catalog from models.dev, then merge in static _PROVIDER_MODELS + # entries that models.dev may be missing (e.g. newly added models not + # yet synced to the registry). catalog = list_provider_models(current_provider) - if not catalog: - return None + try: + from hermes_cli.models import _PROVIDER_MODELS + static = _PROVIDER_MODELS.get(current_provider, []) + if static: + seen = {m.lower() for m in catalog} + for m in static: + if m.lower() not in seen: + catalog.append(m) + except Exception: + pass # For aggregators, models are vendor/model-name format aggregator = is_aggregator(current_provider) - for model_id in catalog: - mid_lower = model_id.lower() - if aggregator: - # Match vendor/family prefix -- e.g. "anthropic/claude-sonnet" - prefix = f"{vendor}/{family}".lower() - if mid_lower.startswith(prefix): - return (current_provider, model_id, key) - else: - # Non-aggregator: bare names -- e.g. "claude-sonnet-4-6" - family_lower = family.lower() - if mid_lower.startswith(family_lower): - return (current_provider, model_id, key) + if aggregator: + prefix = f"{vendor}/{family}".lower() + matches = [ + mid for mid in catalog + if mid.lower().startswith(prefix) + ] + else: + family_lower = family.lower() + matches = [ + mid for mid in catalog + if mid.lower().startswith(family_lower) + ] - return None + if not matches: + return None + + # Sort by version descending — prefer the latest/highest version + prefix_for_sort = f"{vendor}/{family}" if aggregator else family + matches.sort(key=lambda m: _model_sort_key(m, prefix_for_sort)) + return (current_provider, matches[0], key) def get_authenticated_provider_slugs( @@ -453,6 +580,7 @@ def switch_model( ModelSwitchResult with all information the caller needs. """ from hermes_cli.models import ( + copilot_model_api_mode, detect_provider_for_model, validate_requested_model, opencode_model_api_mode, @@ -673,6 +801,7 @@ def switch_model( _da = DIRECT_ALIASES.get(resolved_alias) if _da is not None and _da.base_url: base_url = _da.base_url + api_mode = "" # clear so determine_api_mode re-detects from URL if not api_key: api_key = "no-key-required" @@ -687,12 +816,12 @@ def switch_model( api_key=api_key, base_url=base_url, ) - except Exception: + except Exception as e: validation = { - "accepted": True, - "persist": True, + "accepted": False, + "persist": False, "recognized": False, - "message": None, + "message": f"Could not validate `{new_model}`: {e}", } if not validation.get("accepted"): @@ -706,14 +835,38 @@ def switch_model( error_message=msg, ) + # Apply auto-correction if validation found a closer match + if validation.get("corrected_model"): + new_model = validation["corrected_model"] + + # --- Copilot api_mode override --- + if target_provider in {"copilot", "github-copilot"}: + api_mode = copilot_model_api_mode(new_model, api_key=api_key) + # --- OpenCode api_mode override --- - if target_provider in {"opencode-zen", "opencode-go", "opencode", "opencode-go"}: + if target_provider in {"opencode-zen", "opencode-go", "opencode"}: api_mode = opencode_model_api_mode(target_provider, new_model) # --- Determine api_mode if not already set --- if not api_mode: api_mode = determine_api_mode(target_provider, base_url) + # OpenCode base URLs end with /v1 for OpenAI-compatible models, but the + # Anthropic SDK prepends its own /v1/messages to the base_url. Strip the + # trailing /v1 so the SDK constructs the correct path (e.g. + # https://opencode.ai/zen/go/v1/messages instead of .../v1/v1/messages). + # Mirrors the same logic in hermes_cli.runtime_provider.resolve_runtime_provider; + # without it, /model switches into an anthropic_messages-routed OpenCode + # model (e.g. `/model minimax-m2.7` on opencode-go, `/model claude-sonnet-4-6` + # on opencode-zen) hit a double /v1 and returned OpenCode's website 404 page. + if ( + api_mode == "anthropic_messages" + and target_provider in {"opencode-zen", "opencode-go"} + and isinstance(base_url, str) + and base_url + ): + base_url = re.sub(r"/v1/?$", "", base_url) + # --- Get capabilities (legacy) --- capabilities = get_model_capabilities(target_provider, new_model) @@ -752,6 +905,7 @@ def switch_model( def list_authenticated_providers( current_provider: str = "", + current_base_url: str = "", user_providers: dict = None, custom_providers: list | None = None, max_models: int = 8, @@ -780,10 +934,14 @@ def list_authenticated_providers( get_provider_info as _mdev_pinfo, ) from hermes_cli.auth import PROVIDER_REGISTRY - from hermes_cli.models import OPENROUTER_MODELS, _PROVIDER_MODELS + from hermes_cli.models import ( + OPENROUTER_MODELS, _PROVIDER_MODELS, + _MODELS_DEV_PREFERRED, _merge_with_models_dev, + ) results: List[dict] = [] - seen_slugs: set = set() + seen_slugs: set = set() # lowercase-normalized to catch case variants (#9545) + seen_mdev_ids: set = set() # prevent duplicate entries for aliases (e.g. kimi-coding + kimi-coding-cn) data = fetch_models_dev() @@ -793,9 +951,18 @@ def list_authenticated_providers( # "nous" shares OpenRouter's curated list if not separately defined if "nous" not in curated: curated["nous"] = curated["openrouter"] + # Ollama Cloud uses dynamic discovery (no static curated list) + if "ollama-cloud" not in curated: + from hermes_cli.models import fetch_ollama_cloud_models + curated["ollama-cloud"] = fetch_ollama_cloud_models() # --- 1. Check Hermes-mapped providers --- for hermes_id, mdev_id in PROVIDER_TO_MODELS_DEV.items(): + # Skip aliases that map to the same models.dev provider (e.g. + # kimi-coding and kimi-coding-cn both → kimi-for-coding). + # The first one with valid credentials wins (#10526). + if mdev_id in seen_mdev_ids: + continue pdata = data.get(mdev_id) if not isinstance(pdata, dict): continue @@ -804,6 +971,10 @@ def list_authenticated_providers( # source of truth. models.dev can have wrong mappings (e.g. # minimax-cn → MINIMAX_API_KEY instead of MINIMAX_CN_API_KEY). pconfig = PROVIDER_REGISTRY.get(hermes_id) + # Skip non-API-key auth providers here — they are handled in + # section 2 (HERMES_OVERLAYS) with proper auth store checking. + if pconfig and pconfig.auth_type != "api_key": + continue if pconfig and pconfig.api_key_env_vars: env_vars = list(pconfig.api_key_env_vars) else: @@ -816,8 +987,13 @@ def list_authenticated_providers( if not has_creds: continue - # Use curated list, falling back to models.dev if no curated list + # Use curated list, falling back to models.dev if no curated list. + # For preferred providers, merge models.dev entries into the curated + # catalog so newly released models (e.g. mimo-v2.5-pro on opencode-go) + # show up in the picker without requiring a Hermes release. model_ids = curated.get(hermes_id, []) + if hermes_id in _MODELS_DEV_PREFERRED: + model_ids = _merge_with_models_dev(hermes_id, model_ids) total = len(model_ids) top = model_ids[:max_models] @@ -834,7 +1010,8 @@ def list_authenticated_providers( "total_models": total, "source": "built-in", }) - seen_slugs.add(slug) + seen_slugs.add(slug.lower()) + seen_mdev_ids.add(mdev_id) # --- 2. Check Hermes-only providers (nous, openai-codex, copilot, opencode-go) --- from hermes_cli.providers import HERMES_OVERLAYS @@ -846,12 +1023,12 @@ def list_authenticated_providers( _mdev_to_hermes = {v: k for k, v in PROVIDER_TO_MODELS_DEV.items()} for pid, overlay in HERMES_OVERLAYS.items(): - if pid in seen_slugs: + if pid.lower() in seen_slugs: continue # Resolve Hermes slug — e.g. "github-copilot" → "copilot" hermes_slug = _mdev_to_hermes.get(pid, pid) - if hermes_slug in seen_slugs: + if hermes_slug.lower() in seen_slugs: continue # Check if credentials exist @@ -920,6 +1097,9 @@ def list_authenticated_providers( # Use curated list — look up by Hermes slug, fall back to overlay key model_ids = curated.get(hermes_slug, []) or curated.get(pid, []) + # Merge with models.dev for preferred providers (same rationale as above). + if hermes_slug in _MODELS_DEV_PREFERRED: + model_ids = _merge_with_models_dev(hermes_slug, model_ids) total = len(model_ids) top = model_ids[:max_models] @@ -932,25 +1112,112 @@ def list_authenticated_providers( "total_models": total, "source": "hermes", }) - seen_slugs.add(pid) - seen_slugs.add(hermes_slug) + seen_slugs.add(pid.lower()) + seen_slugs.add(hermes_slug.lower()) + + # --- 2b. Cross-check canonical provider list --- + # Catches providers that are in CANONICAL_PROVIDERS but weren't found + # in PROVIDER_TO_MODELS_DEV or HERMES_OVERLAYS (keeps /model in sync + # with `hermes model`). + try: + from hermes_cli.models import CANONICAL_PROVIDERS as _canon_provs + except ImportError: + _canon_provs = [] + + for _cp in _canon_provs: + if _cp.slug.lower() in seen_slugs: + continue + + # Check credentials via PROVIDER_REGISTRY (auth.py) + _cp_config = _auth_registry.get(_cp.slug) + _cp_has_creds = False + if _cp_config and _cp_config.api_key_env_vars: + _cp_has_creds = any(os.environ.get(ev) for ev in _cp_config.api_key_env_vars) + # Also check auth store and credential pool + if not _cp_has_creds: + try: + from hermes_cli.auth import _load_auth_store + _cp_store = _load_auth_store() + _cp_providers_store = _cp_store.get("providers", {}) + _cp_pool_store = _cp_store.get("credential_pool", {}) + if _cp_store and ( + _cp.slug in _cp_providers_store + or _cp.slug in _cp_pool_store + ): + _cp_has_creds = True + except Exception: + pass + if not _cp_has_creds: + try: + from agent.credential_pool import load_pool + _cp_pool = load_pool(_cp.slug) + if _cp_pool.has_credentials(): + _cp_has_creds = True + except Exception: + pass + + if not _cp_has_creds: + continue + + _cp_model_ids = curated.get(_cp.slug, []) + _cp_total = len(_cp_model_ids) + _cp_top = _cp_model_ids[:max_models] + + results.append({ + "slug": _cp.slug, + "name": _cp.label, + "is_current": _cp.slug == current_provider, + "is_user_defined": False, + "models": _cp_top, + "total_models": _cp_total, + "source": "canonical", + }) + seen_slugs.add(_cp.slug.lower()) # --- 3. User-defined endpoints from config --- + # Track (name, base_url) of what section 3 emits so section 4 can skip + # any overlapping ``custom_providers:`` entries. Callers typically pass + # both (gateway/CLI invoke ``get_compatible_custom_providers()`` which + # merges ``providers:`` into the list) — without this, the same endpoint + # produces two picker rows: one bare-slug ("openrouter") from section 3 + # and one "custom:openrouter" from section 4, both labelled identically. + _section3_emitted_pairs: set = set() if user_providers and isinstance(user_providers, dict): for ep_name, ep_cfg in user_providers.items(): if not isinstance(ep_cfg, dict): continue + # Skip if this slug was already emitted (e.g. canonical provider + # with the same name) or will be picked up by section 4. + if ep_name.lower() in seen_slugs: + continue display_name = ep_cfg.get("name", "") or ep_name - api_url = ep_cfg.get("api", "") or ep_cfg.get("url", "") or "" - default_model = ep_cfg.get("default_model", "") + # ``base_url`` is Hermes's canonical write key (matches + # custom_providers and _save_custom_provider); ``api`` / ``url`` + # remain as fallbacks for hand-edited / legacy configs. + api_url = ( + ep_cfg.get("base_url", "") + or ep_cfg.get("api", "") + or ep_cfg.get("url", "") + or "" + ) + # ``default_model`` is the legacy key; ``model`` matches what + # custom_providers entries use, so accept either. + default_model = ep_cfg.get("default_model", "") or ep_cfg.get("model", "") # Build models list from both default_model and full models array models_list = [] if default_model: models_list.append(default_model) - # Also include the full models list from config + # Also include the full models list from config. + # Hermes writes ``models:`` as a dict keyed by model id + # (see hermes_cli/main.py::_save_custom_provider); older + # configs or hand-edited files may still use a list. cfg_models = ep_cfg.get("models", []) - if isinstance(cfg_models, list): + if isinstance(cfg_models, dict): + for m in cfg_models: + if m and m not in models_list: + models_list.append(m) + elif isinstance(cfg_models, list): for m in cfg_models: if m and m not in models_list: models_list.append(m) @@ -967,47 +1234,144 @@ def list_authenticated_providers( "source": "user-config", "api_url": api_url, }) + seen_slugs.add(ep_name.lower()) + seen_slugs.add(custom_provider_slug(display_name).lower()) + _pair = ( + str(display_name).strip().lower(), + str(api_url).strip().rstrip("/").lower(), + ) + if _pair[0] and _pair[1]: + _section3_emitted_pairs.add(_pair) # --- 4. Saved custom providers from config --- + # Each ``custom_providers`` entry represents one model under a named + # provider. Entries sharing the same endpoint (``base_url`` + ``api_key``) + # are grouped into a single picker row, so e.g. four Ollama entries + # pointing at ``http://localhost:11434/v1`` with per-model display names + # ("Ollama — GLM 5.1", "Ollama — Qwen3-coder", ...) appear as one + # "Ollama" row with four models inside instead of four near-duplicates + # that differ only by suffix. Entries with distinct endpoints still + # produce separate rows. + # + # When the grouped endpoint matches ``current_base_url`` the group's + # slug becomes ``current_provider`` so that selecting a model from the + # picker flows back through the runtime provider that already holds + # valid credentials — no re-resolution needed. if custom_providers and isinstance(custom_providers, list): + from collections import OrderedDict + + # Key by (base_url, api_key) instead of slug: names frequently + # differ per model ("Ollama — X") while the endpoint stays the + # same. Slug-based grouping left them as separate rows. + groups: "OrderedDict[tuple, dict]" = OrderedDict() for entry in custom_providers: if not isinstance(entry, dict): continue - display_name = (entry.get("name") or "").strip() + raw_name = (entry.get("name") or "").strip() api_url = ( entry.get("base_url", "") or entry.get("url", "") or entry.get("api", "") or "" - ).strip() - if not display_name or not api_url: - continue - - slug = custom_provider_slug(display_name) - if slug in seen_slugs: + ).strip().rstrip("/") + if not raw_name or not api_url: continue - - models_list = [] + api_key = (entry.get("api_key") or "").strip() + + group_key = (api_url, api_key) + if group_key not in groups: + # Strip per-model suffix so "Ollama — GLM 5.1" becomes + # "Ollama" for the grouped row. Em dash is the convention + # Hermes's own writer uses; a hyphen variant is accepted + # for hand-edited configs. + display_name = raw_name + for sep in ("—", " - "): + if sep in display_name: + display_name = display_name.split(sep)[0].strip() + break + if not display_name: + display_name = raw_name + # If this endpoint matches the currently active one, use + # ``current_provider`` as the slug so picker-driven switches + # route through the live credential pipeline. + if ( + current_base_url + and api_url == current_base_url.strip().rstrip("/") + ): + slug = current_provider or custom_provider_slug(display_name) + else: + slug = custom_provider_slug(display_name) + groups[group_key] = { + "slug": slug, + "name": display_name, + "api_url": api_url, + "models": [], + } + + # The singular ``model:`` field only holds the currently + # active model. Hermes's own writer (main.py::_save_custom_provider) + # stores every configured model as a dict under ``models:``; + # downstream readers (agent/models_dev.py, gateway/run.py, + # run_agent.py, hermes_cli/config.py) already consume that dict. default_model = (entry.get("model") or "").strip() - if default_model: - models_list.append(default_model) + if default_model and default_model not in groups[group_key]["models"]: + groups[group_key]["models"].append(default_model) + cfg_models = entry.get("models", {}) + if isinstance(cfg_models, dict): + for m in cfg_models: + if m and m not in groups[group_key]["models"]: + groups[group_key]["models"].append(m) + elif isinstance(cfg_models, list): + for m in cfg_models: + if m and m not in groups[group_key]["models"]: + groups[group_key]["models"].append(m) + + _section4_emitted_slugs: set = set() + for grp in groups.values(): + slug = grp["slug"] + # If the slug is already claimed by a built-in / overlay / + # user-provider row (sections 1-3), skip this custom group + # to avoid shadowing a real provider. + if slug.lower() in seen_slugs and slug.lower() not in _section4_emitted_slugs: + continue + # If a prior section-4 group already used this slug (two custom + # endpoints with the same cleaned name — e.g. two OpenAI- + # compatible gateways named identically with different keys), + # append a counter so both rows stay visible in the picker. + if slug.lower() in _section4_emitted_slugs: + base_slug = slug + n = 2 + while f"{base_slug}-{n}".lower() in seen_slugs: + n += 1 + slug = f"{base_slug}-{n}" + grp["slug"] = slug + # Skip if section 3 already emitted this endpoint under its + # ``providers:`` dict key — matches on (display_name, base_url). + # Prevents two picker rows labelled identically when callers + # pass both ``user_providers`` and a compatibility-merged + # ``custom_providers`` list. + _pair_key = ( + str(grp["name"]).strip().lower(), + str(grp["api_url"]).strip().rstrip("/").lower(), + ) + if _pair_key[0] and _pair_key[1] and _pair_key in _section3_emitted_pairs: + continue results.append({ "slug": slug, - "name": display_name, + "name": grp["name"], "is_current": slug == current_provider, "is_user_defined": True, - "models": models_list, - "total_models": len(models_list), + "models": grp["models"], + "total_models": len(grp["models"]), "source": "user-config", - "api_url": api_url, + "api_url": grp["api_url"], }) - seen_slugs.add(slug) + seen_slugs.add(slug.lower()) + _section4_emitted_slugs.add(slug.lower()) # Sort: current provider first, then by model count descending results.sort(key=lambda r: (not r["is_current"], -r["total_models"])) return results - - diff --git a/hermes_cli/models.py b/hermes_cli/models.py index c3f1408d1b80..a1f2cbec619f 100644 --- a/hermes_cli/models.py +++ b/hermes_cli/models.py @@ -11,8 +11,16 @@ import os import urllib.request import urllib.error +import time from difflib import get_close_matches -from typing import Any, Optional +from pathlib import Path +from typing import Any, NamedTuple, Optional + +from hermes_cli import __version__ as _HERMES_VERSION + +# Identify ourselves so endpoints fronted by Cloudflare's Browser Integrity +# Check (error 1010) don't reject the default ``Python-urllib/*`` signature. +_HERMES_USER_AGENT = f"hermes-cli/{_HERMES_VERSION}" COPILOT_BASE_URL = "https://api.githubcopilot.com" COPILOT_MODELS_URL = f"{COPILOT_BASE_URL}/models" @@ -24,14 +32,18 @@ # Fallback OpenRouter snapshot used when the live catalog is unavailable. # (model_id, display description shown in menus) OPENROUTER_MODELS: list[tuple[str, str]] = [ - ("anthropic/claude-opus-4.6", "recommended"), + ("moonshotai/kimi-k2.6", "recommended"), + ("anthropic/claude-opus-4.7", ""), + ("anthropic/claude-opus-4.6", ""), ("anthropic/claude-sonnet-4.6", ""), ("qwen/qwen3.6-plus", ""), ("anthropic/claude-sonnet-4.5", ""), ("anthropic/claude-haiku-4.5", ""), + ("openrouter/elephant-alpha", "free"), ("openai/gpt-5.4", ""), ("openai/gpt-5.4-mini", ""), - ("xiaomi/mimo-v2-pro", ""), + ("xiaomi/mimo-v2.5-pro", ""), + ("xiaomi/mimo-v2.5", ""), ("openai/gpt-5.3-codex", ""), ("google/gemini-3-pro-image-preview", ""), ("google/gemini-3-flash-preview", ""), @@ -42,9 +54,10 @@ ("stepfun/step-3.5-flash", ""), ("minimax/minimax-m2.7", ""), ("minimax/minimax-m2.5", ""), + ("minimax/minimax-m2.5:free", "free"), ("z-ai/glm-5.1", ""), + ("z-ai/glm-5v-turbo", ""), ("z-ai/glm-5-turbo", ""), - ("moonshotai/kimi-k2.5", ""), ("x-ai/grok-4.20", ""), ("nvidia/nemotron-3-super-120b-a12b", ""), ("nvidia/nemotron-3-super-120b-a12b:free", "free"), @@ -57,6 +70,31 @@ _openrouter_catalog_cache: list[tuple[str, str]] | None = None +# Fallback Vercel AI Gateway snapshot used when the live catalog is unavailable. +# OSS / open-weight models prioritized first, then closed-source by family. +# Slugs match Vercel's actual /v1/models catalog (e.g. alibaba/ for Qwen, +# zai/ and xai/ without hyphens). +VERCEL_AI_GATEWAY_MODELS: list[tuple[str, str]] = [ + ("moonshotai/kimi-k2.6", "recommended"), + ("alibaba/qwen3.6-plus", ""), + ("zai/glm-5.1", ""), + ("minimax/minimax-m2.7", ""), + ("anthropic/claude-sonnet-4.6", ""), + ("anthropic/claude-opus-4.7", ""), + ("anthropic/claude-opus-4.6", ""), + ("anthropic/claude-haiku-4.5", ""), + ("openai/gpt-5.4", ""), + ("openai/gpt-5.4-mini", ""), + ("openai/gpt-5.3-codex", ""), + ("google/gemini-3.1-pro-preview", ""), + ("google/gemini-3-flash", ""), + ("google/gemini-3.1-flash-lite-preview", ""), + ("xai/grok-4.20-reasoning", ""), +] + +_ai_gateway_catalog_cache: list[tuple[str, str]] | None = None + + def _codex_curated_models() -> list[str]: """Derive the openai-codex curated list from codex_models.py. @@ -70,7 +108,10 @@ def _codex_curated_models() -> list[str]: _PROVIDER_MODELS: dict[str, list[str]] = { "nous": [ - "xiaomi/mimo-v2-pro", + "moonshotai/kimi-k2.6", + "xiaomi/mimo-v2.5-pro", + "xiaomi/mimo-v2.5", + "anthropic/claude-opus-4.7", "anthropic/claude-opus-4.6", "anthropic/claude-sonnet-4.6", "anthropic/claude-sonnet-4.5", @@ -87,13 +128,12 @@ def _codex_curated_models() -> list[str]: "stepfun/step-3.5-flash", "minimax/minimax-m2.7", "minimax/minimax-m2.5", + "minimax/minimax-m2.5:free", "z-ai/glm-5.1", + "z-ai/glm-5v-turbo", "z-ai/glm-5-turbo", - "moonshotai/kimi-k2.5", "x-ai/grok-4.20-beta", "nvidia/nemotron-3-super-120b-a12b", - "nvidia/nemotron-3-super-120b-a12b:free", - "arcee-ai/trinity-large-preview:free", "arcee-ai/trinity-large-thinking", "openai/gpt-5.4-pro", "openai/gpt-5.4-nano", @@ -120,51 +160,64 @@ def _codex_curated_models() -> list[str]: ], "gemini": [ "gemini-3.1-pro-preview", + "gemini-3-pro-preview", "gemini-3-flash-preview", "gemini-3.1-flash-lite-preview", - "gemini-2.5-pro", - "gemini-2.5-flash", - "gemini-2.5-flash-lite", - # Gemma open models (also served via AI Studio) - "gemma-4-31b-it", - "gemma-4-26b-it", + ], + "google-gemini-cli": [ + "gemini-3.1-pro-preview", + "gemini-3-pro-preview", + "gemini-3-flash-preview", ], "zai": [ "glm-5.1", "glm-5", + "glm-5v-turbo", "glm-5-turbo", "glm-4.7", "glm-4.5", "glm-4.5-flash", ], "xai": [ - "grok-4.20-0309-reasoning", - "grok-4.20-0309-non-reasoning", - "grok-4.20-multi-agent-0309", + "grok-4.20-reasoning", "grok-4-1-fast-reasoning", - "grok-4-1-fast-non-reasoning", - "grok-4-fast-reasoning", - "grok-4-fast-non-reasoning", - "grok-4-0709", - "grok-code-fast-1", - "grok-3", - "grok-3-mini", + ], + "nvidia": [ + # NVIDIA flagship reasoning models + "nvidia/nemotron-3-super-120b-a12b", + "nvidia/nemotron-3-nano-30b-a3b", + "nvidia/llama-3.3-nemotron-super-49b-v1.5", + # Third-party agentic models hosted on build.nvidia.com + # (map to OpenRouter defaults — users get familiar picks on NIM) + "qwen/qwen3.5-397b-a17b", + "deepseek-ai/deepseek-v3.2", + "moonshotai/kimi-k2.6", + "minimaxai/minimax-m2.5", + "z-ai/glm5", + "openai/gpt-oss-120b", ], "kimi-coding": [ - "kimi-for-coding", + "kimi-k2.6", "kimi-k2.5", + "kimi-for-coding", "kimi-k2-thinking", "kimi-k2-thinking-turbo", "kimi-k2-turbo-preview", "kimi-k2-0905-preview", ], "kimi-coding-cn": [ + "kimi-k2.6", "kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview", "kimi-k2-0905-preview", ], + "stepfun": [ + "step-3.5-flash", + "step-3.5-flash-2603", + ], "moonshot": [ + "kimi-k2.6", "kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview", @@ -183,6 +236,7 @@ def _codex_curated_models() -> list[str]: "MiniMax-M2", ], "anthropic": [ + "claude-opus-4-7", "claude-opus-4-6", "claude-sonnet-4-6", "claude-opus-4-5-20251101", @@ -196,15 +250,22 @@ def _codex_curated_models() -> list[str]: "deepseek-reasoner", ], "xiaomi": [ + "mimo-v2.5-pro", + "mimo-v2.5", "mimo-v2-pro", "mimo-v2-omni", "mimo-v2-flash", ], + "arcee": [ + "trinity-large-thinking", + "trinity-large-preview", + "trinity-mini", + ], "opencode-zen": [ + "kimi-k2.5", "gpt-5.4-pro", "gpt-5.4", "gpt-5.3-codex", - "gpt-5.3-codex-spark", "gpt-5.2", "gpt-5.2-codex", "gpt-5.1", @@ -232,33 +293,24 @@ def _codex_curated_models() -> list[str]: "glm-5", "glm-4.7", "glm-4.6", - "kimi-k2.5", "kimi-k2-thinking", "kimi-k2", "qwen3-coder", "big-pickle", ], "opencode-go": [ - "glm-5", + "kimi-k2.6", "kimi-k2.5", + "glm-5.1", + "glm-5", + "mimo-v2.5-pro", + "mimo-v2.5", "mimo-v2-pro", "mimo-v2-omni", "minimax-m2.7", "minimax-m2.5", - ], - "ai-gateway": [ - "anthropic/claude-opus-4.6", - "anthropic/claude-sonnet-4.6", - "anthropic/claude-sonnet-4.5", - "anthropic/claude-haiku-4.5", - "openai/gpt-5", - "openai/gpt-4.1", - "openai/gpt-4.1-mini", - "google/gemini-3-pro-preview", - "google/gemini-3-flash", - "google/gemini-2.5-pro", - "google/gemini-2.5-flash", - "deepseek/deepseek-v3.2", + "qwen3.6-plus", + "qwen3.5-plus", ], "kilocode": [ "anthropic/claude-opus-4.6", @@ -273,40 +325,57 @@ def _codex_curated_models() -> list[str]: # to https://dashscope-intl.aliyuncs.com/compatible-mode/v1 (OpenAI-compat) # or https://dashscope-intl.aliyuncs.com/apps/anthropic (Anthropic-compat). "alibaba": [ + "kimi-k2.5", "qwen3.5-plus", "qwen3-coder-plus", "qwen3-coder-next", # Third-party models available on coding-intl "glm-5", "glm-4.7", - "kimi-k2.5", "MiniMax-M2.5", ], # Curated HF model list — only agentic models that map to OpenRouter defaults. "huggingface": [ + "moonshotai/Kimi-K2.5", "Qwen/Qwen3.5-397B-A17B", "Qwen/Qwen3.5-35B-A3B", "deepseek-ai/DeepSeek-V3.2", - "moonshotai/Kimi-K2.5", "MiniMaxAI/MiniMax-M2.5", "zai-org/GLM-5", "XiaomiMiMo/MiMo-V2-Flash", "moonshotai/Kimi-K2-Thinking", + "moonshotai/Kimi-K2.6", + ], + # AWS Bedrock — static fallback list used when dynamic discovery is + # unavailable (no boto3, no credentials, or API error). The agent + # prefers live discovery via ListFoundationModels + ListInferenceProfiles. + # Use inference profile IDs (us.*) since most models require them. + "bedrock": [ + "us.anthropic.claude-sonnet-4-6", + "us.anthropic.claude-opus-4-6-v1", + "us.anthropic.claude-haiku-4-5-20251001-v1:0", + "us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "us.amazon.nova-pro-v1:0", + "us.amazon.nova-lite-v1:0", + "us.amazon.nova-micro-v1:0", + "deepseek.v3.2", + "us.meta.llama4-maverick-17b-instruct-v1:0", + "us.meta.llama4-scout-17b-instruct-v1:0", ], } +# Vercel AI Gateway: derive the bare-model-id catalog from the curated +# ``VERCEL_AI_GATEWAY_MODELS`` snapshot so both the picker (tuples with descriptions) +# and the static fallback catalog (bare ids) stay in sync from a single +# source of truth. +_PROVIDER_MODELS["ai-gateway"] = [mid for mid, _ in VERCEL_AI_GATEWAY_MODELS] + # --------------------------------------------------------------------------- -# Nous Portal free-model filtering +# Nous Portal free-model helper # --------------------------------------------------------------------------- -# Models that are ALLOWED to appear when priced as free on Nous Portal. -# Any other free model is hidden — prevents promotional/temporary free models -# from cluttering the selection when users are paying subscribers. -# Models in this list are ALSO filtered out if they are NOT free (i.e. they -# should only appear in the menu when they are genuinely free). -_NOUS_ALLOWED_FREE_MODELS: frozenset[str] = frozenset({ - "xiaomi/mimo-v2-pro", - "xiaomi/mimo-v2-omni", -}) +# The Nous Portal models endpoint is the source of truth for which models +# are currently offered (free or paid). We trust whatever it returns and +# surface it to users as-is — no local allowlist filtering. def _is_model_free(model_id: str, pricing: dict[str, dict[str, str]]) -> bool: @@ -320,35 +389,6 @@ def _is_model_free(model_id: str, pricing: dict[str, dict[str, str]]) -> bool: return False -def filter_nous_free_models( - model_ids: list[str], - pricing: dict[str, dict[str, str]], -) -> list[str]: - """Filter the Nous Portal model list according to free-model policy. - - Rules: - • Paid models that are NOT in the allowlist → keep (normal case). - • Free models that are NOT in the allowlist → drop. - • Allowlist models that ARE free → keep. - • Allowlist models that are NOT free → drop. - """ - if not pricing: - return model_ids # no pricing data — can't filter, show everything - - result: list[str] = [] - for mid in model_ids: - free = _is_model_free(mid, pricing) - if mid in _NOUS_ALLOWED_FREE_MODELS: - # Allowlist model: only show when it's actually free - if free: - result.append(mid) - else: - # Regular model: keep only when it's NOT free - if not free: - result.append(mid) - return result - - # --------------------------------------------------------------------------- # Nous Portal account tier detection # --------------------------------------------------------------------------- @@ -412,8 +452,7 @@ def partition_nous_models_by_tier( ) -> tuple[list[str], list[str]]: """Split Nous models into (selectable, unavailable) based on user tier. - For paid-tier users: all models are selectable, none unavailable - (free-model filtering is handled separately by ``filter_nous_free_models``). + For paid-tier users: all models are selectable, none unavailable. For free-tier users: only free models are selectable; paid models are returned as unavailable (shown grayed out in the menu). @@ -452,8 +491,6 @@ def check_nous_free_tier() -> bool: Returns False (assume paid) on any error — never blocks paying users. """ global _free_tier_cache - import time - now = time.monotonic() if _free_tier_cache is not None: cached_result, cached_at = _free_tier_cache @@ -485,30 +522,209 @@ def check_nous_free_tier() -> bool: return False # default to paid on error — don't block users -_PROVIDER_LABELS = { - "openrouter": "OpenRouter", - "openai-codex": "OpenAI Codex", - "copilot-acp": "GitHub Copilot ACP", - "nous": "Nous Portal", - "copilot": "GitHub Copilot", - "gemini": "Google AI Studio", - "zai": "Z.AI / GLM", - "kimi-coding": "Kimi / Moonshot", - "kimi-coding-cn": "Kimi / Moonshot (China)", - "minimax": "MiniMax", - "minimax-cn": "MiniMax (China)", - "anthropic": "Anthropic", - "deepseek": "DeepSeek", - "opencode-zen": "OpenCode Zen", - "opencode-go": "OpenCode Go", - "ai-gateway": "AI Gateway", - "kilocode": "Kilo Code", - "alibaba": "Alibaba Cloud (DashScope)", - "qwen-oauth": "Qwen OAuth (Portal)", - "huggingface": "Hugging Face", - "xiaomi": "Xiaomi MiMo", - "custom": "Custom endpoint", -} +# --------------------------------------------------------------------------- +# Nous Portal recommended models +# +# The Portal publishes a curated list of suggested models (separated into +# paid and free tiers) plus dedicated recommendations for compaction (text +# summarisation / auxiliary) and vision tasks. We fetch it once per process +# with a TTL cache so callers can ask "what's the best aux model right now?" +# without hitting the network on every lookup. +# +# Shape of the response (fields we care about): +# { +# "paidRecommendedModels": [ {modelName, ...}, ... ], +# "freeRecommendedModels": [ {modelName, ...}, ... ], +# "paidRecommendedCompactionModel": {modelName, ...} | null, +# "paidRecommendedVisionModel": {modelName, ...} | null, +# "freeRecommendedCompactionModel": {modelName, ...} | null, +# "freeRecommendedVisionModel": {modelName, ...} | null, +# } +# --------------------------------------------------------------------------- + +NOUS_RECOMMENDED_MODELS_PATH = "/api/nous/recommended-models" +_NOUS_RECOMMENDED_CACHE_TTL: int = 600 # seconds (10 minutes) +# (result_dict, timestamp) keyed by portal_base_url so staging vs prod don't collide. +_nous_recommended_cache: dict[str, tuple[dict[str, Any], float]] = {} + + +def fetch_nous_recommended_models( + portal_base_url: str = "", + timeout: float = 5.0, + *, + force_refresh: bool = False, +) -> dict[str, Any]: + """Fetch the Nous Portal's curated recommended-models payload. + + Hits ``/api/nous/recommended-models``. The endpoint is public — + no auth is required. Results are cached per portal URL for + ``_NOUS_RECOMMENDED_CACHE_TTL`` seconds; pass ``force_refresh=True`` to + bypass the cache. + + Returns the parsed JSON dict on success, or ``{}`` on any failure + (network, parse, non-2xx). Callers must treat missing/null fields as + "no recommendation" and fall back to their own default. + """ + base = (portal_base_url or "https://portal.nousresearch.com").rstrip("/") + now = time.monotonic() + cached = _nous_recommended_cache.get(base) + if not force_refresh and cached is not None: + payload, cached_at = cached + if now - cached_at < _NOUS_RECOMMENDED_CACHE_TTL: + return payload + + url = f"{base}{NOUS_RECOMMENDED_MODELS_PATH}" + try: + req = urllib.request.Request( + url, + headers={"Accept": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=timeout) as resp: + data = json.loads(resp.read().decode()) + if not isinstance(data, dict): + data = {} + except Exception: + data = {} + + _nous_recommended_cache[base] = (data, now) + return data + + +def _resolve_nous_portal_url() -> str: + """Best-effort lookup of the Portal base URL the user is authed against.""" + try: + from hermes_cli.auth import ( + DEFAULT_NOUS_PORTAL_URL, + get_provider_auth_state, + ) + state = get_provider_auth_state("nous") or {} + portal = str(state.get("portal_base_url") or "").strip() + if portal: + return portal.rstrip("/") + return str(DEFAULT_NOUS_PORTAL_URL).rstrip("/") + except Exception: + return "https://portal.nousresearch.com" + + +def _extract_model_name(entry: Any) -> Optional[str]: + """Pull the ``modelName`` field from a recommended-model entry, else None.""" + if not isinstance(entry, dict): + return None + model_name = entry.get("modelName") + if isinstance(model_name, str) and model_name.strip(): + return model_name.strip() + return None + + +def get_nous_recommended_aux_model( + *, + vision: bool = False, + free_tier: Optional[bool] = None, + portal_base_url: str = "", + force_refresh: bool = False, +) -> Optional[str]: + """Return the Portal's recommended model name for an auxiliary task. + + Picks the best field from the Portal's recommended-models payload: + + * ``vision=True`` → ``paidRecommendedVisionModel`` (paid tier) or + ``freeRecommendedVisionModel`` (free tier) + * ``vision=False`` → ``paidRecommendedCompactionModel`` or + ``freeRecommendedCompactionModel`` + + When ``free_tier`` is ``None`` (default) the user's tier is auto-detected + via :func:`check_nous_free_tier`. Pass an explicit bool to bypass the + detection — useful for tests or when the caller already knows the tier. + + For paid-tier users we prefer the paid recommendation but gracefully fall + back to the free recommendation if the Portal returned ``null`` for the + paid field (common during the staged rollout of new paid models). + + Returns ``None`` when every candidate is missing, null, or the fetch + fails — callers should fall back to their own default (currently + ``google/gemini-3-flash-preview``). + """ + base = portal_base_url or _resolve_nous_portal_url() + payload = fetch_nous_recommended_models(base, force_refresh=force_refresh) + if not payload: + return None + + if free_tier is None: + try: + free_tier = check_nous_free_tier() + except Exception: + # On any detection error, assume paid — paid users see both fields + # anyway so this is a safe default that maximises model quality. + free_tier = False + + if vision: + paid_key, free_key = "paidRecommendedVisionModel", "freeRecommendedVisionModel" + else: + paid_key, free_key = "paidRecommendedCompactionModel", "freeRecommendedCompactionModel" + + # Preference order: + # free tier → free only + # paid tier → paid, then free (if paid field is null) + candidates = [free_key] if free_tier else [paid_key, free_key] + for key in candidates: + name = _extract_model_name(payload.get(key)) + if name: + return name + return None + + +# --------------------------------------------------------------------------- +# Canonical provider list — single source of truth for provider identity. +# Every code path that lists, displays, or iterates providers derives from +# this list: hermes model, /model, /provider, list_authenticated_providers. +# +# Fields: +# slug — internal provider ID (used in config.yaml, --provider flag) +# label — short display name +# tui_desc — longer description for the `hermes model` interactive picker +# --------------------------------------------------------------------------- + +class ProviderEntry(NamedTuple): + slug: str + label: str + tui_desc: str # detailed description for `hermes model` TUI + + +CANONICAL_PROVIDERS: list[ProviderEntry] = [ + ProviderEntry("nous", "Nous Portal", "Nous Portal (Nous Research subscription)"), + ProviderEntry("openrouter", "OpenRouter", "OpenRouter (100+ models, pay-per-use)"), + ProviderEntry("ai-gateway", "Vercel AI Gateway", "Vercel AI Gateway (200+ models, $5 free credit, no markup)"), + ProviderEntry("anthropic", "Anthropic", "Anthropic (Claude models — API key or Claude Code)"), + ProviderEntry("openai-codex", "OpenAI Codex", "OpenAI Codex"), + ProviderEntry("xiaomi", "Xiaomi MiMo", "Xiaomi MiMo (MiMo-V2.5 and V2 models — pro, omni, flash)"), + ProviderEntry("nvidia", "NVIDIA NIM", "NVIDIA NIM (Nemotron models — build.nvidia.com or local NIM)"), + ProviderEntry("qwen-oauth", "Qwen OAuth (Portal)", "Qwen OAuth (reuses local Qwen CLI login)"), + ProviderEntry("copilot", "GitHub Copilot", "GitHub Copilot (uses GITHUB_TOKEN or gh auth token)"), + ProviderEntry("copilot-acp", "GitHub Copilot ACP", "GitHub Copilot ACP (spawns `copilot --acp --stdio`)"), + ProviderEntry("huggingface", "Hugging Face", "Hugging Face Inference Providers (20+ open models)"), + ProviderEntry("gemini", "Google AI Studio", "Google AI Studio (Gemini models — native Gemini API)"), + ProviderEntry("google-gemini-cli", "Google Gemini (OAuth)", "Google Gemini via OAuth + Code Assist (free tier supported; no API key needed)"), + ProviderEntry("deepseek", "DeepSeek", "DeepSeek (DeepSeek-V3, R1, coder — direct API)"), + ProviderEntry("xai", "xAI", "xAI (Grok models — direct API)"), + ProviderEntry("zai", "Z.AI / GLM", "Z.AI / GLM (Zhipu AI direct API)"), + ProviderEntry("kimi-coding", "Kimi / Kimi Coding Plan", "Kimi Coding Plan (api.kimi.com) & Moonshot API"), + ProviderEntry("kimi-coding-cn", "Kimi / Moonshot (China)", "Kimi / Moonshot China (Moonshot CN direct API)"), + ProviderEntry("stepfun", "StepFun Step Plan", "StepFun Step Plan (agent/coding models via Step Plan API)"), + ProviderEntry("minimax", "MiniMax", "MiniMax (global direct API)"), + ProviderEntry("minimax-cn", "MiniMax (China)", "MiniMax China (domestic direct API)"), + ProviderEntry("alibaba", "Alibaba Cloud (DashScope)","Alibaba Cloud / DashScope Coding (Qwen + multi-provider)"), + ProviderEntry("ollama-cloud", "Ollama Cloud", "Ollama Cloud (cloud-hosted open models — ollama.com)"), + ProviderEntry("arcee", "Arcee AI", "Arcee AI (Trinity models — direct API)"), + ProviderEntry("kilocode", "Kilo Code", "Kilo Code (Kilo Gateway API)"), + ProviderEntry("opencode-zen", "OpenCode Zen", "OpenCode Zen (35+ curated models, pay-as-you-go)"), + ProviderEntry("opencode-go", "OpenCode Go", "OpenCode Go (open models, $10/month subscription)"), + ProviderEntry("bedrock", "AWS Bedrock", "AWS Bedrock (Claude, Nova, Llama, DeepSeek — IAM or API key)"), +] + +# Derived dicts — used throughout the codebase +_PROVIDER_LABELS = {p.slug: p.label for p in CANONICAL_PROVIDERS} +_PROVIDER_LABELS["custom"] = "Custom endpoint" # special case: not a named provider + _PROVIDER_ALIASES = { "glm": "zai", @@ -528,6 +744,10 @@ def check_nous_free_tier() -> bool: "moonshot": "kimi-coding", "kimi-cn": "kimi-coding-cn", "moonshot-cn": "kimi-coding-cn", + "step": "stepfun", + "stepfun-coding-plan": "stepfun", + "arcee-ai": "arcee", + "arceeai": "arcee", "minimax-china": "minimax-cn", "minimax_cn": "minimax-cn", "claude": "anthropic", @@ -548,11 +768,26 @@ def check_nous_free_tier() -> bool: "qwen": "alibaba", "alibaba-cloud": "alibaba", "qwen-portal": "qwen-oauth", + "gemini-cli": "google-gemini-cli", + "gemini-oauth": "google-gemini-cli", "hf": "huggingface", "hugging-face": "huggingface", "huggingface-hub": "huggingface", "mimo": "xiaomi", "xiaomi-mimo": "xiaomi", + "aws": "bedrock", + "aws-bedrock": "bedrock", + "amazon-bedrock": "bedrock", + "amazon": "bedrock", + "grok": "xai", + "x-ai": "xai", + "x.ai": "xai", + "nim": "nvidia", + "nvidia-nim": "nvidia", + "build-nvidia": "nvidia", + "nemotron": "nvidia", + "ollama": "custom", # bare "ollama" = local; use "ollama-cloud" for cloud + "ollama_cloud": "ollama-cloud", } @@ -580,6 +815,31 @@ def _openrouter_model_is_free(pricing: Any) -> bool: return False +def _openrouter_model_supports_tools(item: Any) -> bool: + """Return True when the model's ``supported_parameters`` advertise tool calling. + + hermes-agent is tool-calling-first — every provider path assumes the model + can invoke tools. Models that don't advertise ``tools`` in their + ``supported_parameters`` (e.g. image-only or completion-only models) cannot + be driven by the agent loop and would fail at the first tool call. + + **Permissive when the field is missing.** Some OpenRouter-compatible gateways + (Nous Portal, private mirrors, older catalog snapshots) don't populate + ``supported_parameters`` at all. Treat that as "unknown capability → allow" + so the picker doesn't silently empty for those users. Only hide models + whose ``supported_parameters`` is an explicit list that omits ``tools``. + + Ported from Kilo-Org/kilocode#9068. + """ + if not isinstance(item, dict): + return True + params = item.get("supported_parameters") + if not isinstance(params, list): + # Field absent / malformed / None — be permissive. + return True + return "tools" in params + + def fetch_openrouter_models( timeout: float = 8.0, *, @@ -622,6 +882,11 @@ def fetch_openrouter_models( live_item = live_by_id.get(preferred_id) if live_item is None: continue + # Hide models that don't advertise tool-calling support — hermes-agent + # requires it and surfacing them leads to immediate runtime failures + # when the user selects them. Ported from Kilo-Org/kilocode#9068. + if not _openrouter_model_supports_tools(live_item): + continue desc = "free" if _openrouter_model_is_free(live_item.get("pricing")) else "" curated.append((preferred_id, desc)) @@ -639,12 +904,92 @@ def model_ids(*, force_refresh: bool = False) -> list[str]: return [mid for mid, _ in fetch_openrouter_models(force_refresh=force_refresh)] -def menu_labels(*, force_refresh: bool = False) -> list[str]: - """Return display labels like 'anthropic/claude-opus-4.6 (recommended)'.""" - labels = [] - for mid, desc in fetch_openrouter_models(force_refresh=force_refresh): - labels.append(f"{mid} ({desc})" if desc else mid) - return labels +def _ai_gateway_model_is_free(pricing: Any) -> bool: + """Return True if an AI Gateway model has $0 input AND output pricing.""" + if not isinstance(pricing, dict): + return False + try: + return float(pricing.get("input", "0")) == 0 and float(pricing.get("output", "0")) == 0 + except (TypeError, ValueError): + return False + + +def fetch_ai_gateway_models( + timeout: float = 8.0, + *, + force_refresh: bool = False, +) -> list[tuple[str, str]]: + """Return the curated AI Gateway picker list, refreshed from the live catalog when possible.""" + global _ai_gateway_catalog_cache + + if _ai_gateway_catalog_cache is not None and not force_refresh: + return list(_ai_gateway_catalog_cache) + + from hermes_constants import AI_GATEWAY_BASE_URL + + fallback = list(VERCEL_AI_GATEWAY_MODELS) + preferred_ids = [mid for mid, _ in fallback] + + try: + req = urllib.request.Request( + f"{AI_GATEWAY_BASE_URL.rstrip('/')}/models", + headers={"Accept": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=timeout) as resp: + payload = json.loads(resp.read().decode()) + except Exception: + return list(_ai_gateway_catalog_cache or fallback) + + live_items = payload.get("data", []) + if not isinstance(live_items, list): + return list(_ai_gateway_catalog_cache or fallback) + + live_by_id: dict[str, dict[str, Any]] = {} + for item in live_items: + if not isinstance(item, dict): + continue + mid = str(item.get("id") or "").strip() + if not mid: + continue + live_by_id[mid] = item + + curated: list[tuple[str, str]] = [] + for preferred_id in preferred_ids: + live_item = live_by_id.get(preferred_id) + if live_item is None: + continue + desc = "free" if _ai_gateway_model_is_free(live_item.get("pricing")) else "" + curated.append((preferred_id, desc)) + + if not curated: + return list(_ai_gateway_catalog_cache or fallback) + + # If the live catalog offers a free Moonshot model, auto-promote it to + # position #1 as "recommended" — dynamic discovery without a PR. + free_moonshot = next( + ( + mid + for mid, item in live_by_id.items() + if mid.startswith("moonshotai/") + and _ai_gateway_model_is_free(item.get("pricing")) + ), + None, + ) + if free_moonshot: + curated = [(mid, desc) for mid, desc in curated if mid != free_moonshot] + curated.insert(0, (free_moonshot, "recommended")) + else: + first_id, _ = curated[0] + curated[0] = (first_id, "recommended") + + _ai_gateway_catalog_cache = curated + return list(curated) + + +def ai_gateway_model_ids(*, force_refresh: bool = False) -> list[str]: + """Return just the AI Gateway model-id strings.""" + return [mid for mid, _ in fetch_ai_gateway_models(force_refresh=force_refresh)] + @@ -790,6 +1135,56 @@ def fetch_models_with_pricing( return result +def fetch_ai_gateway_pricing( + timeout: float = 8.0, + *, + force_refresh: bool = False, +) -> dict[str, dict[str, str]]: + """Fetch Vercel AI Gateway /v1/models and return hermes-shaped pricing. + + Vercel uses ``input`` / ``output`` field names; hermes's picker expects + ``prompt`` / ``completion``. This translates. Cache read/write field names + already match. + """ + from hermes_constants import AI_GATEWAY_BASE_URL + + cache_key = AI_GATEWAY_BASE_URL.rstrip("/") + if not force_refresh and cache_key in _pricing_cache: + return _pricing_cache[cache_key] + + try: + req = urllib.request.Request( + f"{cache_key}/models", + headers={"Accept": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=timeout) as resp: + payload = json.loads(resp.read().decode()) + except Exception: + _pricing_cache[cache_key] = {} + return {} + + result: dict[str, dict[str, str]] = {} + for item in payload.get("data", []): + if not isinstance(item, dict): + continue + mid = item.get("id") + pricing = item.get("pricing") + if not (mid and isinstance(pricing, dict)): + continue + entry: dict[str, str] = { + "prompt": str(pricing.get("input", "")), + "completion": str(pricing.get("output", "")), + } + if pricing.get("input_cache_read"): + entry["input_cache_read"] = str(pricing["input_cache_read"]) + if pricing.get("input_cache_write"): + entry["input_cache_write"] = str(pricing["input_cache_write"]) + result[mid] = entry + + _pricing_cache[cache_key] = result + return result + + def _resolve_openrouter_api_key() -> str: """Best-effort OpenRouter API key for pricing fetch.""" return os.getenv("OPENROUTER_API_KEY", "").strip() @@ -808,7 +1203,7 @@ def _resolve_nous_pricing_credentials() -> tuple[str, str]: def get_pricing_for_provider(provider: str, *, force_refresh: bool = False) -> dict[str, dict[str, str]]: - """Return live pricing for providers that support it (openrouter, nous).""" + """Return live pricing for providers that support it (openrouter, nous, ai-gateway).""" normalized = normalize_provider(provider) if normalized == "openrouter": return fetch_models_with_pricing( @@ -816,6 +1211,8 @@ def get_pricing_for_provider(provider: str, *, force_refresh: bool = False) -> d base_url="https://openrouter.ai/api", force_refresh=force_refresh, ) + if normalized == "ai-gateway": + return fetch_ai_gateway_pricing(force_refresh=force_refresh) if normalized == "nous": api_key, base_url = _resolve_nous_pricing_credentials() if base_url: @@ -845,23 +1242,20 @@ def list_available_providers() -> list[dict[str, str]]: Each dict has ``id``, ``label``, and ``aliases``. Checks which providers have valid credentials configured. + + Derives the provider list from :data:`CANONICAL_PROVIDERS` (single + source of truth shared with ``hermes model``, ``/model``, etc.). """ - # Canonical providers in display order - _PROVIDER_ORDER = [ - "openrouter", "nous", "openai-codex", "copilot", "copilot-acp", - "gemini", "huggingface", - "zai", "kimi-coding", "kimi-coding-cn", "minimax", "minimax-cn", "kilocode", "anthropic", "alibaba", - "qwen-oauth", "xiaomi", - "opencode-zen", "opencode-go", - "ai-gateway", "deepseek", "custom", - ] + # Derive display order from canonical list + custom + provider_order = [p.slug for p in CANONICAL_PROVIDERS] + ["custom"] + # Build reverse alias map aliases_for: dict[str, list[str]] = {} for alias, canonical in _PROVIDER_ALIASES.items(): aliases_for.setdefault(canonical, []).append(alias) result = [] - for pid in _PROVIDER_ORDER: + for pid in provider_order: label = _PROVIDER_LABELS.get(pid, pid) alias_list = aliases_for.get(pid, []) # Check if this provider has credentials available @@ -999,7 +1393,7 @@ def detect_provider_for_model( return (resolved_provider, default_models[0]) # Aggregators list other providers' models — never auto-switch TO them - _AGGREGATORS = {"nous", "openrouter"} + _AGGREGATORS = {"nous", "openrouter", "ai-gateway", "copilot", "kilocode"} # If the model belongs to the current provider's catalog, don't suggest switching current_models = _PROVIDER_MODELS.get(current_provider, []) @@ -1016,29 +1410,41 @@ def detect_provider_for_model( break if direct_match: - # Check if we have credentials for this provider + # Check if we have credentials for this provider — env vars, + # credential pool, or auth store entries. has_creds = False try: from hermes_cli.auth import PROVIDER_REGISTRY pconfig = PROVIDER_REGISTRY.get(direct_match) if pconfig: - import os for env_var in pconfig.api_key_env_vars: if os.getenv(env_var, "").strip(): has_creds = True break except Exception: pass - - if has_creds: - return (direct_match, name) - - # No direct creds — try to find this model on OpenRouter instead - or_slug = _find_openrouter_slug(name) - if or_slug: - return ("openrouter", or_slug) - # Still return the direct provider — credential resolution will - # give a clear error rather than silently using the wrong provider + # Also check credential pool and auth store — covers OAuth, + # Claude Code tokens, and other non-env-var credentials (#10300). + if not has_creds: + try: + from agent.credential_pool import load_pool + pool = load_pool(direct_match) + if pool.has_credentials(): + has_creds = True + except Exception: + pass + if not has_creds: + try: + from hermes_cli.auth import _load_auth_store + store = _load_auth_store() + if direct_match in store.get("providers", {}) or direct_match in store.get("credential_pool", {}): + has_creds = True + except Exception: + pass + + # Always return the direct provider match. If credentials are + # missing, the client init will give a clear error rather than + # silently routing through the wrong provider (#10300). return (direct_match, name) # --- Step 2: check OpenRouter catalog --- @@ -1187,11 +1593,84 @@ def _resolve_copilot_catalog_api_key() -> str: return "" +# Providers where models.dev is treated as authoritative: curated static +# lists are kept only as an offline fallback and to capture custom additions +# the registry doesn't publish yet. Adding a provider here causes its +# curated list to be merged with fresh models.dev entries (fresh first, any +# curated-only names appended) for both the CLI and the gateway /model picker. +# +# DELIBERATELY EXCLUDED: +# - "openrouter": curated list is already a hand-picked agentic subset of +# OpenRouter's 400+ catalog. Blindly merging would dump everything. +# - "nous": curated list and Portal /models endpoint are the source of +# truth for the subscription tier. +# Also excluded: providers that already have dedicated live-endpoint +# branches below (copilot, anthropic, ai-gateway, ollama-cloud, custom, +# stepfun, openai-codex) — those paths handle freshness themselves. +_MODELS_DEV_PREFERRED: frozenset[str] = frozenset({ + "opencode-go", + "opencode-zen", + "deepseek", + "kilocode", + "fireworks", + "mistral", + "togetherai", + "cohere", + "perplexity", + "groq", + "nvidia", + "huggingface", + "zai", + "gemini", + "google", +}) + + +def _merge_with_models_dev(provider: str, curated: list[str]) -> list[str]: + """Merge curated list with fresh models.dev entries for a preferred provider. + + Returns models.dev entries first (in models.dev order), then any + curated-only entries appended. Preserves case for curated fallbacks + (e.g. ``MiniMax-M2.7``) while trusting models.dev for newer variants. + + If models.dev is unreachable or returns nothing, the curated list is + returned unchanged — this is the offline/CI fallback path. + """ + try: + from agent.models_dev import list_agentic_models + mdev = list_agentic_models(provider) + except Exception: + mdev = [] + + if not mdev: + return list(curated) + + # Case-insensitive dedup while preserving order and curated casing. + seen_lower: set[str] = set() + merged: list[str] = [] + for mid in mdev: + key = str(mid).lower() + if key in seen_lower: + continue + seen_lower.add(key) + merged.append(mid) + for mid in curated: + key = str(mid).lower() + if key in seen_lower: + continue + seen_lower.add(key) + merged.append(mid) + return merged + + def provider_model_ids(provider: Optional[str], *, force_refresh: bool = False) -> list[str]: """Return the best known model catalog for a provider. Tries live API endpoints for providers that support them (Codex, Nous), - falling back to static lists. + falling back to static lists. For providers in ``_MODELS_DEV_PREFERRED`` + (opencode-go/zen, xiaomi, deepseek, smaller inference providers, etc.), + models.dev entries are merged on top of curated so new models released + on the platform appear in ``/model`` without a Hermes release. """ normalized = normalize_provider(provider) if normalized == "openrouter": @@ -1199,7 +1678,19 @@ def provider_model_ids(provider: Optional[str], *, force_refresh: bool = False) if normalized == "openai-codex": from hermes_cli.codex_models import get_codex_model_ids - return get_codex_model_ids() + # Pass the live OAuth access token so the picker matches whatever + # ChatGPT lists for this account right now (new models appear without + # a Hermes release). Falls back to the hardcoded catalog if no token + # or the endpoint is unreachable. + access_token = None + try: + from hermes_cli.auth import resolve_codex_runtime_credentials + + creds = resolve_codex_runtime_credentials(refresh_if_expiring=True) + access_token = creds.get("api_key") + except Exception: + access_token = None + return get_codex_model_ids(access_token=access_token) if normalized in {"copilot", "copilot-acp"}: try: live = _fetch_github_models(_resolve_copilot_catalog_api_key()) @@ -1220,6 +1711,19 @@ def provider_model_ids(provider: Optional[str], *, force_refresh: bool = False) return live except Exception: pass + if normalized == "stepfun": + try: + from hermes_cli.auth import resolve_api_key_provider_credentials + + creds = resolve_api_key_provider_credentials("stepfun") + api_key = str(creds.get("api_key") or "").strip() + base_url = str(creds.get("base_url") or "").strip() + if api_key and base_url: + live = fetch_api_models(api_key, base_url) + if live: + return live + except Exception: + pass if normalized == "anthropic": live = _fetch_anthropic_models() if live: @@ -1228,6 +1732,10 @@ def provider_model_ids(provider: Optional[str], *, force_refresh: bool = False) live = _fetch_ai_gateway_models() if live: return live + if normalized == "ollama-cloud": + live = fetch_ollama_cloud_models(force_refresh=force_refresh) + if live: + return live if normalized == "custom": base_url = _get_custom_base_url() if base_url: @@ -1240,7 +1748,10 @@ def provider_model_ids(provider: Optional[str], *, force_refresh: bool = False) live = fetch_api_models(api_key, base_url) if live: return live - return list(_PROVIDER_MODELS.get(normalized, [])) + curated_static = list(_PROVIDER_MODELS.get(normalized, [])) + if normalized in _MODELS_DEV_PREFERRED: + return _merge_with_models_dev(normalized, curated_static) + return curated_static def _fetch_anthropic_models(timeout: float = 5.0) -> Optional[list[str]]: @@ -1414,6 +1925,19 @@ def _fetch_github_models(api_key: Optional[str] = None, timeout: float = 5.0) -> "anthropic/claude-sonnet-4.6": "claude-sonnet-4.6", "anthropic/claude-sonnet-4.5": "claude-sonnet-4.5", "anthropic/claude-haiku-4.5": "claude-haiku-4.5", + # Dash-notation fallbacks: Hermes' default Claude IDs elsewhere use + # hyphens (anthropic native format), but Copilot's API only accepts + # dot-notation. Accept both so users who configure copilot + a + # default hyphenated Claude model don't hit HTTP 400 + # "model_not_supported". See issue #6879. + "claude-opus-4-6": "claude-opus-4.6", + "claude-sonnet-4-6": "claude-sonnet-4.6", + "claude-sonnet-4-5": "claude-sonnet-4.5", + "claude-haiku-4-5": "claude-haiku-4.5", + "anthropic/claude-opus-4-6": "claude-opus-4.6", + "anthropic/claude-sonnet-4-6": "claude-sonnet-4.6", + "anthropic/claude-sonnet-4-5": "claude-sonnet-4.5", + "anthropic/claude-haiku-4-5": "claude-haiku-4.5", } @@ -1512,6 +2036,11 @@ def copilot_model_api_mode( primary signal. Falls back to the catalog's ``supported_endpoints`` only for models not covered by the pattern check. """ + # Fetch the catalog once so normalize + endpoint check share it + # (avoids two redundant network calls for non-GPT-5 models). + if catalog is None and api_key: + catalog = fetch_github_model_catalog(api_key=api_key) + normalized = normalize_copilot_model_id(model_id, catalog=catalog, api_key=api_key) if not normalized: return "chat_completions" @@ -1521,9 +2050,6 @@ def copilot_model_api_mode( return "codex_responses" # Secondary: check catalog for non-GPT-5 models (Claude via /v1/messages, etc.) - if catalog is None and api_key: - catalog = fetch_github_model_catalog(api_key=api_key) - if catalog: catalog_entry = next((item for item in catalog if item.get("id") == normalized), None) if isinstance(catalog_entry, dict): @@ -1666,7 +2192,7 @@ def probe_api_models( candidates.append((alternate_base, True)) tried: list[str] = [] - headers: dict[str, str] = {} + headers: dict[str, str] = {"User-Agent": _HERMES_USER_AGENT} if api_key: headers["Authorization"] = f"Bearer {api_key}" if normalized.startswith(COPILOT_BASE_URL): @@ -1738,6 +2264,125 @@ def fetch_api_models( return probe_api_models(api_key, base_url, timeout=timeout).get("models") +# --------------------------------------------------------------------------- +# Ollama Cloud — merged model discovery with disk cache +# --------------------------------------------------------------------------- + + + +_OLLAMA_CLOUD_CACHE_TTL = 3600 # 1 hour + + +def _ollama_cloud_cache_path() -> Path: + """Return the path for the Ollama Cloud model cache.""" + from hermes_constants import get_hermes_home + return get_hermes_home() / "ollama_cloud_models_cache.json" + + +def _load_ollama_cloud_cache(*, ignore_ttl: bool = False) -> Optional[dict]: + """Load cached Ollama Cloud models from disk. + + Args: + ignore_ttl: If True, return data even if the TTL has expired (stale fallback). + """ + try: + cache_path = _ollama_cloud_cache_path() + if not cache_path.exists(): + return None + with open(cache_path, encoding="utf-8") as f: + data = json.load(f) + if not isinstance(data, dict): + return None + models = data.get("models") + if not (isinstance(models, list) and models): + return None + if not ignore_ttl: + cached_at = data.get("cached_at", 0) + if (time.time() - cached_at) > _OLLAMA_CLOUD_CACHE_TTL: + return None # stale + return data + except Exception: + pass + return None + + +def _save_ollama_cloud_cache(models: list[str]) -> None: + """Persist the merged Ollama Cloud model list to disk.""" + try: + from utils import atomic_json_write + cache_path = _ollama_cloud_cache_path() + cache_path.parent.mkdir(parents=True, exist_ok=True) + atomic_json_write(cache_path, {"models": models, "cached_at": time.time()}, indent=None) + except Exception: + pass + + +def fetch_ollama_cloud_models( + api_key: Optional[str] = None, + base_url: Optional[str] = None, + *, + force_refresh: bool = False, +) -> list[str]: + """Fetch Ollama Cloud models by merging live API + models.dev, with disk cache. + + Resolution order: + 1. Disk cache (if fresh, < 1 hour, and not force_refresh) + 2. Live ``/v1/models`` endpoint (primary — freshest source) + 3. models.dev registry (secondary — fills gaps for unlisted models) + 4. Merge: live models first, then models.dev additions (deduped) + + Returns a list of model IDs (never None — empty list on total failure). + """ + # 1. Check disk cache + if not force_refresh: + cached = _load_ollama_cloud_cache() + if cached is not None: + return cached["models"] + + # 2. Live API probe + if not api_key: + api_key = os.getenv("OLLAMA_API_KEY", "") + if not base_url: + base_url = os.getenv("OLLAMA_BASE_URL", "") or "https://ollama.com/v1" + + live_models: list[str] = [] + if api_key: + result = fetch_api_models(api_key, base_url, timeout=8.0) + if result: + live_models = result + + # 3. models.dev registry + mdev_models: list[str] = [] + try: + from agent.models_dev import list_agentic_models + mdev_models = list_agentic_models("ollama-cloud") + except Exception: + pass + + # 4. Merge: live first, then models.dev additions (deduped, order-preserving) + if live_models or mdev_models: + seen: set[str] = set() + merged: list[str] = [] + for m in live_models: + if m and m not in seen: + seen.add(m) + merged.append(m) + for m in mdev_models: + if m and m not in seen: + seen.add(m) + merged.append(m) + if merged: + _save_ollama_cloud_cache(merged) + return merged + + # Total failure — return stale cache if available (ignore TTL) + stale = _load_ollama_cloud_cache(ignore_ttl=True) + if stale is not None: + return stale["models"] + + return [] + + def validate_requested_model( model_name: str, provider: Optional[str], @@ -1796,6 +2441,17 @@ def validate_requested_model( "message": None, } + # Auto-correct if the top match is very similar (e.g. typo) + auto = get_close_matches(requested_for_lookup, api_models, n=1, cutoff=0.9) + if auto: + return { + "accepted": True, + "persist": True, + "recognized": True, + "corrected_model": auto[0], + "message": f"Auto-corrected `{requested}` → `{auto[0]}`", + } + suggestions = get_close_matches(requested, api_models, n=3, cutoff=0.5) suggestion_text = "" if suggestions: @@ -1813,8 +2469,8 @@ def validate_requested_model( ) return { - "accepted": True, - "persist": True, + "accepted": False, + "persist": False, "recognized": False, "message": message, } @@ -1827,8 +2483,8 @@ def validate_requested_model( message += f"\n If this server expects `/v1`, try base URL: `{probe.get('suggested_base_url')}`" return { - "accepted": True, - "persist": True, + "accepted": False, + "persist": False, "recognized": False, "message": message, } @@ -1847,18 +2503,72 @@ def validate_requested_model( "recognized": True, "message": None, } + # Auto-correct if the top match is very similar (e.g. typo) + auto = get_close_matches(requested_for_lookup, codex_models, n=1, cutoff=0.9) + if auto: + return { + "accepted": True, + "persist": True, + "recognized": True, + "corrected_model": auto[0], + "message": f"Auto-corrected `{requested}` → `{auto[0]}`", + } suggestions = get_close_matches(requested_for_lookup, codex_models, n=3, cutoff=0.5) suggestion_text = "" if suggestions: suggestion_text = "\n Similar models: " + ", ".join(f"`{s}`" for s in suggestions) + return { + "accepted": False, + "persist": False, + "recognized": False, + "message": ( + f"Model `{requested}` was not found in the OpenAI Codex model listing." + f"{suggestion_text}" + ), + } + + # MiniMax providers don't expose a /models endpoint — validate against + # the static catalog instead, similar to openai-codex. + if normalized in ("minimax", "minimax-cn"): + try: + catalog_models = provider_model_ids(normalized) + except Exception: + catalog_models = [] + if catalog_models: + # Case-insensitive lookup (catalog uses mixed case like MiniMax-M2.7) + catalog_lower = {m.lower(): m for m in catalog_models} + if requested_for_lookup.lower() in catalog_lower: + return { + "accepted": True, + "persist": True, + "recognized": True, + "message": None, + } + # Auto-correct close matches (case-insensitive) + catalog_lower_list = list(catalog_lower.keys()) + auto = get_close_matches(requested_for_lookup.lower(), catalog_lower_list, n=1, cutoff=0.9) + if auto: + corrected = catalog_lower[auto[0]] + return { + "accepted": True, + "persist": True, + "recognized": True, + "corrected_model": corrected, + "message": f"Auto-corrected `{requested}` → `{corrected}`", + } + suggestions = get_close_matches(requested_for_lookup.lower(), catalog_lower_list, n=3, cutoff=0.5) + suggestion_text = "" + if suggestions: + suggestion_text = "\n Similar models: " + ", ".join(f"`{catalog_lower[s]}`" for s in suggestions) return { "accepted": True, "persist": True, "recognized": False, "message": ( - f"Note: `{requested}` was not found in the OpenAI Codex model listing. " - f"It may still work if your account has access to it." + f"Note: `{requested}` was not found in the MiniMax catalog." f"{suggestion_text}" + "\n MiniMax does not expose a /models endpoint, so Hermes cannot verify the model name." + "\n The model may still work if it exists on the server." ), } @@ -1879,31 +2589,135 @@ def validate_requested_model( # the user may have access to models not shown in the public # listing (e.g. Z.AI Pro/Max plans can use glm-5 on coding # endpoints even though it's not in /models). Warn but allow. + + # Auto-correct if the top match is very similar (e.g. typo) + auto = get_close_matches(requested_for_lookup, api_models, n=1, cutoff=0.9) + if auto: + return { + "accepted": True, + "persist": True, + "recognized": True, + "corrected_model": auto[0], + "message": f"Auto-corrected `{requested}` → `{auto[0]}`", + } + suggestions = get_close_matches(requested, api_models, n=3, cutoff=0.5) suggestion_text = "" if suggestions: suggestion_text = "\n Similar models: " + ", ".join(f"`{s}`" for s in suggestions) + return { + "accepted": False, + "persist": False, + "recognized": False, + "message": ( + f"Model `{requested}` was not found in this provider's model listing." + f"{suggestion_text}" + ), + } + + # api_models is None — couldn't reach API. Accept and persist, + # but warn so typos don't silently break things. + + # Bedrock: use our own discovery instead of HTTP /models endpoint. + # Bedrock's bedrock-runtime URL doesn't support /models — it uses the + # AWS SDK control plane (ListFoundationModels + ListInferenceProfiles). + if normalized == "bedrock": + try: + from agent.bedrock_adapter import discover_bedrock_models, resolve_bedrock_region + region = resolve_bedrock_region() + discovered = discover_bedrock_models(region) + discovered_ids = {m["id"] for m in discovered} + if requested in discovered_ids: + return { + "accepted": True, + "persist": True, + "recognized": True, + "message": None, + } + # Not in discovered list — still accept (user may have custom + # inference profiles or cross-account access), but warn. + suggestions = get_close_matches(requested, list(discovered_ids), n=3, cutoff=0.4) + suggestion_text = "" + if suggestions: + suggestion_text = "\n Similar models: " + ", ".join(f"`{s}`" for s in suggestions) return { "accepted": True, "persist": True, "recognized": False, "message": ( - f"Note: `{requested}` was not found in this provider's model listing. " - f"It may still work if your plan supports it." + f"Note: `{requested}` was not found in Bedrock model discovery for {region}. " + f"It may still work with custom inference profiles or cross-account access." f"{suggestion_text}" ), } - - # api_models is None — couldn't reach API. Accept and persist, - # but warn so typos don't silently break things. + except Exception: + pass # Fall through to generic warning + + # Static-catalog fallback: when the /models probe was unreachable, + # validate against the curated list from provider_model_ids() — same + # pattern as the openai-codex and minimax branches above. This fixes + # /model switches in the gateway for providers like opencode-go and + # opencode-zen whose /models endpoint returns 404 against the HTML + # marketing site. Without this block, validate_requested_model would + # reject every model on such providers, switch_model() would return + # success=False, and the gateway would never write to + # _session_model_overrides. provider_label = _PROVIDER_LABELS.get(normalized, normalized) + try: + catalog_models = provider_model_ids(normalized) + except Exception: + catalog_models = [] + + if catalog_models: + catalog_lower = {m.lower(): m for m in catalog_models} + if requested_for_lookup.lower() in catalog_lower: + return { + "accepted": True, + "persist": True, + "recognized": True, + "message": None, + } + catalog_lower_list = list(catalog_lower.keys()) + auto = get_close_matches( + requested_for_lookup.lower(), catalog_lower_list, n=1, cutoff=0.9 + ) + if auto: + corrected = catalog_lower[auto[0]] + return { + "accepted": True, + "persist": True, + "recognized": True, + "corrected_model": corrected, + "message": f"Auto-corrected `{requested}` → `{corrected}`", + } + suggestions = get_close_matches( + requested_for_lookup.lower(), catalog_lower_list, n=3, cutoff=0.5 + ) + suggestion_text = "" + if suggestions: + suggestion_text = "\n Similar models: " + ", ".join( + f"`{catalog_lower[s]}`" for s in suggestions + ) + return { + "accepted": True, + "persist": True, + "recognized": False, + "message": ( + f"Note: `{requested}` was not found in the {provider_label} curated catalog " + f"and the /models endpoint was unreachable.{suggestion_text}" + f"\n The model may still work if it exists on the provider." + ), + } + + # No catalog available — accept with a warning, matching the comment's + # stated intent ("Accept and persist, but warn"). return { "accepted": True, "persist": True, "recognized": False, "message": ( - f"Could not reach the {provider_label} API to validate `{requested}`. " + f"Note: could not reach the {provider_label} API to validate `{requested}`. " f"If the service isn't down, this model may not be valid." ), } diff --git a/hermes_cli/nous_subscription.py b/hermes_cli/nous_subscription.py index f1e4366c1be8..78181aab2b3c 100644 --- a/hermes_cli/nous_subscription.py +++ b/hermes_cli/nous_subscription.py @@ -10,6 +10,7 @@ from hermes_cli.config import get_env_value, load_config from tools.managed_tool_gateway import is_managed_tool_gateway_ready from tools.tool_backend_helpers import ( + fal_key_is_configured, has_direct_modal_credentials, managed_nous_tools_enabled, normalize_browser_cloud_provider, @@ -143,6 +144,7 @@ def _tts_label(current_provider: str) -> str: "openai": "OpenAI TTS", "elevenlabs": "ElevenLabs", "edge": "Edge TTS", + "xai": "xAI TTS", "mistral": "Mistral Voxtral TTS", "neutts": "NeuTTS", } @@ -257,11 +259,20 @@ def get_nous_subscription_features( terminal_cfg.get("modal_mode") ) + # use_gateway flags — when True, the user explicitly opted into the + # Tool Gateway via `hermes model`, so direct credentials should NOT + # prevent gateway routing. + web_use_gateway = bool(web_cfg.get("use_gateway")) + tts_use_gateway = bool(tts_cfg.get("use_gateway")) + browser_use_gateway = bool(browser_cfg.get("use_gateway")) + image_gen_cfg = config.get("image_gen") if isinstance(config.get("image_gen"), dict) else {} + image_use_gateway = bool(image_gen_cfg.get("use_gateway")) + direct_exa = bool(get_env_value("EXA_API_KEY")) direct_firecrawl = bool(get_env_value("FIRECRAWL_API_KEY") or get_env_value("FIRECRAWL_API_URL")) direct_parallel = bool(get_env_value("PARALLEL_API_KEY")) direct_tavily = bool(get_env_value("TAVILY_API_KEY")) - direct_fal = bool(get_env_value("FAL_KEY")) + direct_fal = fal_key_is_configured() direct_openai_tts = bool(resolve_openai_audio_api_key()) direct_elevenlabs = bool(get_env_value("ELEVENLABS_API_KEY")) direct_camofox = bool(get_env_value("CAMOFOX_URL")) @@ -269,6 +280,21 @@ def get_nous_subscription_features( direct_browser_use = bool(get_env_value("BROWSER_USE_API_KEY")) direct_modal = has_direct_modal_credentials() + # When use_gateway is set, suppress direct credentials for managed detection + if web_use_gateway: + direct_firecrawl = False + direct_exa = False + direct_parallel = False + direct_tavily = False + if image_use_gateway: + direct_fal = False + if tts_use_gateway: + direct_openai_tts = False + direct_elevenlabs = False + if browser_use_gateway: + direct_browser_use = False + direct_browserbase = False + managed_web_available = managed_tools_flag and nous_auth_present and is_managed_tool_gateway_ready("firecrawl") managed_image_available = managed_tools_flag and nous_auth_present and is_managed_tool_gateway_ready("fal-queue") managed_tts_available = managed_tools_flag and nous_auth_present and is_managed_tool_gateway_ready("openai-audio") @@ -439,38 +465,8 @@ def get_nous_subscription_features( ) -def get_nous_subscription_explainer_lines() -> list[str]: - if not managed_nous_tools_enabled(): - return [] - - return [ - "Nous subscription enables managed web tools, image generation, OpenAI TTS, and browser automation by default.", - "Those managed tools bill to your Nous subscription. Modal execution is optional and can bill to your subscription too.", - "Change these later with: hermes setup tools, hermes setup terminal, or hermes status.", - ] -def apply_nous_provider_defaults(config: Dict[str, object]) -> set[str]: - """Apply provider-level Nous defaults shared by `hermes setup` and `hermes model`.""" - if not managed_nous_tools_enabled(): - return set() - - features = get_nous_subscription_features(config) - if not features.provider_is_nous: - return set() - - tts_cfg = config.get("tts") - if not isinstance(tts_cfg, dict): - tts_cfg = {} - config["tts"] = tts_cfg - - current_tts = str(tts_cfg.get("provider") or "edge").strip().lower() - if current_tts not in {"", "edge"}: - return set() - - tts_cfg["provider"] = "openai" - return {"tts"} - def apply_nous_managed_defaults( config: Dict[str, object], @@ -525,7 +521,258 @@ def apply_nous_managed_defaults( browser_cfg["cloud_provider"] = "browser-use" changed.add("browser") - if "image_gen" in selected_toolsets and not get_env_value("FAL_KEY"): + if "image_gen" in selected_toolsets and not fal_key_is_configured(): changed.add("image_gen") return changed + + +# --------------------------------------------------------------------------- +# Tool Gateway offer — single Y/n prompt after model selection +# --------------------------------------------------------------------------- + +_GATEWAY_TOOL_LABELS = { + "web": "Web search & extract (Firecrawl)", + "image_gen": "Image generation (FAL)", + "tts": "Text-to-speech (OpenAI TTS)", + "browser": "Browser automation (Browser Use)", +} + + +def _get_gateway_direct_credentials() -> Dict[str, bool]: + """Return a dict of tool_key -> has_direct_credentials.""" + return { + "web": bool( + get_env_value("FIRECRAWL_API_KEY") + or get_env_value("FIRECRAWL_API_URL") + or get_env_value("PARALLEL_API_KEY") + or get_env_value("TAVILY_API_KEY") + or get_env_value("EXA_API_KEY") + ), + "image_gen": fal_key_is_configured(), + "tts": bool( + resolve_openai_audio_api_key() + or get_env_value("ELEVENLABS_API_KEY") + ), + "browser": bool( + get_env_value("BROWSER_USE_API_KEY") + or (get_env_value("BROWSERBASE_API_KEY") and get_env_value("BROWSERBASE_PROJECT_ID")) + ), + } + + +_GATEWAY_DIRECT_LABELS = { + "web": "Firecrawl/Exa/Parallel/Tavily key", + "image_gen": "FAL key", + "tts": "OpenAI/ElevenLabs key", + "browser": "Browser Use/Browserbase key", +} + +_ALL_GATEWAY_KEYS = ("web", "image_gen", "tts", "browser") + + +def get_gateway_eligible_tools( + config: Optional[Dict[str, object]] = None, +) -> tuple[list[str], list[str], list[str]]: + """Return (unconfigured, has_direct, already_managed) tool key lists. + + - unconfigured: tools with no direct credentials (easy switch) + - has_direct: tools where the user has their own API keys + - already_managed: tools already routed through the gateway + + All lists are empty when the user is not a paid Nous subscriber or + is not using Nous as their provider. + """ + if not managed_nous_tools_enabled(): + return [], [], [] + + if config is None: + config = load_config() or {} + + # Quick provider check without the heavy get_nous_subscription_features call + model_cfg = config.get("model") + if not isinstance(model_cfg, dict) or str(model_cfg.get("provider") or "").strip().lower() != "nous": + return [], [], [] + + direct = _get_gateway_direct_credentials() + + # Check which tools the user has explicitly opted into the gateway for. + # This is distinct from managed_by_nous which fires implicitly when + # no direct keys exist — we only skip the prompt for tools where + # use_gateway was explicitly set. + opted_in = { + "web": bool((config.get("web") if isinstance(config.get("web"), dict) else {}).get("use_gateway")), + "image_gen": bool((config.get("image_gen") if isinstance(config.get("image_gen"), dict) else {}).get("use_gateway")), + "tts": bool((config.get("tts") if isinstance(config.get("tts"), dict) else {}).get("use_gateway")), + "browser": bool((config.get("browser") if isinstance(config.get("browser"), dict) else {}).get("use_gateway")), + } + + unconfigured: list[str] = [] + has_direct: list[str] = [] + already_managed: list[str] = [] + for key in _ALL_GATEWAY_KEYS: + if opted_in.get(key): + already_managed.append(key) + elif direct.get(key): + has_direct.append(key) + else: + unconfigured.append(key) + return unconfigured, has_direct, already_managed + + +def apply_gateway_defaults( + config: Dict[str, object], + tool_keys: list[str], +) -> set[str]: + """Apply Tool Gateway config for the given tool keys. + + Sets ``use_gateway: true`` in each tool's config section so the + runtime prefers the gateway even when direct API keys are present. + + Returns the set of tools that were actually changed. + """ + changed: set[str] = set() + + web_cfg = config.get("web") + if not isinstance(web_cfg, dict): + web_cfg = {} + config["web"] = web_cfg + + tts_cfg = config.get("tts") + if not isinstance(tts_cfg, dict): + tts_cfg = {} + config["tts"] = tts_cfg + + browser_cfg = config.get("browser") + if not isinstance(browser_cfg, dict): + browser_cfg = {} + config["browser"] = browser_cfg + + if "web" in tool_keys: + web_cfg["backend"] = "firecrawl" + web_cfg["use_gateway"] = True + changed.add("web") + + if "tts" in tool_keys: + tts_cfg["provider"] = "openai" + tts_cfg["use_gateway"] = True + changed.add("tts") + + if "browser" in tool_keys: + browser_cfg["cloud_provider"] = "browser-use" + browser_cfg["use_gateway"] = True + changed.add("browser") + + if "image_gen" in tool_keys: + image_cfg = config.get("image_gen") + if not isinstance(image_cfg, dict): + image_cfg = {} + config["image_gen"] = image_cfg + image_cfg["use_gateway"] = True + changed.add("image_gen") + + return changed + + +def prompt_enable_tool_gateway(config: Dict[str, object]) -> set[str]: + """If eligible tools exist, prompt the user to enable the Tool Gateway. + + Uses prompt_choice() with a description parameter so the curses TUI + shows the tool context alongside the choices. + + Returns the set of tools that were enabled, or empty set if the user + declined or no tools were eligible. + """ + unconfigured, has_direct, already_managed = get_gateway_eligible_tools(config) + if not unconfigured and not has_direct: + return set() + + try: + from hermes_cli.setup import prompt_choice + except Exception: + return set() + + # Build description lines showing full status of all gateway tools + desc_parts: list[str] = [ + "", + " The Tool Gateway gives you access to web search, image generation,", + " text-to-speech, and browser automation through your Nous subscription.", + " No need to sign up for separate API keys — just pick the tools you want.", + "", + ] + if already_managed: + for k in already_managed: + desc_parts.append(f" ✓ {_GATEWAY_TOOL_LABELS[k]} — using Tool Gateway") + if unconfigured: + for k in unconfigured: + desc_parts.append(f" ○ {_GATEWAY_TOOL_LABELS[k]} — not configured") + if has_direct: + for k in has_direct: + desc_parts.append(f" ○ {_GATEWAY_TOOL_LABELS[k]} — using {_GATEWAY_DIRECT_LABELS[k]}") + + # Build short choice labels — detail is in the description above + choices: list[str] = [] + choice_keys: list[str] = [] # maps choice index -> action + + if unconfigured and has_direct: + choices.append("Enable for all tools (existing keys kept, not used)") + choice_keys.append("all") + + choices.append("Enable only for tools without existing keys") + choice_keys.append("unconfigured") + + choices.append("Skip") + choice_keys.append("skip") + + elif unconfigured: + choices.append("Enable Tool Gateway") + choice_keys.append("unconfigured") + + choices.append("Skip") + choice_keys.append("skip") + + else: + choices.append("Enable Tool Gateway (existing keys kept, not used)") + choice_keys.append("all") + + choices.append("Skip") + choice_keys.append("skip") + + description = "\n".join(desc_parts) if desc_parts else None + # Default to "Enable" when user has no direct keys (new user), + # default to "Skip" when they have existing keys to preserve. + default_idx = 0 if not has_direct else len(choices) - 1 + + try: + idx = prompt_choice( + "Your Nous subscription includes the Tool Gateway.", + choices, + default_idx, + description=description, + ) + except (KeyboardInterrupt, EOFError, OSError, SystemExit): + return set() + + action = choice_keys[idx] + if action == "skip": + return set() + + if action == "all": + # Apply to switchable tools + ensure already-managed tools also + # have use_gateway persisted in config for consistency. + to_apply = list(_ALL_GATEWAY_KEYS) + else: + to_apply = unconfigured + + changed = apply_gateway_defaults(config, to_apply) + if changed: + from hermes_cli.config import save_config + save_config(config) + # Only report the tools that actually switched (not already-managed ones) + newly_switched = changed - set(already_managed) + for key in sorted(newly_switched): + label = _GATEWAY_TOOL_LABELS.get(key, key) + print(f" ✓ {label}: enabled via Nous subscription") + if already_managed and not newly_switched: + print(" (all tools already using Tool Gateway)") + return changed diff --git a/hermes_cli/pairing.py b/hermes_cli/pairing.py index 7e04da90237b..887b7e49ffcd 100644 --- a/hermes_cli/pairing.py +++ b/hermes_cli/pairing.py @@ -44,7 +44,7 @@ def _cmd_list(store): for p in pending: print( f" {p['platform']:<12} {p['code']:<10} {p['user_id']:<20} " - f"{p.get('user_name', ''):<20} {p['age_minutes']}m ago" + f"{(p.get('user_name') or ''):<20} {p['age_minutes']}m ago" ) else: print("\n No pending pairing requests.") @@ -54,7 +54,7 @@ def _cmd_list(store): print(f" {'Platform':<12} {'User ID':<20} {'Name':<20}") print(f" {'--------':<12} {'-------':<20} {'----':<20}") for a in approved: - print(f" {a['platform']:<12} {a['user_id']:<20} {a.get('user_name', ''):<20}") + print(f" {a['platform']:<12} {a['user_id']:<20} {(a.get('user_name') or ''):<20}") else: print("\n No approved users.") @@ -69,7 +69,7 @@ def _cmd_approve(store, platform: str, code: str): result = store.approve_code(platform, code) if result: uid = result["user_id"] - name = result.get("user_name", "") + name = result.get("user_name") or "" display = f"{name} ({uid})" if name else uid print(f"\n Approved! User {display} on {platform} can now use the bot~") print(" They'll be recognized automatically on their next message.\n") diff --git a/hermes_cli/platforms.py b/hermes_cli/platforms.py index df47ed095d57..1fc3a3a85011 100644 --- a/hermes_cli/platforms.py +++ b/hermes_cli/platforms.py @@ -35,6 +35,7 @@ class PlatformInfo(NamedTuple): ("wecom", PlatformInfo(label="💬 WeCom", default_toolset="hermes-wecom")), ("wecom_callback", PlatformInfo(label="💬 WeCom Callback", default_toolset="hermes-wecom-callback")), ("weixin", PlatformInfo(label="💬 Weixin", default_toolset="hermes-weixin")), + ("qqbot", PlatformInfo(label="💬 QQBot", default_toolset="hermes-qqbot")), ("webhook", PlatformInfo(label="🔗 Webhook", default_toolset="hermes-webhook")), ("api_server", PlatformInfo(label="🌐 API Server", default_toolset="hermes-api-server")), ]) diff --git a/hermes_cli/plugins.py b/hermes_cli/plugins.py index 94ec20836d7a..28cb3b1b54a5 100644 --- a/hermes_cli/plugins.py +++ b/hermes_cli/plugins.py @@ -2,14 +2,20 @@ Hermes Plugin System ==================== -Discovers, loads, and manages plugins from three sources: +Discovers, loads, and manages plugins from four sources: -1. **User plugins** – ``~/.hermes/plugins//`` -2. **Project plugins** – ``./.hermes/plugins//`` (opt-in via +1. **Bundled plugins** – ``/plugins//`` (shipped with hermes-agent; + ``memory/`` and ``context_engine/`` subdirs are excluded — they have their + own discovery paths) +2. **User plugins** – ``~/.hermes/plugins//`` +3. **Project plugins** – ``./.hermes/plugins//`` (opt-in via ``HERMES_ENABLE_PROJECT_PLUGINS``) -3. **Pip plugins** – packages that expose the ``hermes_agent.plugins`` +4. **Pip plugins** – packages that expose the ``hermes_agent.plugins`` entry-point group. +Later sources override earlier ones on name collision, so a user or project +plugin with the same name as a bundled plugin replaces it. + Each directory plugin must contain a ``plugin.yaml`` manifest **and** an ``__init__.py`` with a ``register(ctx)`` function. @@ -31,7 +37,6 @@ import importlib.metadata import importlib.util import logging -import os import sys import types from dataclasses import dataclass, field @@ -55,6 +60,8 @@ VALID_HOOKS: Set[str] = { "pre_tool_call", "post_tool_call", + "transform_terminal_output", + "transform_tool_result", "pre_llm_call", "post_llm_call", "pre_api_request", @@ -63,6 +70,7 @@ "on_session_end", "on_session_finalize", "on_session_reset", + "subagent_stop", } ENTRY_POINTS_GROUP = "hermes_agent.plugins" @@ -76,7 +84,12 @@ def _env_enabled(name: str) -> bool: def _get_disabled_plugins() -> set: - """Read the disabled plugins list from config.yaml.""" + """Read the disabled plugins list from config.yaml. + + Kept for backward compat and explicit deny-list semantics. A plugin + name in this set will never load, even if it appears in + ``plugins.enabled``. + """ try: from hermes_cli.config import load_config config = load_config() @@ -86,10 +99,43 @@ def _get_disabled_plugins() -> set: return set() +def _get_enabled_plugins() -> Optional[set]: + """Read the enabled-plugins allow-list from config.yaml. + + Plugins are opt-in by default — only plugins whose name appears in + this set are loaded. Returns: + + * ``None`` — the key is missing or malformed. Callers should treat + this as "nothing enabled yet" (the opt-in default); the first + ``migrate_config`` run populates the key with a grandfathered set + of currently-installed user plugins so existing setups don't + break on upgrade. + * ``set()`` — an empty list was explicitly set; nothing loads. + * ``set(...)`` — the concrete allow-list. + """ + try: + from hermes_cli.config import load_config + config = load_config() + plugins_cfg = config.get("plugins") + if not isinstance(plugins_cfg, dict): + return None + if "enabled" not in plugins_cfg: + return None + enabled = plugins_cfg.get("enabled") + if not isinstance(enabled, list): + return None + return set(enabled) + except Exception: + return None + + # --------------------------------------------------------------------------- # Data classes # --------------------------------------------------------------------------- +_VALID_PLUGIN_KINDS: Set[str] = {"standalone", "backend", "exclusive"} + + @dataclass class PluginManifest: """Parsed representation of a plugin.yaml manifest.""" @@ -103,6 +149,23 @@ class PluginManifest: provides_hooks: List[str] = field(default_factory=list) source: str = "" # "user", "project", or "entrypoint" path: Optional[str] = None + # Plugin kind — see plugins.py module docstring for semantics. + # ``standalone`` (default): hooks/tools of its own; opt-in via + # ``plugins.enabled``. + # ``backend``: pluggable backend for an existing core tool (e.g. + # image_gen). Built-in (bundled) backends auto-load; + # user-installed still gated by ``plugins.enabled``. + # ``exclusive``: category with exactly one active provider (memory). + # Selection via ``.provider`` config key; the + # category's own discovery system handles loading and the + # general scanner skips these. + kind: str = "standalone" + # Registry key — path-derived, used by ``plugins.enabled``/``disabled`` + # lookups and by ``hermes plugins list``. For a flat plugin at + # ``plugins/disk-cleanup/`` the key is ``disk-cleanup``; for a nested + # category plugin at ``plugins/image_gen/openai/`` the key is + # ``image_gen/openai``. When empty, falls back to ``name``. + key: str = "" @dataclass @@ -113,6 +176,7 @@ class LoadedPlugin: module: Optional[types.ModuleType] = None tools_registered: List[str] = field(default_factory=list) hooks_registered: List[str] = field(default_factory=list) + commands_registered: List[str] = field(default_factory=list) enabled: bool = False error: Optional[str] = None @@ -212,6 +276,93 @@ def register_cli_command( } logger.debug("Plugin %s registered CLI command: %s", self.manifest.name, name) + # -- slash command registration ------------------------------------------- + + def register_command( + self, + name: str, + handler: Callable, + description: str = "", + args_hint: str = "", + ) -> None: + """Register a slash command (e.g. ``/lcm``) available in CLI and gateway sessions. + + The handler signature is ``fn(raw_args: str) -> str | None``. + It may also be an async callable — the gateway dispatch handles both. + + Unlike ``register_cli_command()`` (which creates ``hermes `` + terminal commands), this registers in-session slash commands that users + invoke during a conversation. + + ``args_hint`` is an optional short string (e.g. ``""`` or + ``"dias:7 formato:json"``) used by gateway adapters to surface the + command with an argument field — for example Discord's native slash + command picker. Plugin commands without ``args_hint`` register as + parameterless in Discord and still accept trailing text when invoked + as free-form chat. + + Names conflicting with built-in commands are rejected with a warning. + """ + clean = name.lower().strip().lstrip("/").replace(" ", "-") + if not clean: + logger.warning( + "Plugin '%s' tried to register a command with an empty name.", + self.manifest.name, + ) + return + + # Reject if it conflicts with a built-in command + try: + from hermes_cli.commands import resolve_command + if resolve_command(clean) is not None: + logger.warning( + "Plugin '%s' tried to register command '/%s' which conflicts " + "with a built-in command. Skipping.", + self.manifest.name, clean, + ) + return + except Exception: + pass # If commands module isn't available, skip the check + + self._manager._plugin_commands[clean] = { + "handler": handler, + "description": description or "Plugin command", + "plugin": self.manifest.name, + "args_hint": (args_hint or "").strip(), + } + logger.debug("Plugin %s registered command: /%s", self.manifest.name, clean) + + # -- tool dispatch ------------------------------------------------------- + + def dispatch_tool(self, tool_name: str, args: dict, **kwargs) -> str: + """Dispatch a tool call through the registry, with parent agent context. + + This is the public interface for plugin slash commands that need to call + tools like ``delegate_task`` without reaching into framework internals. + The parent agent (if available) is resolved automatically — plugins never + need to access the agent directly. + + Args: + tool_name: Registry name of the tool (e.g. ``"delegate_task"``). + args: Tool arguments dict (same as what the model would pass). + **kwargs: Extra keyword args forwarded to the registry dispatch. + + Returns: + JSON string from the tool handler (same format as model tool calls). + """ + from tools.registry import registry + + # Wire up parent agent context when available (CLI mode). + # In gateway mode _cli_ref is None — tools degrade gracefully + # (workspace hints fall back to TERMINAL_CWD, no spinner). + if "parent_agent" not in kwargs: + cli = self._manager._cli_ref + agent = getattr(cli, "agent", None) if cli else None + if agent is not None: + kwargs["parent_agent"] = agent + + return registry.dispatch(tool_name, args, **kwargs) + # -- context engine registration ----------------------------------------- def register_context_engine(self, engine) -> None: @@ -244,6 +395,33 @@ def register_context_engine(self, engine) -> None: self.manifest.name, engine.name, ) + # -- image gen provider registration ------------------------------------ + + def register_image_gen_provider(self, provider) -> None: + """Register an image generation backend. + + ``provider`` must be an instance of + :class:`agent.image_gen_provider.ImageGenProvider`. The + ``provider.name`` attribute is what ``image_gen.provider`` in + ``config.yaml`` matches against when routing ``image_generate`` + tool calls. + """ + from agent.image_gen_provider import ImageGenProvider + from agent.image_gen_registry import register_provider + + if not isinstance(provider, ImageGenProvider): + logger.warning( + "Plugin '%s' tried to register an image_gen provider that does " + "not inherit from ImageGenProvider. Ignoring.", + self.manifest.name, + ) + return + register_provider(provider) + logger.info( + "Plugin '%s' registered image_gen provider: %s", + self.manifest.name, provider.name, + ) + # -- hook registration -------------------------------------------------- def register_hook(self, hook_name: str, callback: Callable) -> None: @@ -263,6 +441,53 @@ def register_hook(self, hook_name: str, callback: Callable) -> None: self._manager._hooks.setdefault(hook_name, []).append(callback) logger.debug("Plugin %s registered hook: %s", self.manifest.name, hook_name) + # -- skill registration ------------------------------------------------- + + def register_skill( + self, + name: str, + path: Path, + description: str = "", + ) -> None: + """Register a read-only skill provided by this plugin. + + The skill becomes resolvable as ``':'`` via + ``skill_view()``. It does **not** enter the flat + ``~/.hermes/skills/`` tree and is **not** listed in the system + prompt's ```` index — plugin skills are + opt-in explicit loads only. + + Raises: + ValueError: if *name* contains ``':'`` or invalid characters. + FileNotFoundError: if *path* does not exist. + """ + from agent.skill_utils import _NAMESPACE_RE + + if ":" in name: + raise ValueError( + f"Skill name '{name}' must not contain ':' " + f"(the namespace is derived from the plugin name " + f"'{self.manifest.name}' automatically)." + ) + if not name or not _NAMESPACE_RE.match(name): + raise ValueError( + f"Invalid skill name '{name}'. Must match [a-zA-Z0-9_-]+." + ) + if not path.exists(): + raise FileNotFoundError(f"SKILL.md not found at {path}") + + qualified = f"{self.manifest.name}:{name}" + self._manager._plugin_skills[qualified] = { + "path": path, + "plugin": self.manifest.name, + "bare_name": name, + "description": description, + } + logger.debug( + "Plugin %s registered skill: %s", + self.manifest.name, qualified, + ) + # --------------------------------------------------------------------------- # PluginManager @@ -277,41 +502,134 @@ def __init__(self) -> None: self._plugin_tool_names: Set[str] = set() self._cli_commands: Dict[str, dict] = {} self._context_engine = None # Set by a plugin via register_context_engine() + self._plugin_commands: Dict[str, dict] = {} # Slash commands registered by plugins self._discovered: bool = False self._cli_ref = None # Set by CLI after plugin discovery + # Plugin skill registry: qualified name → metadata dict. + self._plugin_skills: Dict[str, Dict[str, Any]] = {} # ----------------------------------------------------------------------- # Public # ----------------------------------------------------------------------- - def discover_and_load(self) -> None: - """Scan all plugin sources and load each plugin found.""" - if self._discovered: + def discover_and_load(self, force: bool = False) -> None: + """Scan all plugin sources and load each plugin found. + + When ``force`` is true, clear cached discovery state first so config + changes or newly-added bundled backends become visible in long-lived + sessions without requiring a full agent restart. + """ + if self._discovered and not force: return + if force: + self._plugins.clear() + self._hooks.clear() + self._plugin_tool_names.clear() + self._cli_commands.clear() + self._plugin_commands.clear() + self._plugin_skills.clear() + self._context_engine = None self._discovered = True manifests: List[PluginManifest] = [] - # 1. User plugins (~/.hermes/plugins/) + # 1. Bundled plugins (/plugins//) + # + # Repo-shipped plugins live next to hermes_cli/. Two layouts are + # supported (see ``_scan_directory`` for details): + # + # - flat: ``plugins/disk-cleanup/plugin.yaml`` (standalone) + # - category: ``plugins/image_gen/openai/plugin.yaml`` (backend) + # + # ``memory/`` and ``context_engine/`` are skipped at the top level — + # they have their own discovery systems. Porting those to the + # category-namespace ``kind: exclusive`` model is a future PR. + repo_plugins = Path(__file__).resolve().parent.parent / "plugins" + manifests.extend( + self._scan_directory( + repo_plugins, + source="bundled", + skip_names={"memory", "context_engine"}, + ) + ) + + # 2. User plugins (~/.hermes/plugins/) user_dir = get_hermes_home() / "plugins" manifests.extend(self._scan_directory(user_dir, source="user")) - # 2. Project plugins (./.hermes/plugins/) + # 3. Project plugins (./.hermes/plugins/) if _env_enabled("HERMES_ENABLE_PROJECT_PLUGINS"): project_dir = Path.cwd() / ".hermes" / "plugins" manifests.extend(self._scan_directory(project_dir, source="project")) - # 3. Pip / entry-point plugins + # 4. Pip / entry-point plugins manifests.extend(self._scan_entry_points()) - # Load each manifest (skip user-disabled plugins) + # Load each manifest (skip user-disabled plugins). + # Later sources override earlier ones on key collision — user + # plugins take precedence over bundled, project plugins take + # precedence over user. Dedup here so we only load the final + # winner. Keys are path-derived (``image_gen/openai``, + # ``disk-cleanup``) so ``tts/openai`` and ``image_gen/openai`` + # don't collide even when both manifests say ``name: openai``. disabled = _get_disabled_plugins() + enabled = _get_enabled_plugins() # None = opt-in default (nothing enabled) + winners: Dict[str, PluginManifest] = {} for manifest in manifests: - if manifest.name in disabled: + winners[manifest.key or manifest.name] = manifest + for manifest in winners.values(): + lookup_key = manifest.key or manifest.name + + # Explicit disable always wins (matches on key or on legacy + # bare name for back-compat with existing user configs). + if lookup_key in disabled or manifest.name in disabled: loaded = LoadedPlugin(manifest=manifest, enabled=False) loaded.error = "disabled via config" - self._plugins[manifest.name] = loaded - logger.debug("Skipping disabled plugin '%s'", manifest.name) + self._plugins[lookup_key] = loaded + logger.debug("Skipping disabled plugin '%s'", lookup_key) + continue + + # Exclusive plugins (memory providers) have their own + # discovery/activation path. The general loader records the + # manifest for introspection but does not load the module. + if manifest.kind == "exclusive": + loaded = LoadedPlugin(manifest=manifest, enabled=False) + loaded.error = ( + "exclusive plugin — activate via .provider config" + ) + self._plugins[lookup_key] = loaded + logger.debug( + "Skipping '%s' (exclusive, handled by category discovery)", + lookup_key, + ) + continue + + # Built-in backends auto-load — they ship with hermes and must + # just work. Selection among them (e.g. which image_gen backend + # services calls) is driven by ``.provider`` config, + # enforced by the tool wrapper. + if manifest.kind == "backend" and manifest.source == "bundled": + self._load_plugin(manifest) + continue + + # Everything else (standalone, user-installed backends, + # entry-point plugins) is opt-in via plugins.enabled. + # Accept both the path-derived key and the legacy bare name + # so existing configs keep working. + is_enabled = ( + enabled is not None + and (lookup_key in enabled or manifest.name in enabled) + ) + if not is_enabled: + loaded = LoadedPlugin(manifest=manifest, enabled=False) + loaded.error = ( + "not enabled in config (run `hermes plugins enable {}` to activate)" + .format(lookup_key) + ) + self._plugins[lookup_key] = loaded + logger.debug( + "Skipping '%s' (not in plugins.enabled)", lookup_key + ) continue self._load_plugin(manifest) @@ -326,8 +644,46 @@ def discover_and_load(self) -> None: # Directory scanning # ----------------------------------------------------------------------- - def _scan_directory(self, path: Path, source: str) -> List[PluginManifest]: - """Read ``plugin.yaml`` manifests from subdirectories of *path*.""" + def _scan_directory( + self, + path: Path, + source: str, + skip_names: Optional[Set[str]] = None, + ) -> List[PluginManifest]: + """Read ``plugin.yaml`` manifests from subdirectories of *path*. + + Supports two layouts, mixed freely: + + * **Flat** — ``//plugin.yaml``. Key is + ```` (e.g. ``disk-cleanup``). + * **Category** — ``///plugin.yaml``, + where the ```` directory itself has no ``plugin.yaml``. + Key is ``/`` (e.g. ``image_gen/openai``). + Depth is capped at two segments. + + *skip_names* is an optional allow-list of names to ignore at the + top level (kept for back-compat; the current call sites no longer + pass it now that categories are first-class). + """ + return self._scan_directory_level( + path, source, skip_names=skip_names, prefix="", depth=0 + ) + + def _scan_directory_level( + self, + path: Path, + source: str, + *, + skip_names: Optional[Set[str]], + prefix: str, + depth: int, + ) -> List[PluginManifest]: + """Recursive implementation of :meth:`_scan_directory`. + + ``prefix`` is the category path already accumulated ("" at root, + "image_gen" one level in). ``depth`` is the recursion depth; we + cap at 2 so ``/a/b/c/`` is ignored. + """ manifests: List[PluginManifest] = [] if not path.is_dir(): return manifests @@ -335,35 +691,112 @@ def _scan_directory(self, path: Path, source: str) -> List[PluginManifest]: for child in sorted(path.iterdir()): if not child.is_dir(): continue + if depth == 0 and skip_names and child.name in skip_names: + continue manifest_file = child / "plugin.yaml" if not manifest_file.exists(): manifest_file = child / "plugin.yml" - if not manifest_file.exists(): - logger.debug("Skipping %s (no plugin.yaml)", child) + + if manifest_file.exists(): + manifest = self._parse_manifest( + manifest_file, child, source, prefix + ) + if manifest is not None: + manifests.append(manifest) continue - try: - if yaml is None: - logger.warning("PyYAML not installed – cannot load %s", manifest_file) - continue - data = yaml.safe_load(manifest_file.read_text()) or {} - manifest = PluginManifest( - name=data.get("name", child.name), - version=str(data.get("version", "")), - description=data.get("description", ""), - author=data.get("author", ""), - requires_env=data.get("requires_env", []), - provides_tools=data.get("provides_tools", []), - provides_hooks=data.get("provides_hooks", []), - source=source, - path=str(child), + # No manifest at this level. If we're still within the depth + # cap, treat this directory as a category namespace and recurse + # one level in looking for children with manifests. + if depth >= 1: + logger.debug("Skipping %s (no plugin.yaml, depth cap reached)", child) + continue + + sub_prefix = f"{prefix}/{child.name}" if prefix else child.name + manifests.extend( + self._scan_directory_level( + child, + source, + skip_names=None, + prefix=sub_prefix, + depth=depth + 1, ) - manifests.append(manifest) - except Exception as exc: - logger.warning("Failed to parse %s: %s", manifest_file, exc) + ) return manifests + def _parse_manifest( + self, + manifest_file: Path, + plugin_dir: Path, + source: str, + prefix: str, + ) -> Optional[PluginManifest]: + """Parse a single ``plugin.yaml`` into a :class:`PluginManifest`. + + Returns ``None`` on parse failure (logs a warning). + """ + try: + if yaml is None: + logger.warning("PyYAML not installed – cannot load %s", manifest_file) + return None + data = yaml.safe_load(manifest_file.read_text()) or {} + + name = data.get("name", plugin_dir.name) + key = f"{prefix}/{plugin_dir.name}" if prefix else name + + raw_kind = data.get("kind", "standalone") + if not isinstance(raw_kind, str): + raw_kind = "standalone" + kind = raw_kind.strip().lower() + if kind not in _VALID_PLUGIN_KINDS: + logger.warning( + "Plugin %s: unknown kind '%s' (valid: %s); treating as 'standalone'", + key, raw_kind, ", ".join(sorted(_VALID_PLUGIN_KINDS)), + ) + kind = "standalone" + + # Auto-coerce user-installed memory providers to kind="exclusive" + # so they're routed to plugins/memory discovery instead of being + # loaded by the general PluginManager (which has no + # register_memory_provider on PluginContext). Mirrors the + # heuristic in plugins/memory/__init__.py:_is_memory_provider_dir. + # Bundled memory providers are already skipped via skip_names. + if kind == "standalone" and "kind" not in data: + init_file = plugin_dir / "__init__.py" + if init_file.exists(): + try: + source_text = init_file.read_text(errors="replace")[:8192] + if ( + "register_memory_provider" in source_text + or "MemoryProvider" in source_text + ): + kind = "exclusive" + logger.debug( + "Plugin %s: detected memory provider, " + "treating as kind='exclusive'", + key, + ) + except Exception: + pass + + return PluginManifest( + name=name, + version=str(data.get("version", "")), + description=data.get("description", ""), + author=data.get("author", ""), + requires_env=data.get("requires_env", []), + provides_tools=data.get("provides_tools", []), + provides_hooks=data.get("provides_hooks", []), + source=source, + path=str(plugin_dir), + kind=kind, + key=key, + ) + except Exception as exc: + logger.warning("Failed to parse %s: %s", manifest_file, exc) + return None + # ----------------------------------------------------------------------- # Entry-point scanning # ----------------------------------------------------------------------- @@ -386,6 +819,7 @@ def _scan_entry_points(self) -> List[PluginManifest]: name=ep.name, source="entrypoint", path=ep.value, + key=ep.name, ) manifests.append(manifest) except Exception as exc: @@ -402,7 +836,7 @@ def _load_plugin(self, manifest: PluginManifest) -> None: loaded = LoadedPlugin(manifest=manifest) try: - if manifest.source in ("user", "project"): + if manifest.source in ("user", "project", "bundled"): module = self._load_directory_module(manifest) else: module = self._load_entrypoint_module(manifest) @@ -437,16 +871,26 @@ def _load_plugin(self, manifest: PluginManifest) -> None: for h in p.hooks_registered } ) + loaded.commands_registered = [ + c for c in self._plugin_commands + if self._plugin_commands[c].get("plugin") == manifest.name + ] loaded.enabled = True except Exception as exc: loaded.error = str(exc) logger.warning("Failed to load plugin '%s': %s", manifest.name, exc) - self._plugins[manifest.name] = loaded + self._plugins[manifest.key or manifest.name] = loaded def _load_directory_module(self, manifest: PluginManifest) -> types.ModuleType: - """Import a directory-based plugin as ``hermes_plugins.``.""" + """Import a directory-based plugin as ``hermes_plugins.``. + + The module slug is derived from ``manifest.key`` so category-namespaced + plugins (``image_gen/openai``) import as + ``hermes_plugins.image_gen__openai`` without colliding with any + future ``tts/openai``. + """ plugin_dir = Path(manifest.path) # type: ignore[arg-type] init_file = plugin_dir / "__init__.py" if not init_file.exists(): @@ -459,7 +903,9 @@ def _load_directory_module(self, manifest: PluginManifest) -> types.ModuleType: ns_pkg.__package__ = _NS_PARENT sys.modules[_NS_PARENT] = ns_pkg - module_name = f"{_NS_PARENT}.{manifest.name.replace('-', '_')}" + key = manifest.key or manifest.name + slug = key.replace("/", "__").replace("-", "_") + module_name = f"{_NS_PARENT}.{slug}" spec = importlib.util.spec_from_file_location( module_name, init_file, @@ -540,21 +986,46 @@ def invoke_hook(self, hook_name: str, **kwargs: Any) -> List[Any]: def list_plugins(self) -> List[Dict[str, Any]]: """Return a list of info dicts for all discovered plugins.""" result: List[Dict[str, Any]] = [] - for name, loaded in sorted(self._plugins.items()): + for key, loaded in sorted(self._plugins.items()): result.append( { - "name": name, + "name": loaded.manifest.name, + "key": loaded.manifest.key or loaded.manifest.name, + "kind": loaded.manifest.kind, "version": loaded.manifest.version, "description": loaded.manifest.description, "source": loaded.manifest.source, "enabled": loaded.enabled, "tools": len(loaded.tools_registered), "hooks": len(loaded.hooks_registered), + "commands": len(loaded.commands_registered), "error": loaded.error, } ) return result + # ----------------------------------------------------------------------- + # Plugin skill lookups + # ----------------------------------------------------------------------- + + def find_plugin_skill(self, qualified_name: str) -> Optional[Path]: + """Return the ``Path`` to a plugin skill's SKILL.md, or ``None``.""" + entry = self._plugin_skills.get(qualified_name) + return entry["path"] if entry else None + + def list_plugin_skills(self, plugin_name: str) -> List[str]: + """Return sorted bare names of all skills registered by *plugin_name*.""" + prefix = f"{plugin_name}:" + return sorted( + e["bare_name"] + for qn, e in self._plugin_skills.items() + if qn.startswith(prefix) + ) + + def remove_plugin_skill(self, qualified_name: str) -> None: + """Remove a stale registry entry (silently ignores missing keys).""" + self._plugin_skills.pop(qualified_name, None) + # --------------------------------------------------------------------------- # Module-level singleton & convenience functions @@ -571,9 +1042,13 @@ def get_plugin_manager() -> PluginManager: return _plugin_manager -def discover_plugins() -> None: - """Discover and load all plugins (idempotent).""" - get_plugin_manager().discover_and_load() +def discover_plugins(force: bool = False) -> None: + """Discover and load all plugins. + + Default behavior is idempotent. Pass ``force=True`` to rescan plugin + manifests and reload state in the current process. + """ + get_plugin_manager().discover_and_load(force=force) def invoke_hook(hook_name: str, **kwargs: Any) -> List[Any]: @@ -584,23 +1059,74 @@ def invoke_hook(hook_name: str, **kwargs: Any) -> List[Any]: return get_plugin_manager().invoke_hook(hook_name, **kwargs) -def get_plugin_tool_names() -> Set[str]: - """Return the set of tool names registered by plugins.""" - return get_plugin_manager()._plugin_tool_names + +def get_pre_tool_call_block_message( + tool_name: str, + args: Optional[Dict[str, Any]], + task_id: str = "", + session_id: str = "", + tool_call_id: str = "", +) -> Optional[str]: + """Check ``pre_tool_call`` hooks for a blocking directive. + + Plugins that need to enforce policy (rate limiting, security + restrictions, approval workflows) can return:: + + {"action": "block", "message": "Reason the tool was blocked"} + + from their ``pre_tool_call`` callback. The first valid block + directive wins. Invalid or irrelevant hook return values are + silently ignored so existing observer-only hooks are unaffected. + """ + hook_results = invoke_hook( + "pre_tool_call", + tool_name=tool_name, + args=args if isinstance(args, dict) else {}, + task_id=task_id, + session_id=session_id, + tool_call_id=tool_call_id, + ) + + for result in hook_results: + if not isinstance(result, dict): + continue + if result.get("action") != "block": + continue + message = result.get("message") + if isinstance(message, str) and message: + return message + + return None -def get_plugin_cli_commands() -> Dict[str, dict]: - """Return CLI commands registered by general plugins. +def _ensure_plugins_discovered(force: bool = False) -> PluginManager: + """Return the global manager after ensuring plugin discovery has run. - Returns a dict of ``{name: {help, setup_fn, handler_fn, ...}}`` - suitable for wiring into argparse subparsers. + Pass ``force=True`` to rescan in the current process. """ - return dict(get_plugin_manager()._cli_commands) + manager = get_plugin_manager() + manager.discover_and_load(force=force) + return manager def get_plugin_context_engine(): """Return the plugin-registered context engine, or None.""" - return get_plugin_manager()._context_engine + return _ensure_plugins_discovered()._context_engine + + +def get_plugin_command_handler(name: str) -> Optional[Callable]: + """Return the handler for a plugin-registered slash command, or ``None``.""" + entry = _ensure_plugins_discovered()._plugin_commands.get(name) + return entry["handler"] if entry else None + + +def get_plugin_commands() -> Dict[str, dict]: + """Return the full plugin commands dict (name → {handler, description, plugin}). + + Triggers idempotent plugin discovery so callers can use plugin commands + before any explicit discover_plugins() call. + """ + return _ensure_plugins_discovered()._plugin_commands def get_plugin_toolsets() -> List[tuple]: @@ -622,7 +1148,7 @@ def get_plugin_toolsets() -> List[tuple]: toolset_tools: Dict[str, List[str]] = {} toolset_plugin: Dict[str, LoadedPlugin] = {} for tool_name in manager._plugin_tool_names: - entry = registry._tools.get(tool_name) + entry = registry.get_entry(tool_name) if not entry: continue ts = entry.toolset @@ -631,7 +1157,7 @@ def get_plugin_toolsets() -> List[tuple]: # Map toolsets back to the plugin that registered them for _name, loaded in manager._plugins.items(): for tool_name in loaded.tools_registered: - entry = registry._tools.get(tool_name) + entry = registry.get_entry(tool_name) if entry and entry.toolset in toolset_tools: toolset_plugin.setdefault(entry.toolset, loaded) diff --git a/hermes_cli/plugins_cmd.py b/hermes_cli/plugins_cmd.py index c92d8b0dc6a1..230e13420768 100644 --- a/hermes_cli/plugins_cmd.py +++ b/hermes_cli/plugins_cmd.py @@ -15,6 +15,7 @@ import subprocess import sys from pathlib import Path +from typing import Optional from hermes_constants import get_hermes_home @@ -281,8 +282,16 @@ def _require_installed_plugin(name: str, plugins_dir: Path, console) -> Path: # --------------------------------------------------------------------------- -def cmd_install(identifier: str, force: bool = False) -> None: - """Install a plugin from a Git URL or owner/repo shorthand.""" +def cmd_install( + identifier: str, + force: bool = False, + enable: Optional[bool] = None, +) -> None: + """Install a plugin from a Git URL or owner/repo shorthand. + + After install, prompt "Enable now? [y/N]" unless *enable* is provided + (True = auto-enable without prompting, False = install disabled). + """ import tempfile from rich.console import Console @@ -391,6 +400,40 @@ def cmd_install(identifier: str, force: bool = False) -> None: _display_after_install(target, identifier) + # Determine the canonical plugin name for enable-list bookkeeping. + installed_name = installed_manifest.get("name") or target.name + + # Decide whether to enable: explicit flag > interactive prompt > default off + should_enable = enable + if should_enable is None: + # Interactive prompt unless stdin isn't a TTY (scripted install). + if sys.stdin.isatty() and sys.stdout.isatty(): + try: + answer = input( + f" Enable '{installed_name}' now? [y/N]: " + ).strip().lower() + should_enable = answer in ("y", "yes") + except (EOFError, KeyboardInterrupt): + should_enable = False + else: + should_enable = False + + if should_enable: + enabled = _get_enabled_set() + disabled = _get_disabled_set() + enabled.add(installed_name) + disabled.discard(installed_name) + _save_enabled_set(enabled) + _save_disabled_set(disabled) + console.print( + f"[green]✓[/green] Plugin [bold]{installed_name}[/bold] enabled." + ) + else: + console.print( + f"[dim]Plugin installed but not enabled. " + f"Run `hermes plugins enable {installed_name}` to activate.[/dim]" + ) + console.print("[dim]Restart the gateway for the plugin to take effect:[/dim]") console.print("[dim] hermes gateway restart[/dim]") console.print() @@ -468,7 +511,11 @@ def cmd_remove(name: str) -> None: def _get_disabled_set() -> set: - """Read the disabled plugins set from config.yaml.""" + """Read the disabled plugins set from config.yaml. + + An explicit deny-list. A plugin name here never loads, even if also + listed in ``plugins.enabled``. + """ try: from hermes_cli.config import load_config config = load_config() @@ -488,103 +535,196 @@ def _save_disabled_set(disabled: set) -> None: save_config(config) +def _get_enabled_set() -> set: + """Read the enabled plugins allow-list from config.yaml. + + Plugins are opt-in: only names here are loaded. Returns ``set()`` if + the key is missing (same behaviour as "nothing enabled yet"). + """ + try: + from hermes_cli.config import load_config + config = load_config() + plugins_cfg = config.get("plugins", {}) + if not isinstance(plugins_cfg, dict): + return set() + enabled = plugins_cfg.get("enabled", []) + return set(enabled) if isinstance(enabled, list) else set() + except Exception: + return set() + + +def _save_enabled_set(enabled: set) -> None: + """Write the enabled plugins list to config.yaml.""" + from hermes_cli.config import load_config, save_config + config = load_config() + if "plugins" not in config: + config["plugins"] = {} + config["plugins"]["enabled"] = sorted(enabled) + save_config(config) + + def cmd_enable(name: str) -> None: - """Enable a previously disabled plugin.""" + """Add a plugin to the enabled allow-list (and remove it from disabled).""" from rich.console import Console console = Console() - plugins_dir = _plugins_dir() - - # Verify the plugin exists - target = plugins_dir / name - if not target.is_dir(): - console.print(f"[red]Plugin '{name}' is not installed.[/red]") + # Discover the plugin — check installed (user) AND bundled. + if not _plugin_exists(name): + console.print(f"[red]Plugin '{name}' is not installed or bundled.[/red]") sys.exit(1) + enabled = _get_enabled_set() disabled = _get_disabled_set() - if name not in disabled: + + if name in enabled and name not in disabled: console.print(f"[dim]Plugin '{name}' is already enabled.[/dim]") return + enabled.add(name) disabled.discard(name) + _save_enabled_set(enabled) _save_disabled_set(disabled) - console.print(f"[green]✓[/green] Plugin [bold]{name}[/bold] enabled. Takes effect on next session.") + console.print( + f"[green]✓[/green] Plugin [bold]{name}[/bold] enabled. " + "Takes effect on next session." + ) def cmd_disable(name: str) -> None: - """Disable a plugin without removing it.""" + """Remove a plugin from the enabled allow-list (and add to disabled).""" from rich.console import Console console = Console() - plugins_dir = _plugins_dir() - - # Verify the plugin exists - target = plugins_dir / name - if not target.is_dir(): - console.print(f"[red]Plugin '{name}' is not installed.[/red]") + if not _plugin_exists(name): + console.print(f"[red]Plugin '{name}' is not installed or bundled.[/red]") sys.exit(1) + enabled = _get_enabled_set() disabled = _get_disabled_set() - if name in disabled: + + if name not in enabled and name in disabled: console.print(f"[dim]Plugin '{name}' is already disabled.[/dim]") return + enabled.discard(name) disabled.add(name) + _save_enabled_set(enabled) _save_disabled_set(disabled) - console.print(f"[yellow]\u2298[/yellow] Plugin [bold]{name}[/bold] disabled. Takes effect on next session.") + console.print( + f"[yellow]\u2298[/yellow] Plugin [bold]{name}[/bold] disabled. " + "Takes effect on next session." + ) -def cmd_list() -> None: - """List installed plugins.""" - from rich.console import Console - from rich.table import Table +def _plugin_exists(name: str) -> bool: + """Return True if a plugin with *name* is installed (user) or bundled.""" + # Installed: directory name or manifest name match in user plugins dir + user_dir = _plugins_dir() + if user_dir.is_dir(): + if (user_dir / name).is_dir(): + return True + for child in user_dir.iterdir(): + if not child.is_dir(): + continue + manifest = _read_manifest(child) + if manifest.get("name") == name: + return True + # Bundled: /plugins// + from pathlib import Path as _P + import hermes_cli + repo_plugins = _P(hermes_cli.__file__).resolve().parent.parent / "plugins" + if repo_plugins.is_dir(): + candidate = repo_plugins / name + if candidate.is_dir() and ( + (candidate / "plugin.yaml").exists() + or (candidate / "plugin.yml").exists() + ): + return True + return False + +def _discover_all_plugins() -> list: + """Return a list of (name, version, description, source, dir_path) for + every plugin the loader can see — user + bundled + project. + + Matches the ordering/dedup of ``PluginManager.discover_and_load``: + bundled first, then user, then project; user overrides bundled on + name collision. + """ try: import yaml except ImportError: yaml = None - console = Console() - plugins_dir = _plugins_dir() + seen: dict = {} # name -> (name, version, description, source, path) + + # Bundled (/plugins//), excluding memory/ and context_engine/ + import hermes_cli + repo_plugins = Path(hermes_cli.__file__).resolve().parent.parent / "plugins" + for base, source in ((repo_plugins, "bundled"), (_plugins_dir(), "user")): + if not base.is_dir(): + continue + for d in sorted(base.iterdir()): + if not d.is_dir(): + continue + if source == "bundled" and d.name in ("memory", "context_engine"): + continue + manifest_file = d / "plugin.yaml" + if not manifest_file.exists(): + manifest_file = d / "plugin.yml" + if not manifest_file.exists(): + continue + name = d.name + version = "" + description = "" + if yaml: + try: + with open(manifest_file) as f: + manifest = yaml.safe_load(f) or {} + name = manifest.get("name", d.name) + version = manifest.get("version", "") + description = manifest.get("description", "") + except Exception: + pass + # User plugins override bundled on name collision. + if name in seen and source == "bundled": + continue + src_label = source + if source == "user" and (d / ".git").exists(): + src_label = "git" + seen[name] = (name, version, description, src_label, d) + return list(seen.values()) + - dirs = sorted(d for d in plugins_dir.iterdir() if d.is_dir()) - if not dirs: +def cmd_list() -> None: + """List all plugins (bundled + user) with enabled/disabled state.""" + from rich.console import Console + from rich.table import Table + + console = Console() + entries = _discover_all_plugins() + if not entries: console.print("[dim]No plugins installed.[/dim]") console.print("[dim]Install with:[/dim] hermes plugins install owner/repo") return + enabled = _get_enabled_set() disabled = _get_disabled_set() - table = Table(title="Installed Plugins", show_lines=False) + table = Table(title="Plugins", show_lines=False) table.add_column("Name", style="bold") table.add_column("Status") table.add_column("Version", style="dim") table.add_column("Description") table.add_column("Source", style="dim") - for d in dirs: - manifest_file = d / "plugin.yaml" - name = d.name - version = "" - description = "" - source = "local" - - if manifest_file.exists() and yaml: - try: - with open(manifest_file) as f: - manifest = yaml.safe_load(f) or {} - name = manifest.get("name", d.name) - version = manifest.get("version", "") - description = manifest.get("description", "") - except Exception: - pass - - # Check if it's a git repo (installed via hermes plugins install) - if (d / ".git").exists(): - source = "git" - - is_disabled = name in disabled or d.name in disabled - status = "[red]disabled[/red]" if is_disabled else "[green]enabled[/green]" + for name, version, description, source, _dir in entries: + if name in disabled: + status = "[red]disabled[/red]" + elif name in enabled: + status = "[green]enabled[/green]" + else: + status = "[yellow]not enabled[/yellow]" table.add_row(name, status, str(version), description, source) console.print() @@ -592,6 +732,7 @@ def cmd_list() -> None: console.print() console.print("[dim]Interactive toggle:[/dim] hermes plugins") console.print("[dim]Enable/disable:[/dim] hermes plugins enable/disable ") + console.print("[dim]Plugins are opt-in by default — only 'enabled' plugins load.[/dim]") # --------------------------------------------------------------------------- @@ -742,41 +883,25 @@ def cmd_toggle() -> None: """Interactive composite UI — general plugins + provider plugin categories.""" from rich.console import Console - try: - import yaml - except ImportError: - yaml = None - console = Console() - plugins_dir = _plugins_dir() - # -- General plugins discovery -- - dirs = sorted(d for d in plugins_dir.iterdir() if d.is_dir()) - disabled = _get_disabled_set() + # -- General plugins discovery (bundled + user) -- + entries = _discover_all_plugins() + enabled_set = _get_enabled_set() + disabled_set = _get_disabled_set() plugin_names = [] plugin_labels = [] plugin_selected = set() - for i, d in enumerate(dirs): - manifest_file = d / "plugin.yaml" - name = d.name - description = "" - - if manifest_file.exists() and yaml: - try: - with open(manifest_file) as f: - manifest = yaml.safe_load(f) or {} - name = manifest.get("name", d.name) - description = manifest.get("description", "") - except Exception: - pass - - plugin_names.append(name) + for i, (name, _version, description, source, _d) in enumerate(entries): label = f"{name} \u2014 {description}" if description else name + if source == "bundled": + label = f"{label} [bundled]" + plugin_names.append(name) plugin_labels.append(label) - - if name not in disabled and d.name not in disabled: + # Selected (enabled) when in enabled-set AND not in disabled-set + if name in enabled_set and name not in disabled_set: plugin_selected.add(i) # -- Provider categories -- @@ -804,10 +929,10 @@ def cmd_toggle() -> None: try: import curses _run_composite_ui(curses, plugin_names, plugin_labels, plugin_selected, - disabled, categories, console) + disabled_set, categories, console) except ImportError: _run_composite_fallback(plugin_names, plugin_labels, plugin_selected, - disabled, categories, console) + disabled_set, categories, console) def _run_composite_ui(curses, plugin_names, plugin_labels, plugin_selected, @@ -1020,18 +1145,29 @@ def _draw(stdscr): curses.wrapper(_draw) flush_stdin() - # Persist general plugin changes - new_disabled = set() + # Persist general plugin changes. The new allow-list is the set of + # plugin names that were checked; anything not checked is explicitly + # disabled (written to disabled-list) so it remains off even if the + # plugin code does something clever like auto-enable in the future. + new_enabled: set = set() + new_disabled: set = set(disabled) # preserve existing disabled state for unseen plugins for i, name in enumerate(plugin_names): - if i not in chosen: + if i in chosen: + new_enabled.add(name) + new_disabled.discard(name) + else: new_disabled.add(name) - if new_disabled != disabled: + prev_enabled = _get_enabled_set() + enabled_changed = new_enabled != prev_enabled + disabled_changed = new_disabled != disabled + + if enabled_changed or disabled_changed: + _save_enabled_set(new_enabled) _save_disabled_set(new_disabled) - enabled_count = len(plugin_names) - len(new_disabled) console.print( - f"\n[green]\u2713[/green] General plugins: {enabled_count} enabled, " - f"{len(new_disabled)} disabled." + f"\n[green]\u2713[/green] General plugins: {len(new_enabled)} enabled, " + f"{len(plugin_names) - len(new_enabled)} disabled." ) elif n_plugins > 0: console.print("\n[dim]General plugins unchanged.[/dim]") @@ -1078,11 +1214,17 @@ def _run_composite_fallback(plugin_names, plugin_labels, plugin_selected, return print() - new_disabled = set() + new_enabled: set = set() + new_disabled: set = set(disabled) for i, name in enumerate(plugin_names): - if i not in chosen: + if i in chosen: + new_enabled.add(name) + new_disabled.discard(name) + else: new_disabled.add(name) - if new_disabled != disabled: + prev_enabled = _get_enabled_set() + if new_enabled != prev_enabled or new_disabled != disabled: + _save_enabled_set(new_enabled) _save_disabled_set(new_disabled) # Provider categories @@ -1108,7 +1250,17 @@ def plugins_command(args) -> None: action = getattr(args, "plugins_action", None) if action == "install": - cmd_install(args.identifier, force=getattr(args, "force", False)) + # Map argparse tri-state: --enable=True, --no-enable=False, neither=None (prompt) + enable_arg = None + if getattr(args, "enable", False): + enable_arg = True + elif getattr(args, "no_enable", False): + enable_arg = False + cmd_install( + args.identifier, + force=getattr(args, "force", False), + enable=enable_arg, + ) elif action == "update": cmd_update(args.name) elif action in ("remove", "rm", "uninstall"): diff --git a/hermes_cli/profiles.py b/hermes_cli/profiles.py index 1e9fcae00523..bf6de16dffdd 100644 --- a/hermes_cli/profiles.py +++ b/hermes_cli/profiles.py @@ -300,19 +300,10 @@ def _read_config_model(profile_dir: Path) -> tuple: def _check_gateway_running(profile_dir: Path) -> bool: """Check if a gateway is running for a given profile directory.""" - pid_file = profile_dir / "gateway.pid" - if not pid_file.exists(): - return False try: - raw = pid_file.read_text().strip() - if not raw: - return False - data = json.loads(raw) if raw.startswith("{") else {"pid": int(raw)} - pid = int(data["pid"]) - os.kill(pid, 0) # existence check - return True - except (json.JSONDecodeError, KeyError, ValueError, TypeError, - ProcessLookupError, PermissionError, OSError): + from gateway.status import get_running_pid + return get_running_pid(profile_dir / "gateway.pid", cleanup_stale=False) is not None + except Exception: return False @@ -872,19 +863,15 @@ def _safe_extract_profile_archive(archive: Path, destination: Path) -> None: pass -def import_profile(archive_path: str, name: Optional[str] = None) -> Path: - """Import a profile from a tar.gz archive. +def _inspect_profile_archive_roots(archive: Path) -> set[str]: + """Return the archive's top-level directory names. - If *name* is not given, infers it from the archive's top-level directory. - Returns the imported profile directory. + Profile imports expect exactly one root directory. Inspecting the archive + before extraction lets us stage the import safely instead of mutating a + live profile tree first and reconciling names later. """ import tarfile - archive = Path(archive_path) - if not archive.exists(): - raise FileNotFoundError(f"Archive not found: {archive}") - - # Peek at the archive to find the top-level directory name with tarfile.open(archive, "r:gz") as tf: top_dirs = { parts[0] @@ -898,13 +885,33 @@ def import_profile(archive_path: str, name: Optional[str] = None) -> Path: for member in tf.getmembers() if member.isdir() } + return top_dirs + - inferred_name = name or (top_dirs.pop() if len(top_dirs) == 1 else None) +def import_profile(archive_path: str, name: Optional[str] = None) -> Path: + """Import a profile from a tar.gz archive. + + If *name* is not given, infers it from the archive's top-level directory. + Returns the imported profile directory. + """ + import tempfile + + archive = Path(archive_path) + if not archive.exists(): + raise FileNotFoundError(f"Archive not found: {archive}") + + top_dirs = _inspect_profile_archive_roots(archive) + archive_root = top_dirs.pop() if len(top_dirs) == 1 else None + inferred_name = name or archive_root if not inferred_name: raise ValueError( "Cannot determine profile name from archive. " "Specify it explicitly: hermes profile import --name " ) + if archive_root is None: + raise ValueError( + "Profile archive must contain exactly one top-level directory." + ) # Archives exported from the default profile have "default/" as top-level # dir. Importing as "default" would target ~/.hermes itself — disallow @@ -923,12 +930,22 @@ def import_profile(archive_path: str, name: Optional[str] = None) -> Path: profiles_root = _get_profiles_root() profiles_root.mkdir(parents=True, exist_ok=True) - _safe_extract_profile_archive(archive, profiles_root) + with tempfile.TemporaryDirectory(prefix="hermes_profile_import_") as tmpdir: + staging_root = Path(tmpdir) + _safe_extract_profile_archive(archive, staging_root) + + extracted = staging_root / archive_root + if not extracted.is_dir(): + raise ValueError( + f"Profile archive root is missing or invalid: {archive_root}" + ) + + final_source = extracted + if archive_root != inferred_name: + final_source = staging_root / inferred_name + extracted.rename(final_source) - # If the archive extracted under a different name, rename - extracted = profiles_root / (top_dirs.pop() if top_dirs else inferred_name) - if extracted != profile_dir and extracted.exists(): - extracted.rename(profile_dir) + shutil.move(str(final_source), str(profile_dir)) return profile_dir diff --git a/hermes_cli/providers.py b/hermes_cli/providers.py index ee4beebe0b3d..e842086a412a 100644 --- a/hermes_cli/providers.py +++ b/hermes_cli/providers.py @@ -23,6 +23,8 @@ from dataclasses import dataclass from typing import Any, Dict, List, Optional, Tuple +from utils import base_url_host_matches, base_url_hostname + logger = logging.getLogger(__name__) @@ -64,6 +66,11 @@ class HermesOverlay: base_url_override="https://portal.qwen.ai/v1", base_url_env_var="HERMES_QWEN_BASE_URL", ), + "google-gemini-cli": HermesOverlay( + transport="openai_chat", + auth_type="oauth_external", + base_url_override="cloudcode-pa://google", + ), "copilot-acp": HermesOverlay( transport="codex_responses", auth_type="external_process", @@ -87,6 +94,12 @@ class HermesOverlay: transport="openai_chat", base_url_env_var="KIMI_BASE_URL", ), + "stepfun": HermesOverlay( + transport="openai_chat", + extra_env_vars=("STEPFUN_API_KEY",), + base_url_override="https://api.stepfun.ai/step_plan/v1", + base_url_env_var="STEPFUN_BASE_URL", + ), "minimax": HermesOverlay( transport="anthropic_messages", base_url_env_var="MINIMAX_BASE_URL", @@ -128,14 +141,28 @@ class HermesOverlay: base_url_env_var="HF_BASE_URL", ), "xai": HermesOverlay( - transport="openai_chat", + transport="codex_responses", base_url_override="https://api.x.ai/v1", base_url_env_var="XAI_BASE_URL", ), + "nvidia": HermesOverlay( + transport="openai_chat", + base_url_override="https://integrate.api.nvidia.com/v1", + base_url_env_var="NVIDIA_BASE_URL", + ), "xiaomi": HermesOverlay( transport="openai_chat", base_url_env_var="XIAOMI_BASE_URL", ), + "arcee": HermesOverlay( + transport="openai_chat", + base_url_override="https://api.arcee.ai/api/v1", + base_url_env_var="ARCEE_BASE_URL", + ), + "ollama-cloud": HermesOverlay( + transport="openai_chat", + base_url_env_var="OLLAMA_BASE_URL", + ), } @@ -175,6 +202,13 @@ class ProviderDef: # xai "x-ai": "xai", "x.ai": "xai", + "grok": "xai", + + # nvidia + "nim": "nvidia", + "nvidia-nim": "nvidia", + "build-nvidia": "nvidia", + "nemotron": "nvidia", # kimi-for-coding (models.dev ID) "kimi": "kimi-for-coding", @@ -182,6 +216,10 @@ class ProviderDef: "kimi-coding-cn": "kimi-for-coding", "moonshot": "kimi-for-coding", + # stepfun + "step": "stepfun", + "stepfun-coding-plan": "stepfun", + # minimax-cn "minimax-china": "minimax-cn", "minimax_cn": "minimax-cn", @@ -222,6 +260,11 @@ class ProviderDef: "qwen": "alibaba", "alibaba-cloud": "alibaba", + # google-gemini-cli (OAuth + Code Assist) + "gemini-cli": "google-gemini-cli", + "gemini-oauth": "google-gemini-cli", + + # huggingface "hf": "huggingface", "hugging-face": "huggingface", @@ -231,11 +274,21 @@ class ProviderDef: "mimo": "xiaomi", "xiaomi-mimo": "xiaomi", + # bedrock + "aws": "bedrock", + "aws-bedrock": "bedrock", + "amazon-bedrock": "bedrock", + "amazon": "bedrock", + + # arcee + "arcee-ai": "arcee", + "arceeai": "arcee", + # Local server aliases → virtual "local" concept (resolved via user config) "lmstudio": "lmstudio", "lm-studio": "lmstudio", "lm_studio": "lmstudio", - "ollama": "ollama-cloud", + "ollama": "custom", # bare "ollama" = local; use "ollama-cloud" for cloud "vllm": "local", "llamacpp": "local", "llama.cpp": "local", @@ -251,8 +304,11 @@ class ProviderDef: "nous": "Nous Portal", "openai-codex": "OpenAI Codex", "copilot-acp": "GitHub Copilot ACP", + "stepfun": "StepFun Step Plan", "xiaomi": "Xiaomi MiMo", "local": "Local endpoint", + "bedrock": "AWS Bedrock", + "ollama-cloud": "Ollama Cloud", } @@ -262,6 +318,7 @@ class ProviderDef: "openai_chat": "chat_completions", "anthropic_messages": "anthropic_messages", "codex_responses": "codex_responses", + "bedrock_converse": "bedrock_converse", } @@ -278,12 +335,16 @@ def normalize_provider(name: str) -> str: def get_provider(name: str) -> Optional[ProviderDef]: - """Look up a provider by id or alias, merging all data sources. + """Look up a built-in provider by id or alias. Resolution order: 1. Hermes overlays (for providers not in models.dev: nous, openai-codex, etc.) 2. models.dev catalog + Hermes overlay - 3. User-defined providers from config (TODO: Phase 4) + + User-defined providers from config.yaml (``providers:`` / ``custom_providers:``) + are resolved by :func:`resolve_provider_full`, which layers ``resolve_user_provider`` + and ``resolve_custom_provider`` on top of this function. Callers that need + user-config support should use ``resolve_provider_full`` instead. Returns a fully-resolved ProviderDef or None. """ @@ -377,15 +438,34 @@ def determine_api_mode(provider: str, base_url: str = "") -> str: """ pdef = get_provider(provider) if pdef is not None: + # Even for known providers, check URL heuristics for special endpoints + # (e.g. kimi /coding endpoint needs anthropic_messages even on 'custom') + if base_url: + url_lower = base_url.rstrip("/").lower() + if "api.kimi.com/coding" in url_lower: + return "anthropic_messages" + if url_lower.endswith("/anthropic") or "api.anthropic.com" in url_lower: + return "anthropic_messages" + if "api.openai.com" in url_lower: + return "codex_responses" return TRANSPORT_TO_API_MODE.get(pdef.transport, "chat_completions") + # Direct provider checks for providers not in HERMES_OVERLAYS + if provider == "bedrock": + return "bedrock_converse" + # URL-based heuristics for custom / unknown providers if base_url: url_lower = base_url.rstrip("/").lower() - if url_lower.endswith("/anthropic") or "api.anthropic.com" in url_lower: + hostname = base_url_hostname(base_url) + if url_lower.endswith("/anthropic") or hostname == "api.anthropic.com": + return "anthropic_messages" + if hostname == "api.kimi.com" and "/coding" in url_lower: return "anthropic_messages" - if "api.openai.com" in url_lower: + if hostname == "api.openai.com": return "codex_responses" + if hostname.startswith("bedrock-runtime.") and base_url_host_matches(base_url, "amazonaws.com"): + return "bedrock_converse" return "chat_completions" diff --git a/hermes_cli/runtime_provider.py b/hermes_cli/runtime_provider.py index 54b9ae65c37f..922946e2ad06 100644 --- a/hermes_cli/runtime_provider.py +++ b/hermes_cli/runtime_provider.py @@ -22,12 +22,14 @@ resolve_nous_runtime_credentials, resolve_codex_runtime_credentials, resolve_qwen_runtime_credentials, + resolve_gemini_oauth_runtime_credentials, resolve_api_key_provider_credentials, resolve_external_process_provider_credentials, has_usable_secret, ) from hermes_cli.config import get_compatible_custom_providers, load_config from hermes_constants import OPENROUTER_BASE_URL +from utils import base_url_host_matches, base_url_hostname def _normalize_custom_provider_name(value: str) -> str: @@ -37,12 +39,27 @@ def _normalize_custom_provider_name(value: str) -> str: def _detect_api_mode_for_url(base_url: str) -> Optional[str]: """Auto-detect api_mode from the resolved base URL. - Direct api.openai.com endpoints need the Responses API for GPT-5.x - tool calls with reasoning (chat/completions returns 400). + - Direct api.openai.com endpoints need the Responses API for GPT-5.x + tool calls with reasoning (chat/completions returns 400). + - Third-party Anthropic-compatible gateways (MiniMax, Zhipu GLM, + LiteLLM proxies, etc.) conventionally expose the native Anthropic + protocol under a ``/anthropic`` suffix — treat those as + ``anthropic_messages`` transport instead of the default + ``chat_completions``. + - Kimi Code's ``api.kimi.com/coding`` endpoint also speaks the + Anthropic Messages protocol (the /coding route accepts Claude + Code's native request shape). """ normalized = (base_url or "").strip().lower().rstrip("/") - if "api.openai.com" in normalized and "openrouter" not in normalized: + hostname = base_url_hostname(base_url) + if hostname == "api.x.ai": return "codex_responses" + if hostname == "api.openai.com": + return "codex_responses" + if normalized.endswith("/anthropic"): + return "anthropic_messages" + if hostname == "api.kimi.com" and "/coding" in normalized: + return "anthropic_messages" return None @@ -124,7 +141,7 @@ def _copilot_runtime_api_mode(model_cfg: Dict[str, Any], api_key: str) -> str: return "chat_completions" -_VALID_API_MODES = {"chat_completions", "codex_responses", "anthropic_messages"} +_VALID_API_MODES = {"chat_completions", "codex_responses", "anthropic_messages", "bedrock_converse"} def _parse_api_mode(raw: Any) -> Optional[str]: @@ -154,6 +171,9 @@ def _resolve_runtime_from_pool_entry( elif provider == "qwen-oauth": api_mode = "chat_completions" base_url = base_url or DEFAULT_QWEN_BASE_URL + elif provider == "google-gemini-cli": + api_mode = "chat_completions" + base_url = base_url or "cloudcode-pa://google" elif provider == "anthropic": api_mode = "anthropic_messages" cfg_provider = str(model_cfg.get("provider") or "").strip().lower() @@ -163,10 +183,13 @@ def _resolve_runtime_from_pool_entry( base_url = cfg_base_url or base_url or "https://api.anthropic.com" elif provider == "openrouter": base_url = base_url or OPENROUTER_BASE_URL + elif provider == "xai": + api_mode = "codex_responses" elif provider == "nous": api_mode = "chat_completions" elif provider == "copilot": api_mode = _copilot_runtime_api_mode(model_cfg, getattr(entry, "runtime_api_key", "")) + base_url = base_url or PROVIDER_REGISTRY["copilot"].inference_base_url else: configured_provider = str(model_cfg.get("provider") or "").strip().lower() # Honour model.base_url from config.yaml when the configured provider @@ -185,8 +208,13 @@ def _resolve_runtime_from_pool_entry( elif provider in ("opencode-zen", "opencode-go"): from hermes_cli.models import opencode_model_api_mode api_mode = opencode_model_api_mode(provider, model_cfg.get("default", "")) - elif base_url.rstrip("/").endswith("/anthropic"): - api_mode = "anthropic_messages" + else: + # Auto-detect Anthropic-compatible endpoints (/anthropic suffix, + # Kimi /coding, api.openai.com → codex_responses, api.x.ai → + # codex_responses). + detected = _detect_api_mode_for_url(base_url) + if detected: + api_mode = detected # OpenCode base URLs end with /v1 for OpenAI-compatible models, but the # Anthropic SDK prepends its own /v1/messages to the base_url. Strip the @@ -287,6 +315,9 @@ def _get_named_custom_provider(requested_provider: str) -> Optional[Dict[str, An # Resolve the API key from the env var name stored in key_env key_env = str(entry.get("key_env", "") or "").strip() resolved_api_key = os.getenv(key_env, "").strip() if key_env else "" + # Fall back to inline api_key when key_env is absent or unresolvable + if not resolved_api_key: + resolved_api_key = str(entry.get("api_key", "") or "").strip() if requested_norm in {ep_name, name_norm, f"custom:{name_norm}"}: # Found match by provider key @@ -457,7 +488,7 @@ def _resolve_openrouter_runtime( # When hitting a custom endpoint (e.g. Z.ai, local LLM), prefer # OPENAI_API_KEY so the OpenRouter key doesn't leak to an unrelated # provider (issues #420, #560). - _is_openrouter_url = "openrouter.ai" in base_url + _is_openrouter_url = base_url_host_matches(base_url, "openrouter.ai") if _is_openrouter_url: api_key_candidates = [ explicit_api_key, @@ -467,8 +498,12 @@ def _resolve_openrouter_runtime( else: # Custom endpoint: use api_key from config when using config base_url (#1760). # When the endpoint is Ollama Cloud, check OLLAMA_API_KEY — it's - # the canonical env var for ollama.com authentication. - _is_ollama_url = "ollama.com" in base_url.lower() + # the canonical env var for ollama.com authentication. Match on + # HOST, not substring — a custom base_url whose path contains + # "ollama.com" (e.g. http://127.0.0.1/ollama.com/v1) or whose + # hostname is a look-alike (ollama.com.attacker.test) must not + # receive the Ollama credential. See GHSA-76xc-57q6-vm5m. + _is_ollama_url = base_url_host_matches(base_url, "ollama.com") api_key_candidates = [ explicit_api_key, (cfg_api_key if use_config_base_url else ""), @@ -624,12 +659,18 @@ def _resolve_explicit_runtime( api_mode = "chat_completions" if provider == "copilot": api_mode = _copilot_runtime_api_mode(model_cfg, api_key) + elif provider == "xai": + api_mode = "codex_responses" else: configured_mode = _parse_api_mode(model_cfg.get("api_mode")) if configured_mode: api_mode = configured_mode - elif base_url.rstrip("/").endswith("/anthropic"): - api_mode = "anthropic_messages" + else: + # Auto-detect from URL (Anthropic /anthropic suffix, + # api.openai.com → Responses, Kimi /coding, etc.). + detected = _detect_api_mode_for_url(base_url) + if detected: + api_mode = detected return { "provider": provider, @@ -794,6 +835,26 @@ def resolve_runtime_provider( logger.info("Qwen OAuth credentials failed; " "falling through to next provider.") + if provider == "google-gemini-cli": + try: + creds = resolve_gemini_oauth_runtime_credentials() + return { + "provider": "google-gemini-cli", + "api_mode": "chat_completions", + "base_url": creds.get("base_url", ""), + "api_key": creds.get("api_key", ""), + "source": creds.get("source", "google-oauth"), + "expires_at_ms": creds.get("expires_at_ms"), + "email": creds.get("email", ""), + "project_id": creds.get("project_id", ""), + "requested_provider": requested_provider, + } + except AuthError: + if requested_provider != "auto": + raise + logger.info("Google Gemini OAuth credentials failed; " + "falling through to next provider.") + if provider == "copilot-acp": creds = resolve_external_process_provider_credentials(provider) return { @@ -833,6 +894,76 @@ def resolve_runtime_provider( "requested_provider": requested_provider, } + # AWS Bedrock (native Converse API via boto3) + if provider == "bedrock": + from agent.bedrock_adapter import ( + has_aws_credentials, + resolve_aws_auth_env_var, + resolve_bedrock_region, + is_anthropic_bedrock_model, + ) + # When the user explicitly selected bedrock (not auto-detected), + # trust boto3's credential chain — it handles IMDS, ECS task roles, + # Lambda execution roles, SSO, and other implicit sources that our + # env-var check can't detect. + is_explicit = requested_provider in ("bedrock", "aws", "aws-bedrock", "amazon-bedrock", "amazon") + if not is_explicit and not has_aws_credentials(): + raise AuthError( + "No AWS credentials found for Bedrock. Configure one of:\n" + " - AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY\n" + " - AWS_PROFILE (for SSO / named profiles)\n" + " - IAM instance role (EC2, ECS, Lambda)\n" + "Or run 'aws configure' to set up credentials.", + code="no_aws_credentials", + ) + # Read bedrock-specific config from config.yaml + _bedrock_cfg = load_config().get("bedrock", {}) + # Region priority: config.yaml bedrock.region → env var → us-east-1 + region = (_bedrock_cfg.get("region") or "").strip() or resolve_bedrock_region() + auth_source = resolve_aws_auth_env_var() or "aws-sdk-default-chain" + # Build guardrail config if configured + _gr = _bedrock_cfg.get("guardrail", {}) + guardrail_config = None + if _gr.get("guardrail_identifier") and _gr.get("guardrail_version"): + guardrail_config = { + "guardrailIdentifier": _gr["guardrail_identifier"], + "guardrailVersion": _gr["guardrail_version"], + } + if _gr.get("stream_processing_mode"): + guardrail_config["streamProcessingMode"] = _gr["stream_processing_mode"] + if _gr.get("trace"): + guardrail_config["trace"] = _gr["trace"] + # Dual-path routing: Claude models use AnthropicBedrock SDK for full + # feature parity (prompt caching, thinking budgets, adaptive thinking). + # Non-Claude models use the Converse API for multi-model support. + _current_model = str(model_cfg.get("default") or "").strip() + if is_anthropic_bedrock_model(_current_model): + # Claude on Bedrock → AnthropicBedrock SDK → anthropic_messages path + runtime = { + "provider": "bedrock", + "api_mode": "anthropic_messages", + "base_url": f"https://bedrock-runtime.{region}.amazonaws.com", + "api_key": "aws-sdk", + "source": auth_source, + "region": region, + "bedrock_anthropic": True, # Signal to use AnthropicBedrock client + "requested_provider": requested_provider, + } + else: + # Non-Claude (Nova, DeepSeek, Llama, etc.) → Converse API + runtime = { + "provider": "bedrock", + "api_mode": "bedrock_converse", + "base_url": f"https://bedrock-runtime.{region}.amazonaws.com", + "api_key": "aws-sdk", + "source": auth_source, + "region": region, + "requested_provider": requested_provider, + } + if guardrail_config: + runtime["guardrail_config"] = guardrail_config + return runtime + # API-key providers (z.ai/GLM, Kimi, MiniMax, MiniMax-CN) pconfig = PROVIDER_REGISTRY.get(provider) if pconfig and pconfig.auth_type == "api_key": @@ -849,6 +980,8 @@ def resolve_runtime_provider( api_mode = "chat_completions" if provider == "copilot": api_mode = _copilot_runtime_api_mode(model_cfg, creds.get("api_key", "")) + elif provider == "xai": + api_mode = "codex_responses" else: configured_provider = str(model_cfg.get("provider") or "").strip().lower() # Only honor persisted api_mode when it belongs to the same provider family. @@ -858,10 +991,13 @@ def resolve_runtime_provider( elif provider in ("opencode-zen", "opencode-go"): from hermes_cli.models import opencode_model_api_mode api_mode = opencode_model_api_mode(provider, model_cfg.get("default", "")) - # Auto-detect Anthropic-compatible endpoints by URL convention - # (e.g. https://api.minimax.io/anthropic, https://dashscope.../anthropic) - elif base_url.rstrip("/").endswith("/anthropic"): - api_mode = "anthropic_messages" + else: + # Auto-detect Anthropic-compatible endpoints by URL convention + # (e.g. https://api.minimax.io/anthropic, https://dashscope.../anthropic) + # plus api.openai.com → codex_responses and api.x.ai → codex_responses. + detected = _detect_api_mode_for_url(base_url) + if detected: + api_mode = detected # Strip trailing /v1 for OpenCode Anthropic models (see comment above). if api_mode == "anthropic_messages" and provider in ("opencode-zen", "opencode-go"): base_url = re.sub(r"/v1/?$", "", base_url) diff --git a/hermes_cli/setup.py b/hermes_cli/setup.py index 996dc87daa74..ebc7de94073c 100644 --- a/hermes_cli/setup.py +++ b/hermes_cli/setup.py @@ -20,11 +20,9 @@ from pathlib import Path from typing import Optional, Dict, Any -from hermes_cli.nous_subscription import ( - apply_nous_provider_defaults, - get_nous_subscription_features, -) +from hermes_cli.nous_subscription import get_nous_subscription_features from tools.tool_backend_helpers import managed_nous_tools_enabled +from utils import base_url_hostname from hermes_constants import get_optional_skills_dir logger = logging.getLogger(__name__) @@ -43,14 +41,6 @@ def _model_config_dict(config: Dict[str, Any]) -> Dict[str, Any]: return {} -def _set_default_model(config: Dict[str, Any], model_name: str) -> None: - if not model_name: - return - model_cfg = _model_config_dict(config) - model_cfg["default"] = model_name - config["model"] = model_cfg - - def _get_credential_pool_strategies(config: Dict[str, Any]) -> Dict[str, str]: strategies = config.get("credential_pool_strategies") return dict(strategies) if isinstance(strategies, dict) else {} @@ -100,19 +90,20 @@ def _supports_same_provider_pool_setup(provider: str) -> bool: "grok-code-fast-1", ], "gemini": [ - "gemini-3.1-pro-preview", "gemini-3-flash-preview", "gemini-3.1-flash-lite-preview", - "gemini-2.5-pro", "gemini-2.5-flash", "gemini-2.5-flash-lite", - "gemma-4-31b-it", "gemma-4-26b-it", + "gemini-3.1-pro-preview", "gemini-3-pro-preview", + "gemini-3-flash-preview", "gemini-3.1-flash-lite-preview", ], "zai": ["glm-5.1", "glm-5", "glm-4.7", "glm-4.5", "glm-4.5-flash"], - "kimi-coding": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"], - "kimi-coding-cn": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"], + "kimi-coding": ["kimi-k2.6", "kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"], + "kimi-coding-cn": ["kimi-k2.6", "kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"], + "stepfun": ["step-3.5-flash", "step-3.5-flash-2603"], + "arcee": ["trinity-large-thinking", "trinity-large-preview", "trinity-mini"], "minimax": ["MiniMax-M2.7", "MiniMax-M2.5", "MiniMax-M2.1", "MiniMax-M2"], "minimax-cn": ["MiniMax-M2.7", "MiniMax-M2.5", "MiniMax-M2.1", "MiniMax-M2"], "ai-gateway": ["anthropic/claude-opus-4.6", "anthropic/claude-sonnet-4.6", "openai/gpt-5", "google/gemini-3-flash"], "kilocode": ["anthropic/claude-opus-4.6", "anthropic/claude-sonnet-4.6", "openai/gpt-5.4", "google/gemini-3-pro-preview", "google/gemini-3-flash-preview"], "opencode-zen": ["gpt-5.4", "gpt-5.3-codex", "claude-sonnet-4-6", "gemini-3-flash", "glm-5", "kimi-k2.5", "minimax-m2.7"], - "opencode-go": ["glm-5", "kimi-k2.5", "mimo-v2-pro", "mimo-v2-omni", "minimax-m2.5", "minimax-m2.7"], + "opencode-go": ["kimi-k2.6", "kimi-k2.5", "glm-5.1", "glm-5", "mimo-v2.5-pro", "mimo-v2.5", "mimo-v2-pro", "mimo-v2-omni", "minimax-m2.7", "minimax-m2.5", "qwen3.6-plus", "qwen3.5-plus"], "huggingface": [ "Qwen/Qwen3.5-397B-A17B", "Qwen/Qwen3-235B-A22B-Thinking-2507", "Qwen/Qwen3-Coder-480B-A35B-Instruct", "deepseek-ai/DeepSeek-R1-0528", @@ -136,43 +127,6 @@ def _set_reasoning_effort(config: Dict[str, Any], effort: str) -> None: agent_cfg["reasoning_effort"] = effort -def _setup_copilot_reasoning_selection( - config: Dict[str, Any], - model_id: str, - prompt_choice, - *, - catalog: Optional[list[dict[str, Any]]] = None, - api_key: str = "", -) -> None: - from hermes_cli.models import github_model_reasoning_efforts, normalize_copilot_model_id - - normalized_model = normalize_copilot_model_id( - model_id, - catalog=catalog, - api_key=api_key, - ) or model_id - efforts = github_model_reasoning_efforts(normalized_model, catalog=catalog, api_key=api_key) - if not efforts: - return - - current_effort = _current_reasoning_effort(config) - choices = list(efforts) + ["Disable reasoning", f"Keep current ({current_effort or 'default'})"] - - if current_effort == "none": - default_idx = len(efforts) - elif current_effort in efforts: - default_idx = efforts.index(current_effort) - elif "medium" in efforts: - default_idx = efforts.index("medium") - else: - default_idx = len(choices) - 1 - - effort_idx = prompt_choice("Select reasoning effort:", choices, default_idx) - if effort_idx < len(efforts): - _set_reasoning_effort(config, efforts[effort_idx]) - elif effort_idx == len(efforts): - _set_reasoning_effort(config, "none") - # Import config helpers @@ -257,20 +211,20 @@ def prompt(question: str, default: str = None, password: bool = False) -> str: sys.exit(1) -def _curses_prompt_choice(question: str, choices: list, default: int = 0) -> int: +def _curses_prompt_choice(question: str, choices: list, default: int = 0, description: str | None = None) -> int: """Single-select menu using curses. Delegates to curses_radiolist.""" from hermes_cli.curses_ui import curses_radiolist - return curses_radiolist(question, choices, selected=default, cancel_returns=-1) + return curses_radiolist(question, choices, selected=default, cancel_returns=-1, description=description) -def prompt_choice(question: str, choices: list, default: int = 0) -> int: +def prompt_choice(question: str, choices: list, default: int = 0, description: str | None = None) -> int: """Prompt for a choice from a list with arrow key navigation. Escape keeps the current default (skips the question). Ctrl+C exits the wizard. """ - idx = _curses_prompt_choice(question, choices, default) + idx = _curses_prompt_choice(question, choices, default, description=description) if idx >= 0: if idx == default: print_info(" Skipped (keeping current)") @@ -455,13 +409,36 @@ def _print_setup_summary(config: dict, hermes_home): ("Browser Automation", False, missing_browser_hint) ) - # FAL (image generation) + # Image generation — FAL (direct or via Nous), or any plugin-registered + # provider (OpenAI, etc.) if subscription_features.image_gen.managed_by_nous: tool_status.append(("Image Generation (Nous subscription)", True, None)) elif subscription_features.image_gen.available: tool_status.append(("Image Generation", True, None)) else: - tool_status.append(("Image Generation", False, "FAL_KEY")) + # Fall back to probing plugin-registered providers so OpenAI-only + # setups don't show as "missing FAL_KEY". + _img_backend = None + try: + from agent.image_gen_registry import list_providers + from hermes_cli.plugins import _ensure_plugins_discovered + + _ensure_plugins_discovered() + for _p in list_providers(): + if _p.name == "fal": + continue + try: + if _p.is_available(): + _img_backend = _p.display_name + break + except Exception: + continue + except Exception: + pass + if _img_backend: + tool_status.append((f"Image Generation ({_img_backend})", True, None)) + else: + tool_status.append(("Image Generation", False, "FAL_KEY or OPENAI_API_KEY")) # TTS — show configured provider tts_provider = config.get("tts", {}).get("provider", "edge") @@ -477,9 +454,10 @@ def _print_setup_summary(config: dict, hermes_home): tool_status.append(("Text-to-Speech (MiniMax)", True, None)) elif tts_provider == "mistral" and get_env_value("MISTRAL_API_KEY"): tool_status.append(("Text-to-Speech (Mistral Voxtral)", True, None)) + elif tts_provider == "gemini" and (get_env_value("GEMINI_API_KEY") or get_env_value("GOOGLE_API_KEY")): + tool_status.append(("Text-to-Speech (Google Gemini)", True, None)) elif tts_provider == "neutts": try: - import importlib.util neutts_ok = importlib.util.find_spec("neutts") is not None except Exception: neutts_ok = False @@ -487,6 +465,16 @@ def _print_setup_summary(config: dict, hermes_home): tool_status.append(("Text-to-Speech (NeuTTS local)", True, None)) else: tool_status.append(("Text-to-Speech (NeuTTS — not installed)", False, "run 'hermes setup tts'")) + elif tts_provider == "kittentts": + try: + import importlib.util + kittentts_ok = importlib.util.find_spec("kittentts") is not None + except Exception: + kittentts_ok = False + if kittentts_ok: + tool_status.append(("Text-to-Speech (KittenTTS local)", True, None)) + else: + tool_status.append(("Text-to-Speech (KittenTTS — not installed)", False, "run 'hermes setup tts'")) else: tool_status.append(("Text-to-Speech (Edge TTS)", True, None)) @@ -817,10 +805,11 @@ def setup_model_provider(config: dict, *, quick: bool = False): "zai": "Z.AI / GLM", "kimi-coding": "Kimi / Moonshot", "kimi-coding-cn": "Kimi / Moonshot (China)", + "stepfun": "StepFun Step Plan", "minimax": "MiniMax", "minimax-cn": "MiniMax CN", "anthropic": "Anthropic", - "ai-gateway": "AI Gateway", + "ai-gateway": "Vercel AI Gateway", "custom": "your custom endpoint", } _prov_display = _prov_names.get(selected_provider, selected_provider or "your provider") @@ -849,7 +838,8 @@ def setup_model_provider(config: dict, *, quick: bool = False): elif _vision_idx == 1: # OpenAI-compatible endpoint _base_url = prompt(" Base URL (blank for OpenAI)").strip() or "https://api.openai.com/v1" _api_key_label = " API key" - if "api.openai.com" in _base_url.lower(): + _is_native_openai = base_url_hostname(_base_url) == "api.openai.com" + if _is_native_openai: _api_key_label = " OpenAI API key" _oai_key = prompt(_api_key_label, password=True).strip() if _oai_key: @@ -857,7 +847,7 @@ def setup_model_provider(config: dict, *, quick: bool = False): # Save vision base URL to config (not .env — only secrets go there) _vaux = config.setdefault("auxiliary", {}).setdefault("vision", {}) _vaux["base_url"] = _base_url - if "api.openai.com" in _base_url.lower(): + if _is_native_openai: _oai_vision_models = ["gpt-4o", "gpt-4o-mini", "gpt-4.1", "gpt-4.1-mini", "gpt-4.1-nano"] _vm_choices = _oai_vision_models + ["Use default (gpt-4o-mini)"] _vm_idx = prompt_choice("Select vision model:", _vm_choices, 0) @@ -879,14 +869,7 @@ def setup_model_provider(config: dict, *, quick: bool = False): print_info("Skipped — add later with 'hermes setup' or configure AUXILIARY_VISION_* settings") - if selected_provider == "nous" and nous_subscription_selected: - changed_defaults = apply_nous_provider_defaults(config) - current_tts = str(config.get("tts", {}).get("provider") or "edge") - if "tts" in changed_defaults: - print_success("TTS provider set to: OpenAI TTS via your Nous subscription") - else: - print_info(f"Keeping your existing TTS provider: {current_tts}") - + # Tool Gateway prompt is already shown by _model_flow_nous() above. save_config(config) if not quick and selected_provider != "nous": @@ -900,7 +883,6 @@ def setup_model_provider(config: dict, *, quick: bool = False): def _check_espeak_ng() -> bool: """Check if espeak-ng is installed.""" - import shutil return shutil.which("espeak-ng") is not None or shutil.which("espeak") is not None @@ -954,6 +936,31 @@ def _install_neutts_deps() -> bool: return False +def _install_kittentts_deps() -> bool: + """Install KittenTTS dependencies with user approval. Returns True on success.""" + import subprocess + import sys + + wheel_url = ( + "https://github.com/KittenML/KittenTTS/releases/download/" + "0.8.1/kittentts-0.8.1-py3-none-any.whl" + ) + print() + print_info("Installing kittentts Python package (~25-80MB model downloaded on first use)...") + print() + try: + subprocess.run( + [sys.executable, "-m", "pip", "install", "-U", wheel_url, "soundfile", "--quiet"], + check=True, timeout=300, + ) + print_success("kittentts installed successfully") + return True + except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e: + print_error(f"Failed to install kittentts: {e}") + print_info(f"Try manually: python -m pip install -U '{wheel_url}' soundfile") + return False + + def _setup_tts_provider(config: dict): """Interactive TTS provider selection with install flow for NeuTTS.""" tts_config = config.get("tts", {}) @@ -964,9 +971,12 @@ def _setup_tts_provider(config: dict): "edge": "Edge TTS", "elevenlabs": "ElevenLabs", "openai": "OpenAI TTS", + "xai": "xAI TTS", "minimax": "MiniMax TTS", "mistral": "Mistral Voxtral TTS", + "gemini": "Google Gemini TTS", "neutts": "NeuTTS", + "kittentts": "KittenTTS", } current_label = provider_labels.get(current_provider, current_provider) @@ -985,12 +995,15 @@ def _setup_tts_provider(config: dict): "Edge TTS (free, cloud-based, no setup needed)", "ElevenLabs (premium quality, needs API key)", "OpenAI TTS (good quality, needs API key)", + "xAI TTS (Grok voices, needs API key)", "MiniMax TTS (high quality with voice cloning, needs API key)", "Mistral Voxtral TTS (multilingual, native Opus, needs API key)", + "Google Gemini TTS (30 prebuilt voices, prompt-controllable, needs API key)", "NeuTTS (local on-device, free, ~300MB model download)", + "KittenTTS (local on-device, free, lightweight ~25-80MB ONNX)", ] ) - providers.extend(["edge", "elevenlabs", "openai", "minimax", "mistral", "neutts"]) + providers.extend(["edge", "elevenlabs", "openai", "xai", "minimax", "mistral", "gemini", "neutts", "kittentts"]) choices.append(f"Keep current ({current_label})") keep_current_idx = len(choices) - 1 idx = prompt_choice("Select TTS provider:", choices, keep_current_idx) @@ -1011,7 +1024,6 @@ def _setup_tts_provider(config: dict): if selected == "neutts": # Check if already installed try: - import importlib.util already_installed = importlib.util.find_spec("neutts") is not None except Exception: already_installed = False @@ -1056,6 +1068,23 @@ def _setup_tts_provider(config: dict): print_warning("No API key provided. Falling back to Edge TTS.") selected = "edge" + elif selected == "xai": + existing = get_env_value("XAI_API_KEY") + if not existing: + print() + api_key = prompt("xAI API key for TTS", password=True) + if api_key: + save_env_value("XAI_API_KEY", api_key) + print_success("xAI TTS API key saved") + else: + from hermes_constants import display_hermes_home as _dhh + print_warning( + "No xAI API key provided for TTS. Configure XAI_API_KEY via " + f"hermes setup model or {_dhh()}/.env to use xAI TTS. " + "Falling back to Edge TTS." + ) + selected = "edge" + elif selected == "minimax": existing = get_env_value("MINIMAX_API_KEY") if not existing: @@ -1080,6 +1109,42 @@ def _setup_tts_provider(config: dict): print_warning("No API key provided. Falling back to Edge TTS.") selected = "edge" + elif selected == "gemini": + existing = get_env_value("GEMINI_API_KEY") or get_env_value("GOOGLE_API_KEY") + if not existing: + print() + print_info("Get a free API key at https://aistudio.google.com/app/apikey") + api_key = prompt("Gemini API key for TTS", password=True) + if api_key: + save_env_value("GEMINI_API_KEY", api_key) + print_success("Gemini TTS API key saved") + else: + print_warning("No API key provided. Falling back to Edge TTS.") + selected = "edge" + + elif selected == "kittentts": + # Check if already installed + try: + import importlib.util + already_installed = importlib.util.find_spec("kittentts") is not None + except Exception: + already_installed = False + + if already_installed: + print_success("KittenTTS is already installed") + else: + print() + print_info("KittenTTS is lightweight (~25-80MB, CPU-only, no API key required).") + print_info("Voices: Jasper, Bella, Luna, Bruno, Rosie, Hugo, Kiki, Leo") + print() + if prompt_yes_no("Install KittenTTS now?", True): + if not _install_kittentts_deps(): + print_warning("KittenTTS installation incomplete. Falling back to Edge TTS.") + selected = "edge" + else: + print_info("Skipping install. Set tts.provider to 'kittentts' after installing manually.") + selected = "edge" + # Save the selection if "tts" not in config: config["tts"] = {} @@ -1101,8 +1166,6 @@ def setup_tts(config: dict): def setup_terminal_backend(config: dict): """Configure the terminal execution backend.""" import platform as _platform - import shutil - print_header("Terminal Backend") print_info("Choose where Hermes runs shell commands and code.") print_info("This affects tool execution, file access, and isolation.") @@ -1479,7 +1542,9 @@ def setup_agent_settings(config: dict): ) print_info("Maximum tool-calling iterations per conversation.") print_info("Higher = more complex tasks, but costs more tokens.") - print_info("Default is 90, which works for most tasks. Use 150+ for open exploration.") + print_info( + f"Press Enter to keep {current_max}. Use 90 for most tasks or 150+ for open exploration." + ) max_iter_str = prompt("Max iterations", current_max) try: @@ -1655,9 +1720,19 @@ def _setup_telegram(): return print_info("Create a bot via @BotFather on Telegram") - token = prompt("Telegram bot token", password=True) - if not token: - return + import re + + while True: + token = prompt("Telegram bot token", password=True) + if not token: + return + if not re.match(r"^\d+:[A-Za-z0-9_-]{30,}$", token): + print_error( + "Invalid token format. Expected: : " + "(e.g., 123456789:ABCdefGHI-jklMNOpqrSTUvwxYZ)" + ) + continue + break save_env_value("TELEGRAM_BOT_TOKEN", token) print_success("Telegram token saved") @@ -1781,7 +1856,7 @@ def _setup_slack(): print_info(" 3. Add Bot Token Scopes: Features → OAuth & Permissions") print_info(" Required scopes: chat:write, app_mentions:read,") print_info(" channels:history, channels:read, im:history,") - print_info(" im:read, im:write, users:read, files:write") + print_info(" im:read, im:write, users:read, files:read, files:write") print_info(" Optional for private channels: groups:history") print_info(" 4. Subscribe to Events: Features → Event Subscriptions → Enable") print_info(" Required events: message.im, message.channels, app_mention") @@ -2013,6 +2088,8 @@ def _setup_wecom_callback(): _gw_setup() + + def _setup_bluebubbles(): """Configure BlueBubbles iMessage gateway.""" print_header("BlueBubbles (iMessage)") @@ -2078,6 +2155,12 @@ def _setup_bluebubbles(): print_info(" Install: https://docs.bluebubbles.app/helper-bundle/installation") +def _setup_qqbot(): + """Configure QQ Bot (Official API v2) via gateway setup.""" + from hermes_cli.gateway import _setup_qqbot as _gateway_setup_qqbot + _gateway_setup_qqbot() + + def _setup_webhooks(): """Configure webhook integration.""" print_header("Webhooks") @@ -2141,6 +2224,7 @@ def _setup_webhooks(): ("WeCom Callback (Self-Built App)", "WECOM_CALLBACK_CORP_ID", _setup_wecom_callback), ("Weixin (WeChat)", "WEIXIN_ACCOUNT_ID", _setup_weixin), ("BlueBubbles (iMessage)", "BLUEBUBBLES_SERVER_URL", _setup_bluebubbles), + ("QQ Bot", "QQ_APP_ID", _setup_qqbot), ("Webhooks (GitHub, GitLab, etc.)", "WEBHOOK_ENABLED", _setup_webhooks), ] @@ -2192,6 +2276,7 @@ def setup_gateway(config: dict): or get_env_value("WECOM_BOT_ID") or get_env_value("WEIXIN_ACCOUNT_ID") or get_env_value("BLUEBUBBLES_SERVER_URL") + or get_env_value("QQ_APP_ID") or get_env_value("WEBHOOK_ENABLED") ) if any_messaging: @@ -2213,6 +2298,10 @@ def setup_gateway(config: dict): missing_home.append("Slack") if get_env_value("BLUEBUBBLES_SERVER_URL") and not get_env_value("BLUEBUBBLES_HOME_CHANNEL"): missing_home.append("BlueBubbles") + if get_env_value("QQ_APP_ID") and not ( + get_env_value("QQBOT_HOME_CHANNEL") or get_env_value("QQ_HOME_CHANNEL") + ): + missing_home.append("QQBot") if missing_home: print() @@ -2236,13 +2325,16 @@ def setup_gateway(config: dict): _is_service_running, supports_systemd_services, has_conflicting_systemd_units, + has_legacy_hermes_units, install_linux_gateway_from_setup, print_systemd_scope_conflict_warning, + print_legacy_unit_warning, systemd_start, systemd_restart, launchd_install, launchd_start, launchd_restart, + UserSystemdUnavailableError, ) service_installed = _is_service_installed() @@ -2255,6 +2347,10 @@ def setup_gateway(config: dict): print_systemd_scope_conflict_warning() print() + if supports_systemd and has_legacy_hermes_units(): + print_legacy_unit_warning() + print() + if service_running: if prompt_yes_no(" Restart the gateway to pick up changes?", True): try: @@ -2262,6 +2358,10 @@ def setup_gateway(config: dict): systemd_restart() elif _is_macos: launchd_restart() + except UserSystemdUnavailableError as e: + print_error(" Restart failed — user systemd not reachable:") + for line in str(e).splitlines(): + print(f" {line}") except Exception as e: print_error(f" Restart failed: {e}") elif service_installed: @@ -2271,6 +2371,10 @@ def setup_gateway(config: dict): systemd_start() elif _is_macos: launchd_start() + except UserSystemdUnavailableError as e: + print_error(" Start failed — user systemd not reachable:") + for line in str(e).splitlines(): + print(f" {line}") except Exception as e: print_error(f" Start failed: {e}") elif supports_service_manager: @@ -2294,6 +2398,10 @@ def setup_gateway(config: dict): systemd_start(system=installed_scope == "system") elif _is_macos: launchd_start() + except UserSystemdUnavailableError as e: + print_error(" Start failed — user systemd not reachable:") + for line in str(e).splitlines(): + print(f" {line}") except Exception as e: print_error(f" Start failed: {e}") except Exception as e: @@ -2345,6 +2453,74 @@ def setup_tools(config: dict, first_install: bool = False): # ============================================================================= +def _model_section_has_credentials(config: dict) -> bool: + """Return True when any known inference provider has usable credentials. + + Sources of truth: + * ``PROVIDER_REGISTRY`` in ``hermes_cli.auth`` — lists every supported + provider along with its ``api_key_env_vars``. + * ``active_provider`` in the auth store — covers OAuth device-code / + external-OAuth providers (Nous, Codex, Qwen, Gemini CLI, ...). + * The legacy OpenRouter aggregator env vars, which route generic + ``OPENAI_API_KEY`` / ``OPENROUTER_API_KEY`` values through OpenRouter. + """ + try: + from hermes_cli.auth import get_active_provider + if get_active_provider(): + return True + except Exception: + pass + + try: + from hermes_cli.auth import PROVIDER_REGISTRY + except Exception: + PROVIDER_REGISTRY = {} # type: ignore[assignment] + + def _has_key(pconfig) -> bool: + for env_var in pconfig.api_key_env_vars: + # CLAUDE_CODE_OAUTH_TOKEN is set by Claude Code itself, not by + # the user — mirrors is_provider_explicitly_configured in auth.py. + if env_var == "CLAUDE_CODE_OAUTH_TOKEN": + continue + if get_env_value(env_var): + return True + return False + + # Prefer the provider declared in config.yaml, avoids false positives + # from stray env vars (GH_TOKEN, etc.) when the user has already picked + # a different provider. + model_cfg = config.get("model") if isinstance(config, dict) else None + if isinstance(model_cfg, dict): + provider_id = (model_cfg.get("provider") or "").strip().lower() + if provider_id in PROVIDER_REGISTRY: + if _has_key(PROVIDER_REGISTRY[provider_id]): + return True + if provider_id == "openrouter": + for env_var in ("OPENROUTER_API_KEY", "OPENAI_API_KEY"): + if get_env_value(env_var): + return True + + # OpenRouter aggregator fallback (no provider declared in config). + for env_var in ("OPENROUTER_API_KEY", "OPENAI_API_KEY"): + if get_env_value(env_var): + return True + + for pid, pconfig in PROVIDER_REGISTRY.items(): + # Skip copilot in auto-detect: GH_TOKEN / GITHUB_TOKEN are + # commonly set for git tooling. Mirrors resolve_provider in auth.py. + if pid == "copilot": + continue + if _has_key(pconfig): + return True + return False + + +def _gateway_platform_short_label(label: str) -> str: + """Strip trailing parenthetical qualifiers from a gateway platform label.""" + base = label.split("(", 1)[0].strip() + return base or label + + def _get_section_config_summary(config: dict, section_key: str) -> Optional[str]: """Return a short summary if a setup section is already configured, else None. @@ -2353,20 +2529,7 @@ def _get_section_config_summary(config: dict, section_key: str) -> Optional[str] so that test patches on ``setup_mod.get_env_value`` take effect. """ if section_key == "model": - has_key = bool( - get_env_value("OPENROUTER_API_KEY") - or get_env_value("OPENAI_API_KEY") - or get_env_value("ANTHROPIC_API_KEY") - ) - if not has_key: - # Check for OAuth providers - try: - from hermes_cli.auth import get_active_provider - if get_active_provider(): - has_key = True - except Exception: - pass - if not has_key: + if not _model_section_has_credentials(config): return None model = config.get("model") if isinstance(model, str) and model.strip(): @@ -2384,37 +2547,11 @@ def _get_section_config_summary(config: dict, section_key: str) -> Optional[str] return f"max turns: {max_turns}" elif section_key == "gateway": - platforms = [] - if get_env_value("TELEGRAM_BOT_TOKEN"): - platforms.append("Telegram") - if get_env_value("DISCORD_BOT_TOKEN"): - platforms.append("Discord") - if get_env_value("SLACK_BOT_TOKEN"): - platforms.append("Slack") - if get_env_value("SIGNAL_ACCOUNT"): - platforms.append("Signal") - if get_env_value("EMAIL_ADDRESS"): - platforms.append("Email") - if get_env_value("TWILIO_ACCOUNT_SID"): - platforms.append("SMS") - if get_env_value("MATRIX_ACCESS_TOKEN") or get_env_value("MATRIX_PASSWORD"): - platforms.append("Matrix") - if get_env_value("MATTERMOST_TOKEN"): - platforms.append("Mattermost") - if get_env_value("WHATSAPP_PHONE_NUMBER_ID"): - platforms.append("WhatsApp") - if get_env_value("DINGTALK_CLIENT_ID"): - platforms.append("DingTalk") - if get_env_value("FEISHU_APP_ID"): - platforms.append("Feishu") - if get_env_value("WECOM_BOT_ID"): - platforms.append("WeCom") - if get_env_value("WEIXIN_ACCOUNT_ID"): - platforms.append("Weixin") - if get_env_value("BLUEBUBBLES_SERVER_URL"): - platforms.append("BlueBubbles") - if get_env_value("WEBHOOK_ENABLED"): - platforms.append("Webhooks") + platforms = [ + _gateway_platform_short_label(label) + for label, env_var, _ in _GATEWAY_PLATFORMS + if get_env_value(env_var) + ] if platforms: return ", ".join(platforms) return None # No platforms configured — section must run diff --git a/hermes_cli/skills_config.py b/hermes_cli/skills_config.py index 92424a0ca361..741a8b834166 100644 --- a/hermes_cli/skills_config.py +++ b/hermes_cli/skills_config.py @@ -15,7 +15,7 @@ from hermes_cli.config import load_config, save_config from hermes_cli.colors import Colors, color -from hermes_cli.platforms import PLATFORMS as _PLATFORMS, platform_label +from hermes_cli.platforms import PLATFORMS as _PLATFORMS # Backward-compatible view: {key: label_string} so existing code that # iterates ``PLATFORMS.items()`` or calls ``PLATFORMS.get(key)`` keeps diff --git a/hermes_cli/skills_hub.py b/hermes_cli/skills_hub.py index ed922805b77c..bf92fafe1008 100644 --- a/hermes_cli/skills_hub.py +++ b/hermes_cli/skills_hub.py @@ -515,6 +515,90 @@ def do_inspect(identifier: str, console: Optional[Console] = None) -> None: c.print() +def browse_skills(page: int = 1, page_size: int = 20, source: str = "all") -> dict: + """Paginated hub browse for programmatic callers (e.g. TUI gateway). + + Returns ``{"items": [...], "page": int, "total_pages": int, "total": int}``. + """ + from tools.skills_hub import GitHubAuth, create_source_router + + page_size = max(1, min(page_size, 100)) + _TRUST_RANK = {"builtin": 3, "trusted": 2, "community": 1} + _PER_SOURCE_LIMIT = {"official": 100, "skills-sh": 100, "well-known": 25, "github": 100, "clawhub": 50, + "claude-marketplace": 50, "lobehub": 50} + auth = GitHubAuth() + sources = create_source_router(auth) + all_results: list = [] + for src in sources: + sid = src.source_id() + if source != "all" and sid != source and sid != "official": + continue + try: + limit = _PER_SOURCE_LIMIT.get(sid, 50) + all_results.extend(src.search("", limit=limit)) + except Exception: + continue + if not all_results: + return {"items": [], "page": 1, "total_pages": 1, "total": 0} + seen: dict = {} + for r in all_results: + rank = _TRUST_RANK.get(r.trust_level, 0) + if r.name not in seen or rank > _TRUST_RANK.get(seen[r.name].trust_level, 0): + seen[r.name] = r + deduped = list(seen.values()) + deduped.sort(key=lambda r: (-_TRUST_RANK.get(r.trust_level, 0), r.source != "official", r.name.lower())) + total = len(deduped) + total_pages = max(1, (total + page_size - 1) // page_size) + page = max(1, min(page, total_pages)) + start = (page - 1) * page_size + page_items = deduped[start : min(start + page_size, total)] + return { + "items": [{"name": r.name, "description": r.description, "source": r.source, + "trust": r.trust_level} for r in page_items], + "page": page, + "total_pages": total_pages, + "total": total, + } + + +def inspect_skill(identifier: str) -> Optional[dict]: + """Skill metadata (+ SKILL.md preview) for programmatic callers.""" + from tools.skills_hub import GitHubAuth, create_source_router + + class _Q: + def print(self, *a, **k): + pass + + c = _Q() + auth = GitHubAuth() + sources = create_source_router(auth) + ident = identifier + if "/" not in ident: + ident = _resolve_short_name(ident, sources, c) + if not ident: + return None + meta, bundle, _ = _resolve_source_meta_and_bundle(ident, sources) + if not meta: + return None + out: dict = { + "name": meta.name, + "description": meta.description, + "source": meta.source, + "identifier": meta.identifier, + "tags": list(meta.tags) if meta.tags else [], + } + if bundle and "SKILL.md" in bundle.files: + content = bundle.files["SKILL.md"] + if isinstance(content, bytes): + content = content.decode("utf-8", errors="replace") + lines = content.split("\n") + preview = "\n".join(lines[:50]) + if len(lines) > 50: + preview += f"\n\n... ({len(lines) - 50} more lines)" + out["skill_md_preview"] = preview + return out + + def do_list(source_filter: str = "all", console: Optional[Console] = None) -> None: """List installed skills, distinguishing hub, builtin, and local skills.""" from tools.skills_hub import HubLockFile, ensure_hub_dirs @@ -684,6 +768,51 @@ def do_uninstall(name: str, console: Optional[Console] = None, c.print(f"[bold red]Error:[/] {msg}\n") +def do_reset(name: str, restore: bool = False, + console: Optional[Console] = None, + skip_confirm: bool = False, + invalidate_cache: bool = True) -> None: + """Reset a bundled skill's manifest tracking (+ optionally restore from bundled).""" + from tools.skills_sync import reset_bundled_skill + + c = console or _console + + if not skip_confirm and restore: + c.print(f"\n[bold]Restore '{name}' from bundled source?[/]") + c.print("[dim]This will DELETE your current copy and re-copy the bundled version.[/]") + try: + answer = input("Confirm [y/N]: ").strip().lower() + except (EOFError, KeyboardInterrupt): + answer = "n" + if answer not in ("y", "yes"): + c.print("[dim]Cancelled.[/]\n") + return + + result = reset_bundled_skill(name, restore=restore) + + if not result["ok"]: + c.print(f"[bold red]Error:[/] {result['message']}\n") + return + + c.print(f"[bold green]{result['message']}[/]") + synced = result.get("synced") or {} + if synced.get("copied"): + c.print(f"[dim]Copied: {', '.join(synced['copied'])}[/]") + if synced.get("updated"): + c.print(f"[dim]Updated: {', '.join(synced['updated'])}[/]") + c.print() + + if invalidate_cache: + try: + from agent.prompt_builder import clear_skills_system_prompt_cache + clear_skills_system_prompt_cache(clear_snapshot=True) + except Exception: + pass + else: + c.print("[dim]Change will take effect in your next session.[/]") + c.print("[dim]Use /reset to start a new session now, or --now to apply immediately (invalidates prompt cache).[/]\n") + + def do_tap(action: str, repo: str = "", console: Optional[Console] = None) -> None: """Manage taps (custom GitHub repo sources).""" from tools.skills_hub import TapsManager @@ -1007,6 +1136,9 @@ def skills_command(args) -> None: do_audit(name=getattr(args, "name", None)) elif action == "uninstall": do_uninstall(args.name) + elif action == "reset": + do_reset(args.name, restore=getattr(args, "restore", False), + skip_confirm=getattr(args, "yes", False)) elif action == "publish": do_publish( args.skill_path, @@ -1029,7 +1161,7 @@ def skills_command(args) -> None: return do_tap(tap_action, repo=repo) else: - _console.print("Usage: hermes skills [browse|search|install|inspect|list|check|update|audit|uninstall|publish|snapshot|tap]\n") + _console.print("Usage: hermes skills [browse|search|install|inspect|list|check|update|audit|uninstall|reset|publish|snapshot|tap]\n") _console.print("Run 'hermes skills --help' for details.\n") @@ -1175,6 +1307,19 @@ def handle_skills_slash(cmd: str, console: Optional[Console] = None) -> None: do_uninstall(args[0], console=c, skip_confirm=skip_confirm, invalidate_cache=invalidate_cache) + elif action == "reset": + if not args: + c.print("[bold red]Usage:[/] /skills reset [--restore] [--now]\n") + c.print("[dim]Clears the bundled-skills manifest entry so future updates stop marking it as user-modified.[/]") + c.print("[dim]Pass --restore to also replace the current copy with the bundled version.[/]\n") + return + name = args[0] + restore = "--restore" in args + invalidate_cache = "--now" in args + # Slash commands can't prompt — --restore in slash mode is implicit consent. + do_reset(name, restore=restore, console=c, skip_confirm=True, + invalidate_cache=invalidate_cache) + elif action == "publish": if not args: c.print("[bold red]Usage:[/] /skills publish [--to github] [--repo owner/repo]\n") @@ -1231,6 +1376,7 @@ def _print_skills_help(console: Console) -> None: " [cyan]update[/] [name] Update hub skills with upstream changes\n" " [cyan]audit[/] [name] Re-scan hub skills for security\n" " [cyan]uninstall[/] Remove a hub-installed skill\n" + " [cyan]reset[/] [--restore] Reset bundled-skill tracking (fix 'user-modified' flag)\n" " [cyan]publish[/] --repo Publish a skill to GitHub via PR\n" " [cyan]snapshot[/] export|import Export/import skill configurations\n" " [cyan]tap[/] list|add|remove Manage skill sources\n", diff --git a/hermes_cli/skin_engine.py b/hermes_cli/skin_engine.py index 16ec39cc9b4f..5619e7405c84 100644 --- a/hermes_cli/skin_engine.py +++ b/hermes_cli/skin_engine.py @@ -23,15 +23,29 @@ banner_dim: "#B8860B" # Dim/muted text (separators, labels) banner_text: "#FFF8DC" # Body text (tool names, skill names) ui_accent: "#FFBF00" # General UI accent - ui_label: "#4dd0e1" # UI labels + ui_label: "#DAA520" # UI labels (warm gold; teal clashed w/ default banner gold) ui_ok: "#4caf50" # Success indicators ui_error: "#ef5350" # Error indicators ui_warn: "#ffa726" # Warning indicators prompt: "#FFF8DC" # Prompt text color input_rule: "#CD7F32" # Input area horizontal rule response_border: "#FFD700" # Response box border (ANSI) + status_bar_bg: "#1a1a2e" # Status bar background + status_bar_text: "#C0C0C0" # Status bar default text + status_bar_strong: "#FFD700" # Status bar highlighted text + status_bar_dim: "#8B8682" # Status bar separators/muted text + status_bar_good: "#8FBC8F" # Healthy context usage + status_bar_warn: "#FFD700" # Warning context usage + status_bar_bad: "#FF8C00" # High context usage + status_bar_critical: "#FF6B6B" # Critical context usage session_label: "#DAA520" # Session label color session_border: "#8B8682" # Session ID dim color + status_bar_bg: "#1a1a2e" # TUI status/usage bar background + voice_status_bg: "#1a1a2e" # TUI voice status background + completion_menu_bg: "#1a1a2e" # Completion menu background + completion_menu_current_bg: "#333355" # Active completion row background + completion_menu_meta_bg: "#1a1a2e" # Completion meta column background + completion_menu_meta_current_bg: "#333355" # Active completion meta background # Spinner: customize the animated spinner during API calls spinner: @@ -87,6 +101,8 @@ - ``ares`` — Crimson/bronze war-god theme with custom spinner wings - ``mono`` — Clean grayscale monochrome - ``slate`` — Cool blue developer-focused theme +- ``daylight`` — Light background theme with dark text and blue accents +- ``warm-lightmode`` — Warm brown/gold text for light terminal backgrounds USER SKINS ========== @@ -126,10 +142,6 @@ def get_color(self, key: str, fallback: str = "") -> str: """Get a color value with fallback.""" return self.colors.get(key, fallback) - def get_spinner_list(self, key: str) -> List[str]: - """Get a spinner list (faces, verbs, etc.).""" - return self.spinner.get(key, []) - def get_spinner_wings(self) -> List[Tuple[str, str]]: """Get spinner wing pairs, or empty list if none.""" raw = self.spinner.get("wings", []) @@ -159,13 +171,14 @@ def get_branding(self, key: str, fallback: str = "") -> str: "banner_dim": "#B8860B", "banner_text": "#FFF8DC", "ui_accent": "#FFBF00", - "ui_label": "#4dd0e1", + "ui_label": "#DAA520", "ui_ok": "#4caf50", "ui_error": "#ef5350", "ui_warn": "#ffa726", "prompt": "#FFF8DC", "input_rule": "#CD7F32", "response_border": "#FFD700", + "status_bar_bg": "#1a1a2e", "session_label": "#DAA520", "session_border": "#8B8682", }, @@ -199,6 +212,14 @@ def get_branding(self, key: str, fallback: str = "") -> str: "prompt": "#F1E6CF", "input_rule": "#9F1C1C", "response_border": "#C7A96B", + "status_bar_bg": "#2A1212", + "status_bar_text": "#F1E6CF", + "status_bar_strong": "#C7A96B", + "status_bar_dim": "#6E584B", + "status_bar_good": "#7BC96F", + "status_bar_warn": "#C7A96B", + "status_bar_bad": "#DD4A3A", + "status_bar_critical": "#EF5350", "session_label": "#C7A96B", "session_border": "#6E584B", }, @@ -263,6 +284,14 @@ def get_branding(self, key: str, fallback: str = "") -> str: "prompt": "#c9d1d9", "input_rule": "#444444", "response_border": "#aaaaaa", + "status_bar_bg": "#1F1F1F", + "status_bar_text": "#C9D1D9", + "status_bar_strong": "#E6EDF3", + "status_bar_dim": "#777777", + "status_bar_good": "#B5B5B5", + "status_bar_warn": "#AAAAAA", + "status_bar_bad": "#D0D0D0", + "status_bar_critical": "#F0F0F0", "session_label": "#888888", "session_border": "#555555", }, @@ -294,6 +323,14 @@ def get_branding(self, key: str, fallback: str = "") -> str: "prompt": "#c9d1d9", "input_rule": "#4169e1", "response_border": "#7eb8f6", + "status_bar_bg": "#151C2F", + "status_bar_text": "#C9D1D9", + "status_bar_strong": "#7EB8F6", + "status_bar_dim": "#4B5563", + "status_bar_good": "#63D0A6", + "status_bar_warn": "#E6A855", + "status_bar_bad": "#F7A072", + "status_bar_critical": "#FF7A7A", "session_label": "#7eb8f6", "session_border": "#4b5563", }, @@ -308,6 +345,80 @@ def get_branding(self, key: str, fallback: str = "") -> str: }, "tool_prefix": "┊", }, + "daylight": { + "name": "daylight", + "description": "Light theme for bright terminals with dark text and cool blue accents", + "colors": { + "banner_border": "#2563EB", + "banner_title": "#0F172A", + "banner_accent": "#1D4ED8", + "banner_dim": "#475569", + "banner_text": "#111827", + "ui_accent": "#2563EB", + "ui_label": "#0F766E", + "ui_ok": "#15803D", + "ui_error": "#B91C1C", + "ui_warn": "#B45309", + "prompt": "#111827", + "input_rule": "#93C5FD", + "response_border": "#2563EB", + "session_label": "#1D4ED8", + "session_border": "#64748B", + "status_bar_bg": "#E5EDF8", + "voice_status_bg": "#E5EDF8", + "completion_menu_bg": "#F8FAFC", + "completion_menu_current_bg": "#DBEAFE", + "completion_menu_meta_bg": "#EEF2FF", + "completion_menu_meta_current_bg": "#BFDBFE", + }, + "spinner": {}, + "branding": { + "agent_name": "Hermes Agent", + "welcome": "Welcome to Hermes Agent! Type your message or /help for commands.", + "goodbye": "Goodbye! ⚕", + "response_label": " ⚕ Hermes ", + "prompt_symbol": "❯ ", + "help_header": "[?] Available Commands", + }, + "tool_prefix": "│", + }, + "warm-lightmode": { + "name": "warm-lightmode", + "description": "Warm light mode — dark brown/gold text for light terminal backgrounds", + "colors": { + "banner_border": "#8B6914", + "banner_title": "#5C3D11", + "banner_accent": "#8B4513", + "banner_dim": "#8B7355", + "banner_text": "#2C1810", + "ui_accent": "#8B4513", + "ui_label": "#5C3D11", + "ui_ok": "#2E7D32", + "ui_error": "#C62828", + "ui_warn": "#E65100", + "prompt": "#2C1810", + "input_rule": "#8B6914", + "response_border": "#8B6914", + "session_label": "#5C3D11", + "session_border": "#A0845C", + "status_bar_bg": "#F5F0E8", + "voice_status_bg": "#F5F0E8", + "completion_menu_bg": "#F5EFE0", + "completion_menu_current_bg": "#E8DCC8", + "completion_menu_meta_bg": "#F0E8D8", + "completion_menu_meta_current_bg": "#DFCFB0", + }, + "spinner": {}, + "branding": { + "agent_name": "Hermes Agent", + "welcome": "Welcome to Hermes Agent! Type your message or /help for commands.", + "goodbye": "Goodbye! \u2695", + "response_label": " \u2695 Hermes ", + "prompt_symbol": "\u276f ", + "help_header": "(^_^)? Available Commands", + }, + "tool_prefix": "\u250a", + }, "poseidon": { "name": "poseidon", "description": "Ocean-god theme — deep blue and seafoam", @@ -325,6 +436,14 @@ def get_branding(self, key: str, fallback: str = "") -> str: "prompt": "#EAF7FF", "input_rule": "#2A6FB9", "response_border": "#5DB8F5", + "status_bar_bg": "#0F2440", + "status_bar_text": "#EAF7FF", + "status_bar_strong": "#A9DFFF", + "status_bar_dim": "#496884", + "status_bar_good": "#6ED7B0", + "status_bar_warn": "#5DB8F5", + "status_bar_bad": "#2A6FB9", + "status_bar_critical": "#D94F4F", "session_label": "#A9DFFF", "session_border": "#496884", }, @@ -389,6 +508,14 @@ def get_branding(self, key: str, fallback: str = "") -> str: "prompt": "#F5F5F5", "input_rule": "#656565", "response_border": "#B7B7B7", + "status_bar_bg": "#202020", + "status_bar_text": "#D3D3D3", + "status_bar_strong": "#F5F5F5", + "status_bar_dim": "#656565", + "status_bar_good": "#B7B7B7", + "status_bar_warn": "#D3D3D3", + "status_bar_bad": "#E7E7E7", + "status_bar_critical": "#F5F5F5", "session_label": "#919191", "session_border": "#656565", }, @@ -454,6 +581,14 @@ def get_branding(self, key: str, fallback: str = "") -> str: "prompt": "#FFF0D4", "input_rule": "#C75B1D", "response_border": "#F29C38", + "status_bar_bg": "#2B160E", + "status_bar_text": "#FFF0D4", + "status_bar_strong": "#FFD39A", + "status_bar_dim": "#6C4724", + "status_bar_good": "#6BCB77", + "status_bar_warn": "#F29C38", + "status_bar_bad": "#E2832B", + "status_bar_critical": "#EF5350", "session_label": "#FFD39A", "session_border": "#6C4724", }, @@ -630,7 +765,9 @@ def init_skin_from_config(config: dict) -> None: Call this once during CLI init with the loaded config dict. """ - display = config.get("display", {}) + display = config.get("display") or {} + if not isinstance(display, dict): + display = {} skin_name = display.get("skin", "default") if isinstance(skin_name, str) and skin_name.strip(): set_active_skin(skin_name.strip()) @@ -689,6 +826,19 @@ def get_prompt_toolkit_style_overrides() -> Dict[str, str]: label = skin.get_color("ui_label", title) warn = skin.get_color("ui_warn", "#FF8C00") error = skin.get_color("ui_error", "#FF6B6B") + status_bg = skin.get_color("status_bar_bg", "#1a1a2e") + status_text = skin.get_color("status_bar_text", text) + status_strong = skin.get_color("status_bar_strong", title) + status_dim = skin.get_color("status_bar_dim", dim) + status_good = skin.get_color("status_bar_good", skin.get_color("ui_ok", "#8FBC8F")) + status_warn = skin.get_color("status_bar_warn", warn) + status_bad = skin.get_color("status_bar_bad", skin.get_color("banner_accent", warn)) + status_critical = skin.get_color("status_bar_critical", error) + voice_bg = skin.get_color("voice_status_bg", status_bg) + menu_bg = skin.get_color("completion_menu_bg", "#1a1a2e") + menu_current_bg = skin.get_color("completion_menu_current_bg", "#333355") + menu_meta_bg = skin.get_color("completion_menu_meta_bg", menu_bg) + menu_meta_current_bg = skin.get_color("completion_menu_meta_current_bg", menu_current_bg) return { "input-area": prompt, @@ -696,13 +846,20 @@ def get_prompt_toolkit_style_overrides() -> Dict[str, str]: "prompt": prompt, "prompt-working": f"{dim} italic", "hint": f"{dim} italic", + "status-bar": f"bg:{status_bg} {status_text}", + "status-bar-strong": f"bg:{status_bg} {status_strong} bold", + "status-bar-dim": f"bg:{status_bg} {status_dim}", + "status-bar-good": f"bg:{status_bg} {status_good} bold", + "status-bar-warn": f"bg:{status_bg} {status_warn} bold", + "status-bar-bad": f"bg:{status_bg} {status_bad} bold", + "status-bar-critical": f"bg:{status_bg} {status_critical} bold", "input-rule": input_rule, "image-badge": f"{label} bold", - "completion-menu": f"bg:#1a1a2e {text}", - "completion-menu.completion": f"bg:#1a1a2e {text}", - "completion-menu.completion.current": f"bg:#333355 {title}", - "completion-menu.meta.completion": f"bg:#1a1a2e {dim}", - "completion-menu.meta.completion.current": f"bg:#333355 {label}", + "completion-menu": f"bg:{menu_bg} {text}", + "completion-menu.completion": f"bg:{menu_bg} {text}", + "completion-menu.completion.current": f"bg:{menu_current_bg} {title}", + "completion-menu.meta.completion": f"bg:{menu_meta_bg} {dim}", + "completion-menu.meta.completion.current": f"bg:{menu_meta_current_bg} {label}", "clarify-border": input_rule, "clarify-title": f"{title} bold", "clarify-question": f"{text} bold", @@ -720,4 +877,6 @@ def get_prompt_toolkit_style_overrides() -> Dict[str, str]: "approval-cmd": f"{dim} italic", "approval-choice": dim, "approval-selected": f"{title} bold", + "voice-status": f"bg:{voice_bg} {label}", + "voice-status-recording": f"bg:{voice_bg} {error} bold", } diff --git a/hermes_cli/status.py b/hermes_cli/status.py index a7745d65f914..8541f0a05faf 100644 --- a/hermes_cli/status.py +++ b/hermes_cli/status.py @@ -122,6 +122,7 @@ def show_status(args): "OpenAI": "OPENAI_API_KEY", "Z.AI/GLM": "GLM_API_KEY", "Kimi": "KIMI_API_KEY", + "StepFun Step Plan": "STEPFUN_API_KEY", "MiniMax": "MINIMAX_API_KEY", "MiniMax-CN": "MINIMAX_CN_API_KEY", "Firecrawl": "FIRECRAWL_API_KEY", @@ -212,7 +213,7 @@ def show_status(args): if managed_nous_tools_enabled(): features = get_nous_subscription_features(config) print() - print(color("◆ Nous Subscription Features", Colors.CYAN, Colors.BOLD)) + print(color("◆ Nous Tool Gateway", Colors.CYAN, Colors.BOLD)) if not features.nous_auth_present: print(" Nous Portal ✗ not logged in") else: @@ -230,6 +231,18 @@ def show_status(args): else: state = "not configured" print(f" {feature.label:<15} {check_mark(feature.available or feature.active or feature.managed_by_nous)} {state}") + elif nous_logged_in: + # Logged into Nous but on the free tier — show upgrade nudge + print() + print(color("◆ Nous Tool Gateway", Colors.CYAN, Colors.BOLD)) + print(" Your free-tier Nous account does not include Tool Gateway access.") + print(" Upgrade your subscription to unlock managed web, image, TTS, and browser tools.") + try: + portal_url = nous_status.get("portal_base_url", "").rstrip("/") + if portal_url: + print(f" Upgrade: {portal_url}") + except Exception: + pass # ========================================================================= # API-Key Providers @@ -240,6 +253,7 @@ def show_status(args): apikey_providers = { "Z.AI / GLM": ("GLM_API_KEY", "ZAI_API_KEY", "Z_AI_API_KEY"), "Kimi / Moonshot": ("KIMI_API_KEY",), + "StepFun Step Plan": ("STEPFUN_API_KEY",), "MiniMax": ("MINIMAX_API_KEY",), "MiniMax (China)": ("MINIMAX_CN_API_KEY",), } @@ -305,6 +319,7 @@ def show_status(args): "WeCom Callback": ("WECOM_CALLBACK_CORP_ID", None), "Weixin": ("WEIXIN_ACCOUNT_ID", "WEIXIN_HOME_CHANNEL"), "BlueBubbles": ("BLUEBUBBLES_SERVER_URL", "BLUEBUBBLES_HOME_CHANNEL"), + "QQBot": ("QQ_APP_ID", "QQBOT_HOME_CHANNEL"), } for name, (token_var, home_var) in platforms.items(): @@ -314,6 +329,9 @@ def show_status(args): home_channel = "" if home_var: home_channel = os.getenv(home_var, "") + # Back-compat: QQBot home channel was renamed from QQ_HOME_CHANNEL to QQBOT_HOME_CHANNEL + if not home_channel and home_var == "QQBOT_HOME_CHANNEL": + home_channel = os.getenv("QQ_HOME_CHANNEL", "") status = "configured" if has_token else "not configured" if home_channel: @@ -326,73 +344,36 @@ def show_status(args): # ========================================================================= print() print(color("◆ Gateway Service", Colors.CYAN, Colors.BOLD)) - - if _is_termux(): - try: - from hermes_cli.gateway import find_gateway_pids - gateway_pids = find_gateway_pids() - except Exception: - gateway_pids = [] - is_running = bool(gateway_pids) + + try: + from hermes_cli.gateway import get_gateway_runtime_snapshot, _format_gateway_pids + + snapshot = get_gateway_runtime_snapshot() + is_running = snapshot.running print(f" Status: {check_mark(is_running)} {'running' if is_running else 'stopped'}") - print(" Manager: Termux / manual process") - if gateway_pids: - rendered = ", ".join(str(pid) for pid in gateway_pids[:3]) - if len(gateway_pids) > 3: - rendered += ", ..." - print(f" PID(s): {rendered}") - else: + print(f" Manager: {snapshot.manager}") + if snapshot.gateway_pids: + print(f" PID(s): {_format_gateway_pids(snapshot.gateway_pids)}") + if snapshot.has_process_service_mismatch: + print(" Service: installed but not managing the current running gateway") + elif _is_termux() and not snapshot.gateway_pids: print(" Start with: hermes gateway") print(" Note: Android may stop background jobs when Termux is suspended") - - elif sys.platform.startswith('linux'): - from hermes_constants import is_container - if is_container(): - # Docker/Podman: no systemd — check for running gateway processes - try: - from hermes_cli.gateway import find_gateway_pids - gateway_pids = find_gateway_pids() - is_active = len(gateway_pids) > 0 - except Exception: - is_active = False - print(f" Status: {check_mark(is_active)} {'running' if is_active else 'stopped'}") - print(" Manager: docker (foreground)") + elif snapshot.service_installed and not snapshot.service_running: + print(" Service: installed but stopped") + except Exception: + if _is_termux(): + print(f" Status: {color('unknown', Colors.DIM)}") + print(" Manager: Termux / manual process") + elif sys.platform.startswith('linux'): + print(f" Status: {color('unknown', Colors.DIM)}") + print(" Manager: systemd/manual") + elif sys.platform == 'darwin': + print(f" Status: {color('unknown', Colors.DIM)}") + print(" Manager: launchd") else: - try: - from hermes_cli.gateway import get_service_name - _gw_svc = get_service_name() - except Exception: - _gw_svc = "hermes-gateway" - try: - result = subprocess.run( - ["systemctl", "--user", "is-active", _gw_svc], - capture_output=True, - text=True, - timeout=5 - ) - is_active = result.stdout.strip() == "active" - except (FileNotFoundError, subprocess.TimeoutExpired): - is_active = False - print(f" Status: {check_mark(is_active)} {'running' if is_active else 'stopped'}") - print(" Manager: systemd (user)") - - elif sys.platform == 'darwin': - from hermes_cli.gateway import get_launchd_label - try: - result = subprocess.run( - ["launchctl", "list", get_launchd_label()], - capture_output=True, - text=True, - timeout=5 - ) - is_loaded = result.returncode == 0 - except subprocess.TimeoutExpired: - is_loaded = False - print(f" Status: {check_mark(is_loaded)} {'loaded' if is_loaded else 'not loaded'}") - print(" Manager: launchd") - else: - print(f" Status: {color('N/A', Colors.DIM)}") - print(" Manager: (not supported on this platform)") + print(f" Status: {color('N/A', Colors.DIM)}") + print(" Manager: (not supported on this platform)") # ========================================================================= # Cron Jobs diff --git a/hermes_cli/timeouts.py b/hermes_cli/timeouts.py new file mode 100644 index 000000000000..59db4012bea8 --- /dev/null +++ b/hermes_cli/timeouts.py @@ -0,0 +1,82 @@ +from __future__ import annotations + + +def _coerce_timeout(raw: object) -> float | None: + try: + timeout = float(raw) + except (TypeError, ValueError): + return None + if timeout <= 0: + return None + return timeout + + +def get_provider_request_timeout( + provider_id: str, model: str | None = None +) -> float | None: + """Return a configured provider request timeout in seconds, if any.""" + if not provider_id: + return None + + try: + from hermes_cli.config import load_config + except ImportError: + return None + + config = load_config() + providers = config.get("providers", {}) if isinstance(config, dict) else {} + provider_config = ( + providers.get(provider_id, {}) if isinstance(providers, dict) else {} + ) + if not isinstance(provider_config, dict): + return None + + model_config = _get_model_config(provider_config, model) + if model_config is not None: + timeout = _coerce_timeout(model_config.get("timeout_seconds")) + if timeout is not None: + return timeout + + return _coerce_timeout(provider_config.get("request_timeout_seconds")) + + +def get_provider_stale_timeout( + provider_id: str, model: str | None = None +) -> float | None: + """Return a configured non-stream stale timeout in seconds, if any.""" + if not provider_id: + return None + + try: + from hermes_cli.config import load_config + except ImportError: + return None + + config = load_config() + providers = config.get("providers", {}) if isinstance(config, dict) else {} + provider_config = ( + providers.get(provider_id, {}) if isinstance(providers, dict) else {} + ) + if not isinstance(provider_config, dict): + return None + + model_config = _get_model_config(provider_config, model) + if model_config is not None: + timeout = _coerce_timeout(model_config.get("stale_timeout_seconds")) + if timeout is not None: + return timeout + + return _coerce_timeout(provider_config.get("stale_timeout_seconds")) + + +def _get_model_config( + provider_config: dict[str, object], model: str | None +) -> dict[str, object] | None: + if not model: + return None + + models = provider_config.get("models", {}) + model_config = models.get(model, {}) if isinstance(models, dict) else {} + if isinstance(model_config, dict): + return model_config + return None diff --git a/hermes_cli/tips.py b/hermes_cli/tips.py index bb9f9e60cd1d..0c1bebe67e3b 100644 --- a/hermes_cli/tips.py +++ b/hermes_cli/tips.py @@ -1,7 +1,7 @@ """Random tips shown at CLI session start to help users discover features.""" import random -from typing import Optional + # --------------------------------------------------------------------------- # Tip corpus — one-liners covering slash commands, CLI flags, config, @@ -127,7 +127,7 @@ # --- Tools & Capabilities --- "execute_code runs Python scripts that call Hermes tools programmatically — results stay out of context.", - "delegate_task spawns up to 3 concurrent sub-agents with isolated contexts for parallel work.", + "delegate_task spawns up to 3 concurrent sub-agents by default (configurable via delegation.max_concurrent_children) with isolated contexts for parallel work.", "web_extract works on PDF URLs — pass any PDF link and it converts to markdown.", "search_files is ripgrep-backed and faster than grep — use it instead of terminal grep.", "patch uses 9 fuzzy matching strategies so minor whitespace differences won't break edits.", @@ -245,7 +245,7 @@ "Three plugin types: general (tools/hooks), memory providers, and context engines.", "hermes plugins install owner/repo installs plugins directly from GitHub.", "8 external memory providers available: Honcho, OpenViking, Mem0, Hindsight, and more.", - "Plugin hooks include pre_tool_call, post_tool_call, pre_llm_call, and post_llm_call.", + "Plugin hooks include pre/post_tool_call, pre/post_llm_call, and transform_terminal_output for output canonicalization.", # --- Miscellaneous --- "Prompt caching (Anthropic) reduces costs by reusing cached system prompt prefixes.", @@ -289,6 +289,7 @@ "When a provider returns HTTP 402 (payment required), the auxiliary client auto-falls back to the next one.", "agent.tool_use_enforcement steers models that describe actions instead of calling tools — auto for GPT/Codex.", "agent.restart_drain_timeout (default 60s) lets running agents finish before a gateway restart takes effect.", + "agent.api_max_retries (default 3) controls how many times the agent retries a failed API call before surfacing the error — lower it for fast fallback.", "The gateway caches AIAgent instances per session — destroying this cache breaks Anthropic prompt caching.", "Any website can expose skills via /.well-known/skills/index.json — the skills hub discovers them automatically.", "The skills audit log at ~/.hermes/skills/.hub/audit.log tracks every install and removal operation.", @@ -323,7 +324,6 @@ "GPT-5 and Codex use 'developer' role instead of 'system' in the message format.", "Per-task auxiliary overrides: auxiliary.vision.provider, auxiliary.compression.model, etc. in config.yaml.", "The auxiliary client treats 'main' as a provider alias — resolves to your actual primary provider + model.", - "Smart routing can auto-route simple queries to a cheaper model — set smart_model_routing.enabled: true.", "hermes claw migrate --dry-run previews OpenClaw migration without writing anything.", "File paths pasted with quotes or escaped spaces are handled automatically — no manual cleanup needed.", "Slash commands never trigger the large-paste collapse — /command with big arguments works correctly.", @@ -346,6 +346,3 @@ def get_random_tip(exclude_recent: int = 0) -> str: return random.choice(TIPS) -def get_tip_count() -> int: - """Return the total number of tips available.""" - return len(TIPS) diff --git a/hermes_cli/tools_config.py b/hermes_cli/tools_config.py index 343007cabc41..e89f961781f3 100644 --- a/hermes_cli/tools_config.py +++ b/hermes_cli/tools_config.py @@ -24,7 +24,8 @@ apply_nous_managed_defaults, get_nous_subscription_features, ) -from tools.tool_backend_helpers import managed_nous_tools_enabled +from tools.tool_backend_helpers import fal_key_is_configured, managed_nous_tools_enabled +from utils import base_url_hostname logger = logging.getLogger(__name__) @@ -63,6 +64,7 @@ ("clarify", "❓ Clarifying Questions", "clarify"), ("delegation", "👥 Task Delegation", "delegate_task"), ("cronjob", "⏰ Cron Jobs", "create/list/update/pause/resume/run, with optional attached skills"), + ("messaging", "📨 Cross-Platform Messaging", "send_message"), ("rl", "🧪 RL Training", "Tinker-Atropos training tools"), ("homeassistant", "🏠 Home Assistant", "smart home device control"), ] @@ -121,6 +123,7 @@ def _get_plugin_toolset_keys() -> set: "providers": [ { "name": "Nous Subscription", + "badge": "subscription", "tag": "Managed OpenAI TTS billed to your subscription", "env_vars": [], "tts_provider": "openai", @@ -130,21 +133,32 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Microsoft Edge TTS", - "tag": "Free - no API key needed", + "badge": "★ recommended · free", + "tag": "Good quality, no API key needed", "env_vars": [], "tts_provider": "edge", }, { "name": "OpenAI TTS", - "tag": "Premium - high quality voices", + "badge": "paid", + "tag": "High quality voices", "env_vars": [ {"key": "VOICE_TOOLS_OPENAI_KEY", "prompt": "OpenAI API key", "url": "https://platform.openai.com/api-keys"}, ], "tts_provider": "openai", }, + { + "name": "xAI TTS", + "tag": "Grok voices - requires xAI API key", + "env_vars": [ + {"key": "XAI_API_KEY", "prompt": "xAI API key", "url": "https://console.x.ai/"}, + ], + "tts_provider": "xai", + }, { "name": "ElevenLabs", - "tag": "Premium - most natural voices", + "badge": "paid", + "tag": "Most natural voices", "env_vars": [ {"key": "ELEVENLABS_API_KEY", "prompt": "ElevenLabs API key", "url": "https://elevenlabs.io/app/settings/api-keys"}, ], @@ -152,12 +166,30 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Mistral (Voxtral TTS)", - "tag": "Multilingual, native Opus, needs MISTRAL_API_KEY", + "badge": "paid", + "tag": "Multilingual, native Opus", "env_vars": [ {"key": "MISTRAL_API_KEY", "prompt": "Mistral API key", "url": "https://console.mistral.ai/"}, ], "tts_provider": "mistral", }, + { + "name": "Google Gemini TTS", + "badge": "preview", + "tag": "30 prebuilt voices, controllable via prompts", + "env_vars": [ + {"key": "GEMINI_API_KEY", "prompt": "Gemini API key", "url": "https://aistudio.google.com/app/apikey"}, + ], + "tts_provider": "gemini", + }, + { + "name": "KittenTTS", + "badge": "local · free", + "tag": "Lightweight local ONNX TTS (~25MB), no API key", + "env_vars": [], + "tts_provider": "kittentts", + "post_setup": "kittentts", + }, ], }, "web": { @@ -168,6 +200,7 @@ def _get_plugin_toolset_keys() -> set: "providers": [ { "name": "Nous Subscription", + "badge": "subscription", "tag": "Managed Firecrawl billed to your subscription", "web_backend": "firecrawl", "env_vars": [], @@ -177,7 +210,8 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Firecrawl Cloud", - "tag": "Hosted service - search, extract, and crawl", + "badge": "★ recommended", + "tag": "Full-featured search, extract, and crawl", "web_backend": "firecrawl", "env_vars": [ {"key": "FIRECRAWL_API_KEY", "prompt": "Firecrawl API key", "url": "https://firecrawl.dev"}, @@ -185,7 +219,8 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Exa", - "tag": "AI-native search and contents", + "badge": "paid", + "tag": "Neural search with semantic understanding", "web_backend": "exa", "env_vars": [ {"key": "EXA_API_KEY", "prompt": "Exa API key", "url": "https://exa.ai"}, @@ -193,7 +228,8 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Parallel", - "tag": "AI-native search and extract", + "badge": "paid", + "tag": "AI-powered search and extract", "web_backend": "parallel", "env_vars": [ {"key": "PARALLEL_API_KEY", "prompt": "Parallel API key", "url": "https://parallel.ai"}, @@ -201,7 +237,8 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Tavily", - "tag": "AI-native search, extract, and crawl", + "badge": "free tier", + "tag": "Search, extract, and crawl — 1000 free searches/mo", "web_backend": "tavily", "env_vars": [ {"key": "TAVILY_API_KEY", "prompt": "Tavily API key", "url": "https://app.tavily.com/home"}, @@ -209,7 +246,8 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Firecrawl Self-Hosted", - "tag": "Free - run your own instance", + "badge": "free · self-hosted", + "tag": "Run your own Firecrawl instance (Docker)", "web_backend": "firecrawl", "env_vars": [ {"key": "FIRECRAWL_API_URL", "prompt": "Your Firecrawl instance URL (e.g., http://localhost:3002)"}, @@ -223,18 +261,22 @@ def _get_plugin_toolset_keys() -> set: "providers": [ { "name": "Nous Subscription", + "badge": "subscription", "tag": "Managed FAL image generation billed to your subscription", "env_vars": [], "requires_nous_auth": True, "managed_nous_feature": "image_gen", "override_env_vars": ["FAL_KEY"], + "imagegen_backend": "fal", }, { "name": "FAL.ai", - "tag": "FLUX 2 Pro with auto-upscaling", + "badge": "paid", + "tag": "Pick from flux-2-klein, flux-2-pro, gpt-image, nano-banana, etc.", "env_vars": [ {"key": "FAL_KEY", "prompt": "FAL API key", "url": "https://fal.ai/dashboard/keys"}, ], + "imagegen_backend": "fal", }, ], }, @@ -244,6 +286,7 @@ def _get_plugin_toolset_keys() -> set: "providers": [ { "name": "Nous Subscription (Browser Use cloud)", + "badge": "subscription", "tag": "Managed Browser Use billed to your subscription", "env_vars": [], "browser_provider": "browser-use", @@ -254,14 +297,16 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Local Browser", - "tag": "Free headless Chromium (no API key needed)", + "badge": "★ recommended · free", + "tag": "Headless Chromium, no API key needed", "env_vars": [], "browser_provider": "local", "post_setup": "agent_browser", }, { "name": "Browserbase", - "tag": "Cloud browser with stealth & proxies", + "badge": "paid", + "tag": "Cloud browser with stealth and proxies", "env_vars": [ {"key": "BROWSERBASE_API_KEY", "prompt": "Browserbase API key", "url": "https://browserbase.com"}, {"key": "BROWSERBASE_PROJECT_ID", "prompt": "Browserbase project ID"}, @@ -271,6 +316,7 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Browser Use", + "badge": "paid", "tag": "Cloud browser with remote execution", "env_vars": [ {"key": "BROWSER_USE_API_KEY", "prompt": "Browser Use API key", "url": "https://browser-use.com"}, @@ -280,6 +326,7 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Firecrawl", + "badge": "paid", "tag": "Cloud browser with remote execution", "env_vars": [ {"key": "FIRECRAWL_API_KEY", "prompt": "Firecrawl API key", "url": "https://firecrawl.dev"}, @@ -289,7 +336,8 @@ def _get_plugin_toolset_keys() -> set: }, { "name": "Camofox", - "tag": "Local anti-detection browser (Firefox/Camoufox)", + "badge": "free · local", + "tag": "Anti-detection browser (Firefox/Camoufox)", "env_vars": [ {"key": "CAMOFOX_URL", "prompt": "Camofox server URL", "default": "http://localhost:9377", "url": "https://github.com/jo-inc/camofox-browser"}, @@ -362,7 +410,7 @@ def _run_post_setup(post_setup_key: str): _print_warning(" Node.js not found - browser tools require: npm install (in hermes-agent directory)") elif post_setup_key == "camofox": - camofox_dir = PROJECT_ROOT / "node_modules" / "@askjo" / "camoufox-browser" + camofox_dir = PROJECT_ROOT / "node_modules" / "@askjo" / "camofox-browser" if not camofox_dir.exists() and shutil.which("npm"): _print_info(" Installing Camofox browser server...") import subprocess @@ -376,13 +424,43 @@ def _run_post_setup(post_setup_key: str): _print_warning(" npm install failed - run manually: npm install") if camofox_dir.exists(): _print_info(" Start the Camofox server:") - _print_info(" npx @askjo/camoufox-browser") + _print_info(" npx @askjo/camofox-browser") _print_info(" First run downloads the Camoufox engine (~300MB)") _print_info(" Or use Docker: docker run -p 9377:9377 -e CAMOFOX_PORT=9377 jo-inc/camofox-browser") elif not shutil.which("npm"): _print_warning(" Node.js not found. Install Camofox via Docker:") _print_info(" docker run -p 9377:9377 -e CAMOFOX_PORT=9377 jo-inc/camofox-browser") + elif post_setup_key == "kittentts": + try: + __import__("kittentts") + _print_success(" kittentts is already installed") + return + except ImportError: + pass + import subprocess + _print_info(" Installing kittentts (~25-80MB model, CPU-only)...") + wheel_url = ( + "https://github.com/KittenML/KittenTTS/releases/download/" + "0.8.1/kittentts-0.8.1-py3-none-any.whl" + ) + try: + result = subprocess.run( + [sys.executable, "-m", "pip", "install", "-U", wheel_url, "soundfile", "--quiet"], + capture_output=True, text=True, timeout=300, + ) + if result.returncode == 0: + _print_success(" kittentts installed") + _print_info(" Voices: Jasper, Bella, Luna, Bruno, Rosie, Hugo, Kiki, Leo") + _print_info(" Models: KittenML/kitten-tts-nano-0.8-int8 (25MB), micro (41MB), mini (80MB)") + else: + _print_warning(" kittentts install failed:") + _print_info(f" {result.stderr.strip()[:300]}") + _print_info(f" Run manually: python -m pip install -U '{wheel_url}' soundfile") + except subprocess.TimeoutExpired: + _print_warning(" kittentts install timed out (>5min)") + _print_info(f" Run manually: python -m pip install -U '{wheel_url}' soundfile") + elif post_setup_key == "rl_training": try: __import__("tinker_atropos") @@ -426,6 +504,8 @@ def _get_enabled_platforms() -> List[str]: enabled.append("slack") if get_env_value("WHATSAPP_ENABLED"): enabled.append("whatsapp") + if get_env_value("QQ_APP_ID"): + enabled.append("qqbot") return enabled @@ -471,7 +551,7 @@ def _get_platform_tools( """Resolve which individual toolset names are enabled for a platform.""" from toolsets import resolve_toolset - platform_toolsets = config.get("platform_toolsets", {}) + platform_toolsets = config.get("platform_toolsets") or {} toolset_names = platform_toolsets.get(platform) if toolset_names is None or not isinstance(toolset_names, list): @@ -505,6 +585,10 @@ def _get_platform_tools( ts_tools = set(resolve_toolset(ts_key)) if ts_tools and ts_tools.issubset(all_tool_names): enabled_toolsets.add(ts_key) + default_off = set(_DEFAULT_OFF_TOOLSETS) + if platform in default_off: + default_off.remove(platform) + enabled_toolsets -= default_off # Plugin toolsets: enabled by default unless explicitly disabled. # A plugin toolset is "known" for a platform once `hermes tools` @@ -763,6 +847,51 @@ def _configure_toolset(ts_key: str, config: dict): _configure_simple_requirements(ts_key) +def _plugin_image_gen_providers() -> list[dict]: + """Build picker-row dicts from plugin-registered image gen providers. + + Each returned dict looks like a regular ``TOOL_CATEGORIES`` provider + row but carries an ``image_gen_plugin_name`` marker so downstream + code (config writing, model picker) knows to route through the + plugin registry instead of the in-tree FAL backend. + + FAL is skipped — it's already exposed by the hardcoded + ``TOOL_CATEGORIES["image_gen"]`` entries. When FAL gets ported to + a plugin in a follow-up PR, the hardcoded entries go away and this + function surfaces it alongside OpenAI automatically. + """ + try: + from agent.image_gen_registry import list_providers + from hermes_cli.plugins import _ensure_plugins_discovered + + _ensure_plugins_discovered() + providers = list_providers() + except Exception: + return [] + + rows: list[dict] = [] + for provider in providers: + if getattr(provider, "name", None) == "fal": + # FAL has its own hardcoded rows today. + continue + try: + schema = provider.get_setup_schema() + except Exception: + continue + if not isinstance(schema, dict): + continue + rows.append( + { + "name": schema.get("name", provider.display_name), + "badge": schema.get("badge", ""), + "tag": schema.get("tag", ""), + "env_vars": schema.get("env_vars", []), + "image_gen_plugin_name": provider.name, + } + ) + return rows + + def _visible_providers(cat: dict, config: dict) -> list[dict]: """Return provider entries visible for the current auth/config state.""" features = get_nous_subscription_features(config) @@ -773,6 +902,12 @@ def _visible_providers(cat: dict, config: dict) -> list[dict]: if provider.get("requires_nous_auth") and not features.nous_auth_present: continue visible.append(provider) + + # Inject plugin-registered image_gen backends (OpenAI today, more + # later) so the picker lists them alongside FAL / Nous Subscription. + if cat.get("name") == "Image Generation": + visible.extend(_plugin_image_gen_providers()) + return visible @@ -792,7 +927,24 @@ def _toolset_needs_configuration_prompt(ts_key: str, config: dict) -> bool: browser_cfg = config.get("browser", {}) return not isinstance(browser_cfg, dict) or "cloud_provider" not in browser_cfg if ts_key == "image_gen": - return not get_env_value("FAL_KEY") + # Satisfied when the in-tree FAL backend is configured OR any + # plugin-registered image gen provider is available. + if fal_key_is_configured(): + return False + try: + from agent.image_gen_registry import list_providers + from hermes_cli.plugins import _ensure_plugins_discovered + + _ensure_plugins_discovered() + for provider in list_providers(): + try: + if provider.is_available(): + return False + except Exception: + continue + except Exception: + pass + return True return not _toolset_has_keys(ts_key, config) @@ -836,7 +988,8 @@ def _configure_tool_category(ts_key: str, cat: dict, config: dict): # Plain text labels only (no ANSI codes in menu items) provider_choices = [] for p in providers: - tag = f" ({p['tag']})" if p.get("tag") else "" + badge = f" [{p['badge']}]" if p.get("badge") else "" + tag = f" — {p['tag']}" if p.get("tag") else "" configured = "" env_vars = p.get("env_vars", []) if not env_vars or all(get_env_value(v["key"]) for v in env_vars): @@ -846,7 +999,7 @@ def _configure_tool_category(ts_key: str, cat: dict, config: dict): configured = "" else: configured = " [configured]" - provider_choices.append(f"{p['name']}{tag}{configured}") + provider_choices.append(f"{p['name']}{badge}{tag}{configured}") # Add skip option provider_choices.append("Skip — keep defaults / configure later") @@ -866,6 +1019,11 @@ def _configure_tool_category(ts_key: str, cat: dict, config: dict): def _is_provider_active(provider: dict, config: dict) -> bool: """Check if a provider entry matches the currently active config.""" + plugin_name = provider.get("image_gen_plugin_name") + if plugin_name: + image_cfg = config.get("image_gen", {}) + return isinstance(image_cfg, dict) and image_cfg.get("provider") == plugin_name + managed_feature = provider.get("managed_nous_feature") if managed_feature: features = get_nous_subscription_features(config) @@ -873,6 +1031,13 @@ def _is_provider_active(provider: dict, config: dict) -> bool: if feature is None: return False if managed_feature == "image_gen": + image_cfg = config.get("image_gen", {}) + if isinstance(image_cfg, dict): + configured_provider = image_cfg.get("provider") + if configured_provider not in (None, "", "fal"): + return False + if image_cfg.get("use_gateway") is False: + return False return feature.managed_by_nous if provider.get("tts_provider"): return ( @@ -895,6 +1060,16 @@ def _is_provider_active(provider: dict, config: dict) -> bool: if provider.get("web_backend"): current = config.get("web", {}).get("backend") return current == provider["web_backend"] + if provider.get("imagegen_backend"): + image_cfg = config.get("image_gen", {}) + if not isinstance(image_cfg, dict): + return False + configured_provider = image_cfg.get("provider") + return ( + provider["imagegen_backend"] == "fal" + and configured_provider in (None, "", "fal") + and not image_cfg.get("use_gateway") + ) return False @@ -910,6 +1085,200 @@ def _detect_active_provider_index(providers: list, config: dict) -> int: return 0 +# ─── Image Generation Model Pickers ─────────────────────────────────────────── +# +# IMAGEGEN_BACKENDS is a per-backend catalog. Each entry exposes: +# - config_key: top-level config.yaml key for this backend's settings +# - model_catalog_fn: returns an OrderedDict-like {model_id: metadata} +# - default_model: fallback when nothing is configured +# +# This prepares for future imagegen backends (Replicate, Stability, etc.): +# each new backend registers its own entry; the FAL provider entry in +# TOOL_CATEGORIES tags itself with `imagegen_backend: "fal"` to select the +# right catalog at picker time. + + +def _fal_model_catalog(): + """Lazy-load the FAL model catalog from the tool module.""" + from tools.image_generation_tool import FAL_MODELS, DEFAULT_MODEL + return FAL_MODELS, DEFAULT_MODEL + + +IMAGEGEN_BACKENDS = { + "fal": { + "display": "FAL.ai", + "config_key": "image_gen", + "catalog_fn": _fal_model_catalog, + }, +} + + +def _format_imagegen_model_row(model_id: str, meta: dict, widths: dict) -> str: + """Format a single picker row with column-aligned speed / strengths / price.""" + return ( + f"{model_id:<{widths['model']}} " + f"{meta.get('speed', ''):<{widths['speed']}} " + f"{meta.get('strengths', ''):<{widths['strengths']}} " + f"{meta.get('price', '')}" + ) + + +def _configure_imagegen_model(backend_name: str, config: dict) -> None: + """Prompt the user to pick a model for the given imagegen backend. + + Writes selection to ``config[backend_config_key]["model"]``. Safe to + call even when stdin is not a TTY — curses_radiolist falls back to + keeping the current selection. + """ + backend = IMAGEGEN_BACKENDS.get(backend_name) + if not backend: + return + + catalog, default_model = backend["catalog_fn"]() + if not catalog: + return + + cfg_key = backend["config_key"] + cur_cfg = config.setdefault(cfg_key, {}) + if not isinstance(cur_cfg, dict): + cur_cfg = {} + config[cfg_key] = cur_cfg + current_model = cur_cfg.get("model") or default_model + if current_model not in catalog: + current_model = default_model + + model_ids = list(catalog.keys()) + # Put current model at the top so the cursor lands on it by default. + ordered = [current_model] + [m for m in model_ids if m != current_model] + + # Column widths + widths = { + "model": max(len(m) for m in model_ids), + "speed": max((len(catalog[m].get("speed", "")) for m in model_ids), default=6), + "strengths": max((len(catalog[m].get("strengths", "")) for m in model_ids), default=0), + } + + print() + header = ( + f" {'Model':<{widths['model']}} " + f"{'Speed':<{widths['speed']}} " + f"{'Strengths':<{widths['strengths']}} " + f"Price" + ) + print(color(header, Colors.CYAN)) + + rows = [] + for mid in ordered: + row = _format_imagegen_model_row(mid, catalog[mid], widths) + if mid == current_model: + row += " ← currently in use" + rows.append(row) + + idx = _prompt_choice( + f" Choose {backend['display']} model:", + rows, + default=0, + ) + + chosen = ordered[idx] + cur_cfg["model"] = chosen + _print_success(f" Model set to: {chosen}") + + +def _plugin_image_gen_catalog(plugin_name: str): + """Return ``(catalog_dict, default_model_id)`` for a plugin provider. + + ``catalog_dict`` is shaped like the legacy ``FAL_MODELS`` table — + ``{model_id: {"display", "speed", "strengths", "price", ...}}`` — + so the existing picker code paths work without change. Returns + ``({}, None)`` if the provider isn't registered or has no models. + """ + try: + from agent.image_gen_registry import get_provider + from hermes_cli.plugins import _ensure_plugins_discovered + + _ensure_plugins_discovered() + provider = get_provider(plugin_name) + except Exception: + return {}, None + if provider is None: + return {}, None + try: + models = provider.list_models() or [] + default = provider.default_model() + except Exception: + return {}, None + catalog = {m["id"]: m for m in models if isinstance(m, dict) and "id" in m} + return catalog, default + + +def _configure_imagegen_model_for_plugin(plugin_name: str, config: dict) -> None: + """Prompt the user to pick a model for a plugin-registered backend. + + Writes selection to ``image_gen.model``. Mirrors + :func:`_configure_imagegen_model` but sources its catalog from the + plugin registry instead of :data:`IMAGEGEN_BACKENDS`. + """ + catalog, default_model = _plugin_image_gen_catalog(plugin_name) + if not catalog: + return + + cur_cfg = config.setdefault("image_gen", {}) + if not isinstance(cur_cfg, dict): + cur_cfg = {} + config["image_gen"] = cur_cfg + current_model = cur_cfg.get("model") or default_model + if current_model not in catalog: + current_model = default_model + + model_ids = list(catalog.keys()) + ordered = [current_model] + [m for m in model_ids if m != current_model] + + widths = { + "model": max(len(m) for m in model_ids), + "speed": max((len(catalog[m].get("speed", "")) for m in model_ids), default=6), + "strengths": max((len(catalog[m].get("strengths", "")) for m in model_ids), default=0), + } + + print() + header = ( + f" {'Model':<{widths['model']}} " + f"{'Speed':<{widths['speed']}} " + f"{'Strengths':<{widths['strengths']}} " + f"Price" + ) + print(color(header, Colors.CYAN)) + + rows = [] + for mid in ordered: + row = _format_imagegen_model_row(mid, catalog[mid], widths) + if mid == current_model: + row += " ← currently in use" + rows.append(row) + + idx = _prompt_choice( + f" Choose {plugin_name} model:", + rows, + default=0, + ) + + chosen = ordered[idx] + cur_cfg["model"] = chosen + _print_success(f" Model set to: {chosen}") + + +def _select_plugin_image_gen_provider(plugin_name: str, config: dict) -> None: + """Persist a plugin-backed image generation provider selection.""" + img_cfg = config.setdefault("image_gen", {}) + if not isinstance(img_cfg, dict): + img_cfg = {} + config["image_gen"] = img_cfg + img_cfg["provider"] = plugin_name + img_cfg["use_gateway"] = False + _print_success(f" image_gen.provider set to: {plugin_name}") + _configure_imagegen_model_for_plugin(plugin_name, config) + + def _configure_provider(provider: dict, config: dict): """Configure a single provider - prompt for API keys and set config.""" env_vars = provider.get("env_vars", []) @@ -923,34 +1292,65 @@ def _configure_provider(provider: dict, config: dict): # Set TTS provider in config if applicable if provider.get("tts_provider"): - config.setdefault("tts", {})["provider"] = provider["tts_provider"] + tts_cfg = config.setdefault("tts", {}) + tts_cfg["provider"] = provider["tts_provider"] + tts_cfg["use_gateway"] = bool(managed_feature) # Set browser cloud provider in config if applicable if "browser_provider" in provider: bp = provider["browser_provider"] + browser_cfg = config.setdefault("browser", {}) if bp == "local": - config.setdefault("browser", {})["cloud_provider"] = "local" + browser_cfg["cloud_provider"] = "local" _print_success(" Browser set to local mode") elif bp: - config.setdefault("browser", {})["cloud_provider"] = bp + browser_cfg["cloud_provider"] = bp _print_success(f" Browser cloud provider set to: {bp}") + browser_cfg["use_gateway"] = bool(managed_feature) # Set web search backend in config if applicable if provider.get("web_backend"): - config.setdefault("web", {})["backend"] = provider["web_backend"] + web_cfg = config.setdefault("web", {}) + web_cfg["backend"] = provider["web_backend"] + web_cfg["use_gateway"] = bool(managed_feature) _print_success(f" Web backend set to: {provider['web_backend']}") + # For tools without a specific config key (e.g. image_gen), still + # track use_gateway so the runtime knows the user's intent. + if managed_feature and managed_feature not in ("web", "tts", "browser"): + config.setdefault(managed_feature, {})["use_gateway"] = True + elif not managed_feature: + # User picked a non-gateway provider — find which category this + # belongs to and clear use_gateway if it was previously set. + for cat_key, cat in TOOL_CATEGORIES.items(): + if provider in cat.get("providers", []): + section = config.get(cat_key) + if isinstance(section, dict) and section.get("use_gateway"): + section["use_gateway"] = False + break + if not env_vars: if provider.get("post_setup"): _run_post_setup(provider["post_setup"]) _print_success(f" {provider['name']} - no configuration needed!") if managed_feature: _print_info(" Requests for this tool will be billed to your Nous subscription.") - override_envs = provider.get("override_env_vars", []) - if any(get_env_value(env_var) for env_var in override_envs): - _print_warning( - " Direct credentials are still configured and may take precedence until you remove them from ~/.hermes/.env." - ) + # Plugin-registered image_gen provider: write image_gen.provider + # and route model selection to the plugin's own catalog. + plugin_name = provider.get("image_gen_plugin_name") + if plugin_name: + _select_plugin_image_gen_provider(plugin_name, config) + return + # Imagegen backends prompt for model selection after backend pick. + backend = provider.get("imagegen_backend") + if backend: + _configure_imagegen_model(backend, config) + # In-tree FAL is the only non-plugin backend today. Keep + # image_gen.provider clear so the dispatch shim falls through + # to the legacy FAL path. + img_cfg = config.setdefault("image_gen", {}) + if isinstance(img_cfg, dict) and img_cfg.get("provider") not in (None, "", "fal"): + img_cfg["provider"] = "fal" return # Prompt for each required env var @@ -985,6 +1385,17 @@ def _configure_provider(provider: dict, config: dict): if all_configured: _print_success(f" {provider['name']} configured!") + plugin_name = provider.get("image_gen_plugin_name") + if plugin_name: + _select_plugin_image_gen_provider(plugin_name, config) + return + # Imagegen backends prompt for model selection after env vars are in. + backend = provider.get("imagegen_backend") + if backend: + _configure_imagegen_model(backend, config) + img_cfg = config.setdefault("image_gen", {}) + if isinstance(img_cfg, dict) and img_cfg.get("provider") not in (None, "", "fal"): + img_cfg["provider"] = "fal" def _configure_simple_requirements(ts_key: str): @@ -1010,17 +1421,17 @@ def _configure_simple_requirements(ts_key: str): _print_warning(" Skipped") elif idx == 1: base_url = _prompt(" OPENAI_BASE_URL (blank for OpenAI)").strip() or "https://api.openai.com/v1" - key_label = " OPENAI_API_KEY" if "api.openai.com" in base_url.lower() else " API key" + is_native_openai = base_url_hostname(base_url) == "api.openai.com" + key_label = " OPENAI_API_KEY" if is_native_openai else " API key" api_key = _prompt(key_label, password=True) if api_key and api_key.strip(): save_env_value("OPENAI_API_KEY", api_key.strip()) # Save vision base URL to config (not .env — only secrets go there) - from hermes_cli.config import load_config, save_config _cfg = load_config() _aux = _cfg.setdefault("auxiliary", {}).setdefault("vision", {}) _aux["base_url"] = base_url save_config(_cfg) - if "api.openai.com" in base_url.lower(): + if is_native_openai: save_env_value("AUXILIARY_VISION_MODEL", "gpt-4o-mini") _print_success(" Saved") else: @@ -1102,7 +1513,8 @@ def _configure_tool_category_for_reconfig(ts_key: str, cat: dict, config: dict): provider_choices = [] for p in providers: - tag = f" ({p['tag']})" if p.get("tag") else "" + badge = f" [{p['badge']}]" if p.get("badge") else "" + tag = f" — {p['tag']}" if p.get("tag") else "" configured = "" env_vars = p.get("env_vars", []) if not env_vars or all(get_env_value(v["key"]) for v in env_vars): @@ -1112,7 +1524,7 @@ def _configure_tool_category_for_reconfig(ts_key: str, cat: dict, config: dict): configured = "" else: configured = " [configured]" - provider_choices.append(f"{p['name']}{tag}{configured}") + provider_choices.append(f"{p['name']}{badge}{tag}{configured}") default_idx = _detect_active_provider_index(providers, config) @@ -1149,17 +1561,39 @@ def _reconfigure_provider(provider: dict, config: dict): config.setdefault("web", {})["backend"] = provider["web_backend"] _print_success(f" Web backend set to: {provider['web_backend']}") + if managed_feature and managed_feature not in ("web", "tts", "browser"): + section = config.setdefault(managed_feature, {}) + if not isinstance(section, dict): + section = {} + config[managed_feature] = section + section["use_gateway"] = True + elif not managed_feature: + for cat_key, cat in TOOL_CATEGORIES.items(): + if provider in cat.get("providers", []): + section = config.get(cat_key) + if isinstance(section, dict) and section.get("use_gateway"): + section["use_gateway"] = False + break + if not env_vars: if provider.get("post_setup"): _run_post_setup(provider["post_setup"]) _print_success(f" {provider['name']} - no configuration needed!") if managed_feature: _print_info(" Requests for this tool will be billed to your Nous subscription.") - override_envs = provider.get("override_env_vars", []) - if any(get_env_value(env_var) for env_var in override_envs): - _print_warning( - " Direct credentials are still configured and may take precedence until you remove them from ~/.hermes/.env." - ) + plugin_name = provider.get("image_gen_plugin_name") + if plugin_name: + _select_plugin_image_gen_provider(plugin_name, config) + return + # Imagegen backends prompt for model selection on reconfig too. + backend = provider.get("imagegen_backend") + if backend: + _configure_imagegen_model(backend, config) + if backend == "fal": + img_cfg = config.setdefault("image_gen", {}) + if isinstance(img_cfg, dict): + img_cfg["provider"] = "fal" + img_cfg["use_gateway"] = False return for var in env_vars: @@ -1177,6 +1611,21 @@ def _reconfigure_provider(provider: dict, config: dict): else: _print_info(" Kept current") + # Imagegen backends prompt for model selection on reconfig too. + plugin_name = provider.get("image_gen_plugin_name") + if plugin_name: + _select_plugin_image_gen_provider(plugin_name, config) + return + + backend = provider.get("imagegen_backend") + if backend: + _configure_imagegen_model(backend, config) + if backend == "fal": + img_cfg = config.setdefault("image_gen", {}) + if isinstance(img_cfg, dict): + img_cfg["provider"] = "fal" + img_cfg["use_gateway"] = False + def _reconfigure_simple_requirements(ts_key: str): """Reconfigure simple env var requirements.""" diff --git a/hermes_cli/uninstall.py b/hermes_cli/uninstall.py index c073598d14db..67cea418209a 100644 --- a/hermes_cli/uninstall.py +++ b/hermes_cli/uninstall.py @@ -7,7 +7,6 @@ """ import os -import platform import shutil import subprocess from pathlib import Path @@ -119,57 +118,164 @@ def remove_wrapper_script(): def uninstall_gateway_service(): - """Stop and uninstall the gateway service if running.""" + """Stop and uninstall the gateway service (systemd, launchd) and kill any + standalone gateway processes. + + Delegates to the gateway module which handles: + - Linux: user + system systemd services (with proper DBUS env setup) + - macOS: launchd plists + - All platforms: standalone ``hermes gateway run`` processes + - Termux/Android: skips systemd (no systemd on Android), still kills standalone processes + """ import platform - - if platform.system() != "Linux": - return False + stopped_something = False + + # 1. Kill any standalone gateway processes (all platforms, including Termux) + try: + from hermes_cli.gateway import kill_gateway_processes, find_gateway_pids + pids = find_gateway_pids() + if pids: + killed = kill_gateway_processes() + if killed: + log_success(f"Killed {killed} running gateway process(es)") + stopped_something = True + except Exception as e: + log_warn(f"Could not check for gateway processes: {e}") + system = platform.system() + + # Termux/Android has no systemd and no launchd — nothing left to do. prefix = os.getenv("PREFIX", "") - if os.getenv("TERMUX_VERSION") or "com.termux/files/usr" in prefix: + is_termux = bool(os.getenv("TERMUX_VERSION") or "com.termux/files/usr" in prefix) + if is_termux: + return stopped_something + + # 2. Linux: uninstall systemd services (both user and system scopes) + if system == "Linux": + try: + from hermes_cli.gateway import ( + get_systemd_unit_path, + get_service_name, + _systemctl_cmd, + ) + svc_name = get_service_name() + + for is_system in (False, True): + unit_path = get_systemd_unit_path(system=is_system) + if not unit_path.exists(): + continue + + scope = "system" if is_system else "user" + try: + if is_system and os.geteuid() != 0: + log_warn(f"System gateway service exists at {unit_path} " + f"but needs sudo to remove") + continue + + cmd = _systemctl_cmd(is_system) + subprocess.run(cmd + ["stop", svc_name], + capture_output=True, check=False) + subprocess.run(cmd + ["disable", svc_name], + capture_output=True, check=False) + unit_path.unlink() + subprocess.run(cmd + ["daemon-reload"], + capture_output=True, check=False) + log_success(f"Removed {scope} gateway service ({unit_path})") + stopped_something = True + except Exception as e: + log_warn(f"Could not remove {scope} gateway service: {e}") + except Exception as e: + log_warn(f"Could not check systemd gateway services: {e}") + + # 3. macOS: uninstall launchd plist + elif system == "Darwin": + try: + from hermes_cli.gateway import get_launchd_plist_path + plist_path = get_launchd_plist_path() + if plist_path.exists(): + subprocess.run(["launchctl", "unload", str(plist_path)], + capture_output=True, check=False) + plist_path.unlink() + log_success(f"Removed macOS gateway service ({plist_path})") + stopped_something = True + except Exception as e: + log_warn(f"Could not remove launchd gateway service: {e}") + + return stopped_something + + +def _is_default_hermes_home(hermes_home: Path) -> bool: + """Return True when ``hermes_home`` points at the default (non-profile) root.""" + try: + from hermes_constants import get_default_hermes_root + return hermes_home.resolve() == get_default_hermes_root().resolve() + except Exception: return False - + + +def _discover_named_profiles(): + """Return a list of ``ProfileInfo`` for every non-default profile, or ``[]`` + if profile support is unavailable or nothing is installed beyond the + default root.""" try: - from hermes_cli.gateway import get_service_name - svc_name = get_service_name() + from hermes_cli.profiles import list_profiles except Exception: - svc_name = "hermes-gateway" + return [] + try: + return [p for p in list_profiles() if not getattr(p, "is_default", False)] + except Exception as e: + log_warn(f"Could not enumerate profiles: {e}") + return [] - service_file = Path.home() / ".config" / "systemd" / "user" / f"{svc_name}.service" - - if not service_file.exists(): - return False - + +def _uninstall_profile(profile) -> None: + """Fully uninstall a single named profile: stop its gateway service, + remove its alias wrapper, and wipe its HERMES_HOME directory. + + We shell out to ``hermes -p gateway stop|uninstall`` because + service names, unit paths, and plist paths are all derived from the + current HERMES_HOME and can't be easily switched in-process. + """ + import sys as _sys + name = profile.name + profile_home = profile.path + + log_info(f"Uninstalling profile '{name}'...") + + # 1. Stop and remove this profile's gateway service. + # Use `python -m hermes_cli.main` so we don't depend on a `hermes` + # wrapper that may be half-removed mid-uninstall. + hermes_invocation = [_sys.executable, "-m", "hermes_cli.main", "--profile", name] + for subcmd in ("stop", "uninstall"): + try: + subprocess.run( + hermes_invocation + ["gateway", subcmd], + capture_output=True, + text=True, + timeout=60, + check=False, + ) + except subprocess.TimeoutExpired: + log_warn(f" Gateway {subcmd} timed out for '{name}'") + except Exception as e: + log_warn(f" Could not run gateway {subcmd} for '{name}': {e}") + + # 2. Remove the wrapper alias script at ~/.local/bin/ (if any). + alias_path = getattr(profile, "alias_path", None) + if alias_path and alias_path.exists(): + try: + alias_path.unlink() + log_success(f" Removed alias {alias_path}") + except Exception as e: + log_warn(f" Could not remove alias {alias_path}: {e}") + + # 3. Wipe the profile's HERMES_HOME directory. try: - # Stop the service - subprocess.run( - ["systemctl", "--user", "stop", svc_name], - capture_output=True, - check=False - ) - - # Disable the service - subprocess.run( - ["systemctl", "--user", "disable", svc_name], - capture_output=True, - check=False - ) - - # Remove service file - service_file.unlink() - - # Reload systemd - subprocess.run( - ["systemctl", "--user", "daemon-reload"], - capture_output=True, - check=False - ) - - return True - + if profile_home.exists(): + shutil.rmtree(profile_home) + log_success(f" Removed {profile_home}") except Exception as e: - log_warn(f"Could not fully remove gateway service: {e}") - return False + log_warn(f" Could not remove {profile_home}: {e}") def run_uninstall(args): @@ -182,7 +288,13 @@ def run_uninstall(args): """ project_root = get_project_root() hermes_home = get_hermes_home() - + + # Detect named profiles when uninstalling from the default root — + # offer to clean them up too instead of leaving zombie HERMES_HOMEs + # and systemd units behind. + is_default_profile = _is_default_hermes_home(hermes_home) + named_profiles = _discover_named_profiles() if is_default_profile else [] + print() print(color("┌─────────────────────────────────────────────────────────┐", Colors.MAGENTA, Colors.BOLD)) print(color("│ ⚕ Hermes Agent Uninstaller │", Colors.MAGENTA, Colors.BOLD)) @@ -196,6 +308,13 @@ def run_uninstall(args): print(f" Secrets: {hermes_home / '.env'}") print(f" Data: {hermes_home / 'cron/'}, {hermes_home / 'sessions/'}, {hermes_home / 'logs/'}") print() + + if named_profiles: + print(color("Other profiles detected:", Colors.CYAN, Colors.BOLD)) + for p in named_profiles: + running = " (gateway running)" if getattr(p, "gateway_running", False) else "" + print(f" • {p.name}{running}: {p.path}") + print() # Ask for confirmation print(color("Uninstall Options:", Colors.YELLOW, Colors.BOLD)) @@ -222,12 +341,40 @@ def run_uninstall(args): return full_uninstall = (choice == "2") - + + # When doing a full uninstall from the default profile, also offer to + # remove any named profiles — stopping their gateway services, unlinking + # their alias wrappers, and wiping their HERMES_HOME dirs. Otherwise + # those leave zombie services and data behind. + remove_profiles = False + if full_uninstall and named_profiles: + print() + print(color("Other profiles will NOT be removed by default.", Colors.YELLOW)) + print(f"Found {len(named_profiles)} named profile(s): " + + ", ".join(p.name for p in named_profiles)) + print() + try: + resp = input(color( + f"Also stop and remove these {len(named_profiles)} profile(s)? [y/N]: ", + Colors.BOLD + )).strip().lower() + except (KeyboardInterrupt, EOFError): + print() + print("Cancelled.") + return + remove_profiles = resp in ("y", "yes") + # Final confirmation print() if full_uninstall: print(color("⚠️ WARNING: This will permanently delete ALL Hermes data!", Colors.RED, Colors.BOLD)) print(color(" Including: configs, API keys, sessions, scheduled jobs, logs", Colors.RED)) + if remove_profiles: + print(color( + f" Plus {len(named_profiles)} profile(s): " + + ", ".join(p.name for p in named_profiles), + Colors.RED + )) else: print("This will remove the Hermes code but keep your configuration and data.") @@ -248,12 +395,10 @@ def run_uninstall(args): print(color("Uninstalling...", Colors.CYAN, Colors.BOLD)) print() - # 1. Stop and uninstall gateway service - log_info("Checking for gateway service...") - if uninstall_gateway_service(): - log_success("Gateway service stopped and removed") - else: - log_info("No gateway service found") + # 1. Stop and uninstall gateway service + kill standalone processes + log_info("Checking for running gateway...") + if not uninstall_gateway_service(): + log_info("No gateway service or processes found") # 2. Remove PATH entries from shell configs log_info("Removing PATH entries from shell configs...") @@ -292,8 +437,17 @@ def run_uninstall(args): log_warn(f"Could not fully remove {project_root}: {e}") log_info("You may need to manually remove it") - # 5. Optionally remove ~/.hermes/ data directory + # 5. Optionally remove ~/.hermes/ data directory (and named profiles) if full_uninstall: + # 5a. Stop and remove each named profile's gateway service and + # alias wrapper. The profile HERMES_HOME dirs live under + # ``/profiles//`` and will be swept away by the + # rmtree below, but services + alias scripts live OUTSIDE the + # default root and have to be cleaned up explicitly. + if remove_profiles and named_profiles: + for prof in named_profiles: + _uninstall_profile(prof) + log_info("Removing configuration and data...") try: if hermes_home.exists(): diff --git a/hermes_cli/web_server.py b/hermes_cli/web_server.py index 77053292e49f..083e0714fda1 100644 --- a/hermes_cli/web_server.py +++ b/hermes_cli/web_server.py @@ -10,10 +10,13 @@ """ import asyncio +import hmac +import importlib.util import json import logging import os import secrets +import subprocess import sys import threading import time @@ -48,16 +51,16 @@ try: from fastapi import FastAPI, HTTPException, Request from fastapi.middleware.cors import CORSMiddleware - from fastapi.responses import FileResponse, JSONResponse + from fastapi.responses import FileResponse, HTMLResponse, JSONResponse from fastapi.staticfiles import StaticFiles from pydantic import BaseModel except ImportError: raise SystemExit( "Web UI requires fastapi and uvicorn.\n" - "Run 'hermes web' to auto-install, or: pip install hermes-agent[web]" + f"Install with: {sys.executable} -m pip install 'fastapi' 'uvicorn[standard]'" ) -WEB_DIST = Path(__file__).parent / "web_dist" +WEB_DIST = Path(os.environ["HERMES_WEB_DIST"]) if "HERMES_WEB_DIST" in os.environ else Path(__file__).parent / "web_dist" _log = logging.getLogger(__name__) app = FastAPI(title="Hermes Agent", version=__version__) @@ -68,6 +71,7 @@ # Injected into the SPA HTML so only the legitimate web UI can use it. # --------------------------------------------------------------------------- _SESSION_TOKEN = secrets.token_urlsafe(32) +_SESSION_HEADER_NAME = "X-Hermes-Session-Token" # Simple rate limiter for the reveal endpoint _reveal_timestamps: List[float] = [] @@ -85,6 +89,145 @@ allow_headers=["*"], ) +# --------------------------------------------------------------------------- +# Endpoints that do NOT require the session token. Everything else under +# /api/ is gated by the auth middleware below. Keep this list minimal — +# only truly non-sensitive, read-only endpoints belong here. +# --------------------------------------------------------------------------- +_PUBLIC_API_PATHS: frozenset = frozenset({ + "/api/status", + "/api/config/defaults", + "/api/config/schema", + "/api/model/info", + "/api/dashboard/themes", + "/api/dashboard/plugins", + "/api/dashboard/plugins/rescan", +}) + + +def _has_valid_session_token(request: Request) -> bool: + """True if the request carries a valid dashboard session token. + + The dedicated session header avoids collisions with reverse proxies that + already use ``Authorization`` (for example Caddy ``basic_auth``). We still + accept the legacy Bearer path for backward compatibility with older + dashboard bundles. + """ + session_header = request.headers.get(_SESSION_HEADER_NAME, "") + if session_header and hmac.compare_digest( + session_header.encode(), + _SESSION_TOKEN.encode(), + ): + return True + + auth = request.headers.get("authorization", "") + expected = f"Bearer {_SESSION_TOKEN}" + return hmac.compare_digest(auth.encode(), expected.encode()) + + +def _require_token(request: Request) -> None: + """Validate the ephemeral session token. Raises 401 on mismatch.""" + if not _has_valid_session_token(request): + raise HTTPException(status_code=401, detail="Unauthorized") + + +# Accepted Host header values for loopback binds. DNS rebinding attacks +# point a victim browser at an attacker-controlled hostname (evil.test) +# which resolves to 127.0.0.1 after a TTL flip — bypassing same-origin +# checks because the browser now considers evil.test and our dashboard +# "same origin". Validating the Host header at the app layer rejects any +# request whose Host isn't one we bound for. See GHSA-ppp5-vxwm-4cf7. +_LOOPBACK_HOST_VALUES: frozenset = frozenset({ + "localhost", "127.0.0.1", "::1", +}) + + +def _is_accepted_host(host_header: str, bound_host: str) -> bool: + """True if the Host header targets the interface we bound to. + + Accepts: + - Exact bound host (with or without port suffix) + - Loopback aliases when bound to loopback + - Any host when bound to 0.0.0.0 (explicit opt-in to non-loopback, + no protection possible at this layer) + """ + if not host_header: + return False + # Strip port suffix. IPv6 addresses use bracket notation: + # [::1] — no port + # [::1]:9119 — with port + # Plain hosts/v4: + # localhost:9119 + # 127.0.0.1:9119 + h = host_header.strip() + if h.startswith("["): + # IPv6 bracketed — port (if any) follows "]:" + close = h.find("]") + if close != -1: + host_only = h[1:close] # strip brackets + else: + host_only = h.strip("[]") + else: + host_only = h.rsplit(":", 1)[0] if ":" in h else h + host_only = host_only.lower() + + # 0.0.0.0 bind means operator explicitly opted into all-interfaces + # (requires --insecure per web_server.start_server). No Host-layer + # defence can protect that mode; rely on operator network controls. + if bound_host in ("0.0.0.0", "::"): + return True + + # Loopback bind: accept the loopback names + bound_lc = bound_host.lower() + if bound_lc in _LOOPBACK_HOST_VALUES: + return host_only in _LOOPBACK_HOST_VALUES + + # Explicit non-loopback bind: require exact host match + return host_only == bound_lc + + +@app.middleware("http") +async def host_header_middleware(request: Request, call_next): + """Reject requests whose Host header doesn't match the bound interface. + + Defends against DNS rebinding: a victim browser on a localhost + dashboard is tricked into fetching from an attacker hostname that + TTL-flips to 127.0.0.1. CORS and same-origin checks don't help — + the browser now treats the attacker origin as same-origin with the + dashboard. Host-header validation at the app layer catches it. + + See GHSA-ppp5-vxwm-4cf7. + """ + # Store the bound host on app.state so this middleware can read it — + # set by start_server() at listen time. + bound_host = getattr(app.state, "bound_host", None) + if bound_host: + host_header = request.headers.get("host", "") + if not _is_accepted_host(host_header, bound_host): + return JSONResponse( + status_code=400, + content={ + "detail": ( + "Invalid Host header. Dashboard requests must use " + "the hostname the server was bound to." + ), + }, + ) + return await call_next(request) + + +@app.middleware("http") +async def auth_middleware(request: Request, call_next): + """Require the session token on all /api/ routes except the public list.""" + path = request.url.path + if path.startswith("/api/") and path not in _PUBLIC_API_PATHS and not path.startswith("/api/plugins/"): + if not _has_valid_session_token(request): + return JSONResponse( + status_code=401, + content={"detail": "Unauthorized"}, + ) + return await call_next(request) + # --------------------------------------------------------------------------- # Config schema — auto-generated from DEFAULT_CONFIG @@ -97,6 +240,11 @@ "description": "Default model (e.g. anthropic/claude-sonnet-4.6)", "category": "general", }, + "model_context_length": { + "type": "number", + "description": "Context window override (0 = auto-detect from model metadata)", + "category": "general", + }, "terminal.backend": { "type": "select", "description": "Terminal execution backend", @@ -122,6 +270,11 @@ "description": "CLI visual theme", "options": ["default", "ares", "mono", "slate"], }, + "dashboard.theme": { + "type": "select", + "description": "Web dashboard visual theme", + "options": ["default", "midnight", "ember", "mono", "cyberpunk", "rose"], + }, "display.resume_display": { "type": "select", "description": "How resumed sessions display history", @@ -179,7 +332,8 @@ "checkpoints": "agent", "approvals": "security", "human_delay": "display", - "smart_model_routing": "agent", + "dashboard": "display", + "code_execution": "agent", } # Display order for tabs — unlisted categories sort alphabetically after these. @@ -247,6 +401,17 @@ def _build_schema_from_config( CONFIG_SCHEMA = _build_schema_from_config(DEFAULT_CONFIG) +# Inject virtual fields that don't live in DEFAULT_CONFIG but are surfaced +# by the normalize/denormalize cycle. Insert model_context_length right after +# the "model" key so it renders adjacent in the frontend. +_mcl_entry = _SCHEMA_OVERRIDES["model_context_length"] +_ordered_schema: Dict[str, Dict[str, Any]] = {} +for _k, _v in CONFIG_SCHEMA.items(): + _ordered_schema[_k] = _v + if _k == "model": + _ordered_schema["model_context_length"] = _mcl_entry +CONFIG_SCHEMA = _ordered_schema + class ConfigUpdate(BaseModel): config: dict @@ -265,12 +430,75 @@ class EnvVarReveal(BaseModel): key: str +_GATEWAY_HEALTH_URL = os.getenv("GATEWAY_HEALTH_URL") +try: + _GATEWAY_HEALTH_TIMEOUT = float(os.getenv("GATEWAY_HEALTH_TIMEOUT", "3")) +except (ValueError, TypeError): + _log.warning( + "Invalid GATEWAY_HEALTH_TIMEOUT value %r — using default 3.0s", + os.getenv("GATEWAY_HEALTH_TIMEOUT"), + ) + _GATEWAY_HEALTH_TIMEOUT = 3.0 + + +def _probe_gateway_health() -> tuple[bool, dict | None]: + """Probe the gateway via its HTTP health endpoint (cross-container). + + Uses ``/health/detailed`` first (returns full state), falling back to + the simpler ``/health`` endpoint. Returns ``(is_alive, body_dict)``. + + Accepts any of these as ``GATEWAY_HEALTH_URL``: + - ``http://gateway:8642`` (base URL — recommended) + - ``http://gateway:8642/health`` (explicit health path) + - ``http://gateway:8642/health/detailed`` (explicit detailed path) + + This is a **blocking** call — run via ``run_in_executor`` from async code. + """ + if not _GATEWAY_HEALTH_URL: + return False, None + + # Normalise to base URL so we always probe the right paths regardless of + # whether the user included /health or /health/detailed in the env var. + base = _GATEWAY_HEALTH_URL.rstrip("/") + if base.endswith("/health/detailed"): + base = base[: -len("/health/detailed")] + elif base.endswith("/health"): + base = base[: -len("/health")] + + for path in (f"{base}/health/detailed", f"{base}/health"): + try: + req = urllib.request.Request(path, method="GET") + with urllib.request.urlopen(req, timeout=_GATEWAY_HEALTH_TIMEOUT) as resp: + if resp.status == 200: + body = json.loads(resp.read()) + return True, body + except Exception: + continue + return False, None + + @app.get("/api/status") async def get_status(): current_ver, latest_ver = check_config_version() + # --- Gateway liveness detection --- + # Try local PID check first (same-host). If that fails and a remote + # GATEWAY_HEALTH_URL is configured, probe the gateway over HTTP so the + # dashboard works when the gateway runs in a separate container. gateway_pid = get_running_pid() gateway_running = gateway_pid is not None + remote_health_body: dict | None = None + + if not gateway_running and _GATEWAY_HEALTH_URL: + loop = asyncio.get_event_loop() + alive, remote_health_body = await loop.run_in_executor( + None, _probe_gateway_health + ) + if alive: + gateway_running = True + # PID from the remote container (display only — not locally valid) + if remote_health_body: + gateway_pid = remote_health_body.get("pid") gateway_state = None gateway_platforms: dict = {} @@ -287,7 +515,12 @@ async def get_status(): except Exception: configured_gateway_platforms = None + # Prefer the detailed health endpoint response (has full state) when the + # local runtime status file is absent or stale (cross-container). runtime = read_runtime_status() + if runtime is None and remote_health_body and remote_health_body.get("gateway_state"): + runtime = remote_health_body + if runtime: gateway_state = runtime.get("gateway_state") gateway_platforms = runtime.get("platforms") or {} @@ -302,6 +535,17 @@ async def get_status(): if not gateway_running: gateway_state = gateway_state if gateway_state in ("stopped", "startup_failed") else "stopped" gateway_platforms = {} + elif gateway_running and remote_health_body is not None: + # The health probe confirmed the gateway is alive, but the local + # runtime status file may be stale (cross-container). Override + # stopped/None state so the dashboard shows the correct badge. + if gateway_state in (None, "stopped"): + gateway_state = "running" + + # If there was no runtime info at all but the health probe confirmed alive, + # ensure we still report the gateway as running (no shared volume scenario). + if gateway_running and gateway_state is None and remote_health_body is not None: + gateway_state = "running" active_sessions = 0 try: @@ -330,6 +574,7 @@ async def get_status(): "latest_config_version": latest_ver, "gateway_running": gateway_running, "gateway_pid": gateway_pid, + "gateway_health_url": _GATEWAY_HEALTH_URL, "gateway_state": gateway_state, "gateway_platforms": gateway_platforms, "gateway_exit_reason": gateway_exit_reason, @@ -338,6 +583,138 @@ async def get_status(): } +# --------------------------------------------------------------------------- +# Gateway + update actions (invoked from the Status page). +# +# Both commands are spawned as detached subprocesses so the HTTP request +# returns immediately. stdin is closed (``DEVNULL``) so any stray ``input()`` +# calls fail fast with EOF rather than hanging forever. stdout/stderr are +# streamed to a per-action log file under ``~/.hermes/logs/.log`` so +# the dashboard can tail them back to the user. +# --------------------------------------------------------------------------- + +_ACTION_LOG_DIR: Path = get_hermes_home() / "logs" + +# Short ``name`` (from the URL) → absolute log file path. +_ACTION_LOG_FILES: Dict[str, str] = { + "gateway-restart": "gateway-restart.log", + "hermes-update": "hermes-update.log", +} + +# ``name`` → most recently spawned Popen handle. Used so ``status`` can +# report liveness and exit code without shelling out to ``ps``. +_ACTION_PROCS: Dict[str, subprocess.Popen] = {} + + +def _spawn_hermes_action(subcommand: List[str], name: str) -> subprocess.Popen: + """Spawn ``hermes `` detached and record the Popen handle. + + Uses the running interpreter's ``hermes_cli.main`` module so the action + inherits the same venv/PYTHONPATH the web server is using. + """ + log_file_name = _ACTION_LOG_FILES[name] + _ACTION_LOG_DIR.mkdir(parents=True, exist_ok=True) + log_path = _ACTION_LOG_DIR / log_file_name + log_file = open(log_path, "ab", buffering=0) + log_file.write( + f"\n=== {name} started {time.strftime('%Y-%m-%d %H:%M:%S')} ===\n".encode() + ) + + cmd = [sys.executable, "-m", "hermes_cli.main", *subcommand] + + popen_kwargs: Dict[str, Any] = { + "cwd": str(PROJECT_ROOT), + "stdin": subprocess.DEVNULL, + "stdout": log_file, + "stderr": subprocess.STDOUT, + "env": {**os.environ, "HERMES_NONINTERACTIVE": "1"}, + } + if sys.platform == "win32": + popen_kwargs["creationflags"] = ( + subprocess.CREATE_NEW_PROCESS_GROUP # type: ignore[attr-defined] + | getattr(subprocess, "DETACHED_PROCESS", 0) + ) + else: + popen_kwargs["start_new_session"] = True + + proc = subprocess.Popen(cmd, **popen_kwargs) + _ACTION_PROCS[name] = proc + return proc + + +def _tail_lines(path: Path, n: int) -> List[str]: + """Return the last ``n`` lines of ``path``. Reads the whole file — fine + for our small per-action logs. Binary-decoded with ``errors='replace'`` + so log corruption doesn't 500 the endpoint.""" + if not path.exists(): + return [] + try: + text = path.read_text(errors="replace") + except OSError: + return [] + lines = text.splitlines() + return lines[-n:] if n > 0 else lines + + +@app.post("/api/gateway/restart") +async def restart_gateway(): + """Kick off a ``hermes gateway restart`` in the background.""" + try: + proc = _spawn_hermes_action(["gateway", "restart"], "gateway-restart") + except Exception as exc: + _log.exception("Failed to spawn gateway restart") + raise HTTPException(status_code=500, detail=f"Failed to restart gateway: {exc}") + return { + "ok": True, + "pid": proc.pid, + "name": "gateway-restart", + } + + +@app.post("/api/hermes/update") +async def update_hermes(): + """Kick off ``hermes update`` in the background.""" + try: + proc = _spawn_hermes_action(["update"], "hermes-update") + except Exception as exc: + _log.exception("Failed to spawn hermes update") + raise HTTPException(status_code=500, detail=f"Failed to start update: {exc}") + return { + "ok": True, + "pid": proc.pid, + "name": "hermes-update", + } + + +@app.get("/api/actions/{name}/status") +async def get_action_status(name: str, lines: int = 200): + """Tail an action log and report whether the process is still running.""" + log_file_name = _ACTION_LOG_FILES.get(name) + if log_file_name is None: + raise HTTPException(status_code=404, detail=f"Unknown action: {name}") + + log_path = _ACTION_LOG_DIR / log_file_name + tail = _tail_lines(log_path, min(max(lines, 1), 2000)) + + proc = _ACTION_PROCS.get(name) + if proc is None: + running = False + exit_code: Optional[int] = None + pid: Optional[int] = None + else: + exit_code = proc.poll() + running = exit_code is None + pid = proc.pid + + return { + "name": name, + "running": running, + "exit_code": exit_code, + "pid": pid, + "lines": tail, + } + + @app.get("/api/sessions") async def get_sessions(limit: int = 20, offset: int = 0): try: @@ -409,11 +786,19 @@ def _normalize_config_for_web(config: Dict[str, Any]) -> Dict[str, Any]: or a dict (``{default: ..., provider: ..., base_url: ...}``). The schema is built from DEFAULT_CONFIG where ``model`` is a string, but user configs often have the dict form. Normalize to the string form so the frontend schema matches. + + Also surfaces ``model_context_length`` as a top-level field so the web UI can + display and edit it. A value of 0 means "auto-detect". """ config = dict(config) # shallow copy model_val = config.get("model") if isinstance(model_val, dict): + # Extract context_length before flattening the dict + ctx_len = model_val.get("context_length", 0) config["model"] = model_val.get("default", model_val.get("name", "")) + config["model_context_length"] = ctx_len if isinstance(ctx_len, int) else 0 + else: + config["model_context_length"] = 0 return config @@ -434,6 +819,93 @@ async def get_schema(): return {"fields": CONFIG_SCHEMA, "category_order": _CATEGORY_ORDER} +_EMPTY_MODEL_INFO: dict = { + "model": "", + "provider": "", + "auto_context_length": 0, + "config_context_length": 0, + "effective_context_length": 0, + "capabilities": {}, +} + + +@app.get("/api/model/info") +def get_model_info(): + """Return resolved model metadata for the currently configured model. + + Calls the same context-length resolution chain the agent uses, so the + frontend can display "Auto-detected: 200K" alongside the override field. + Also returns model capabilities (vision, reasoning, tools) when available. + """ + try: + cfg = load_config() + model_cfg = cfg.get("model", "") + + # Extract model name and provider from the config + if isinstance(model_cfg, dict): + model_name = model_cfg.get("default", model_cfg.get("name", "")) + provider = model_cfg.get("provider", "") + base_url = model_cfg.get("base_url", "") + config_ctx = model_cfg.get("context_length") + else: + model_name = str(model_cfg) if model_cfg else "" + provider = "" + base_url = "" + config_ctx = None + + if not model_name: + return dict(_EMPTY_MODEL_INFO, provider=provider) + + # Resolve auto-detected context length (pass config_ctx=None to get + # purely auto-detected value, then separately report the override) + try: + from agent.model_metadata import get_model_context_length + auto_ctx = get_model_context_length( + model=model_name, + base_url=base_url, + provider=provider, + config_context_length=None, # ignore override — we want auto value + ) + except Exception: + auto_ctx = 0 + + config_ctx_int = 0 + if isinstance(config_ctx, int) and config_ctx > 0: + config_ctx_int = config_ctx + + # Effective is what the agent actually uses + effective_ctx = config_ctx_int if config_ctx_int > 0 else auto_ctx + + # Try to get model capabilities from models.dev + caps = {} + try: + from agent.models_dev import get_model_capabilities + mc = get_model_capabilities(provider=provider, model=model_name) + if mc is not None: + caps = { + "supports_tools": mc.supports_tools, + "supports_vision": mc.supports_vision, + "supports_reasoning": mc.supports_reasoning, + "context_window": mc.context_window, + "max_output_tokens": mc.max_output_tokens, + "model_family": mc.model_family, + } + except Exception: + pass + + return { + "model": model_name, + "provider": provider, + "auto_context_length": auto_ctx, + "config_context_length": config_ctx_int, + "effective_context_length": effective_ctx, + "capabilities": caps, + } + except Exception: + _log.exception("GET /api/model/info failed") + return dict(_EMPTY_MODEL_INFO) + + def _denormalize_config_from_web(config: Dict[str, Any]) -> Dict[str, Any]: """Reverse _normalize_config_for_web before saving. @@ -441,12 +913,24 @@ def _denormalize_config_from_web(config: Dict[str, Any]) -> Dict[str, Any]: to recover model subkeys (provider, base_url, api_mode, etc.) that were stripped from the GET response. The frontend only sees model as a flat string; the rest is preserved transparently. + + Also handles ``model_context_length`` — writes it back into the model dict + as ``context_length``. A value of 0 or absent means "auto-detect" (omitted + from the dict so get_model_context_length() uses its normal resolution). """ config = dict(config) # Remove any _model_meta that might have leaked in (shouldn't happen # with the stripped GET response, but be defensive) config.pop("_model_meta", None) + # Extract and remove model_context_length before processing model + ctx_override = config.pop("model_context_length", 0) + if not isinstance(ctx_override, int): + try: + ctx_override = int(ctx_override) + except (TypeError, ValueError): + ctx_override = 0 + model_val = config.get("model") if isinstance(model_val, str) and model_val: # Read the current disk config to recover model subkeys @@ -456,7 +940,20 @@ def _denormalize_config_from_web(config: Dict[str, Any]) -> Dict[str, Any]: if isinstance(disk_model, dict): # Preserve all subkeys, update default with the new value disk_model["default"] = model_val + # Write context_length into the model dict (0 = remove/auto) + if ctx_override > 0: + disk_model["context_length"] = ctx_override + else: + disk_model.pop("context_length", None) config["model"] = disk_model + else: + # Model was previously a bare string — upgrade to dict if + # user is setting a context_length override + if ctx_override > 0: + config["model"] = { + "default": model_val, + "context_length": ctx_override, + } except Exception: pass # can't read disk config — just use the string form return config @@ -472,17 +969,6 @@ async def update_config(body: ConfigUpdate): raise HTTPException(status_code=500, detail="Internal server error") -@app.get("/api/auth/session-token") -async def get_session_token(): - """Return the ephemeral session token for this server instance. - - The token protects sensitive endpoints (reveal). It's served to the SPA - which stores it in memory — it's never persisted and dies when the server - process exits. CORS already restricts this to localhost origins. - """ - return {"token": _SESSION_TOKEN} - - @app.get("/api/env") async def get_env_vars(): env_on_disk = load_env() @@ -536,9 +1022,7 @@ async def reveal_env_var(body: EnvVarReveal, request: Request): - Audit logging """ # --- Token check --- - auth = request.headers.get("authorization", "") - if auth != f"Bearer {_SESSION_TOKEN}": - raise HTTPException(status_code=401, detail="Unauthorized") + _require_token(request) # --- Rate limit --- now = time.time() @@ -809,9 +1293,7 @@ async def list_oauth_providers(): @app.delete("/api/providers/oauth/{provider_id}") async def disconnect_oauth_provider(provider_id: str, request: Request): """Disconnect an OAuth provider. Token-protected (matches /env/reveal).""" - auth = request.headers.get("authorization", "") - if auth != f"Bearer {_SESSION_TOKEN}": - raise HTTPException(status_code=401, detail="Unauthorized") + _require_token(request) valid_ids = {p["id"] for p in _OAUTH_PROVIDER_CATALOG} if provider_id not in valid_ids: @@ -1201,22 +1683,8 @@ def _nous_poller(session_id: str) -> None: auth_state, min_key_ttl_seconds=300, timeout_seconds=15.0, force_refresh=False, force_mint=True, ) - # Save into credential pool same as auth_commands.py does - from agent.credential_pool import ( - PooledCredential, - load_pool, - AUTH_TYPE_OAUTH, - SOURCE_MANUAL, - ) - pool = load_pool("nous") - entry = PooledCredential.from_dict("nous", { - **full_state, - "label": "dashboard device_code", - "auth_type": AUTH_TYPE_OAUTH, - "source": f"{SOURCE_MANUAL}:dashboard_device_code", - "base_url": full_state.get("inference_base_url"), - }) - pool.add_entry(entry) + from hermes_cli.auth import persist_nous_credentials + persist_nous_credentials(full_state) with _oauth_sessions_lock: sess["status"] = "approved" _log.info("oauth/device: nous login completed (session=%s)", session_id) @@ -1367,9 +1835,7 @@ def _codex_full_login_worker(session_id: str) -> None: @app.post("/api/providers/oauth/{provider_id}/start") async def start_oauth_login(provider_id: str, request: Request): """Initiate an OAuth login flow. Token-protected.""" - auth = request.headers.get("authorization", "") - if auth != f"Bearer {_SESSION_TOKEN}": - raise HTTPException(status_code=401, detail="Unauthorized") + _require_token(request) _gc_oauth_sessions() valid = {p["id"] for p in _OAUTH_PROVIDER_CATALOG} if provider_id not in valid: @@ -1401,9 +1867,7 @@ class OAuthSubmitBody(BaseModel): @app.post("/api/providers/oauth/{provider_id}/submit") async def submit_oauth_code(provider_id: str, body: OAuthSubmitBody, request: Request): """Submit the auth code for PKCE flows. Token-protected.""" - auth = request.headers.get("authorization", "") - if auth != f"Bearer {_SESSION_TOKEN}": - raise HTTPException(status_code=401, detail="Unauthorized") + _require_token(request) if provider_id == "anthropic": return await asyncio.get_event_loop().run_in_executor( None, _submit_anthropic_pkce, body.session_id, body.code, @@ -1431,9 +1895,7 @@ async def poll_oauth_session(provider_id: str, session_id: str): @app.delete("/api/providers/oauth/sessions/{session_id}") async def cancel_oauth_session(session_id: str, request: Request): """Cancel a pending OAuth session. Token-protected.""" - auth = request.headers.get("authorization", "") - if auth != f"Bearer {_SESSION_TOKEN}": - raise HTTPException(status_code=401, detail="Unauthorized") + _require_token(request) with _oauth_sessions_lock: sess = _oauth_sessions.pop(session_id, None) if sess is None: @@ -1735,6 +2197,8 @@ async def update_config_raw(body: RawConfigUpdate): @app.get("/api/analytics/usage") async def get_usage_analytics(days: int = 30): from hermes_state import SessionDB + from agent.insights import InsightsEngine + db = SessionDB() try: cutoff = time.time() - (days * 86400) @@ -1746,7 +2210,8 @@ async def get_usage_analytics(days: int = 30): SUM(reasoning_tokens) as reasoning_tokens, COALESCE(SUM(estimated_cost_usd), 0) as estimated_cost, COALESCE(SUM(actual_cost_usd), 0) as actual_cost, - COUNT(*) as sessions + COUNT(*) as sessions, + SUM(COALESCE(api_call_count, 0)) as api_calls FROM sessions WHERE started_at > ? GROUP BY day ORDER BY day """, (cutoff,)) @@ -1757,7 +2222,8 @@ async def get_usage_analytics(days: int = 30): SUM(input_tokens) as input_tokens, SUM(output_tokens) as output_tokens, COALESCE(SUM(estimated_cost_usd), 0) as estimated_cost, - COUNT(*) as sessions + COUNT(*) as sessions, + SUM(COALESCE(api_call_count, 0)) as api_calls FROM sessions WHERE started_at > ? AND model IS NOT NULL GROUP BY model ORDER BY SUM(input_tokens) + SUM(output_tokens) DESC """, (cutoff,)) @@ -1770,18 +2236,40 @@ async def get_usage_analytics(days: int = 30): SUM(reasoning_tokens) as total_reasoning, COALESCE(SUM(estimated_cost_usd), 0) as total_estimated_cost, COALESCE(SUM(actual_cost_usd), 0) as total_actual_cost, - COUNT(*) as total_sessions + COUNT(*) as total_sessions, + SUM(COALESCE(api_call_count, 0)) as total_api_calls FROM sessions WHERE started_at > ? """, (cutoff,)) totals = dict(cur3.fetchone()) + insights_report = InsightsEngine(db).generate(days=days) + skills = insights_report.get("skills", { + "summary": { + "total_skill_loads": 0, + "total_skill_edits": 0, + "total_skill_actions": 0, + "distinct_skills_used": 0, + }, + "top_skills": [], + }) - return {"daily": daily, "by_model": by_model, "totals": totals, "period_days": days} + return { + "daily": daily, + "by_model": by_model, + "totals": totals, + "period_days": days, + "skills": skills, + } finally: db.close() def mount_spa(application: FastAPI): - """Mount the built SPA. Falls back to index.html for client-side routing.""" + """Mount the built SPA. Falls back to index.html for client-side routing. + + The session token is injected into index.html via a ``' + ) + html = html.replace("", f"{token_script}", 1) + return HTMLResponse( + html, + headers={"Cache-Control": "no-store, no-cache, must-revalidate"}, + ) + application.mount("/assets", StaticFiles(directory=WEB_DIST / "assets"), name="assets") @application.get("/{full_path:path}") @@ -1804,33 +2306,524 @@ async def serve_spa(full_path: str): and file_path.is_file() ): return FileResponse(file_path) - return FileResponse( - WEB_DIST / "index.html", - headers={"Cache-Control": "no-store, no-cache, must-revalidate"}, - ) + return _serve_index() + + +# --------------------------------------------------------------------------- +# Dashboard theme endpoints +# --------------------------------------------------------------------------- + +# Built-in dashboard themes — label + description only. The actual color +# definitions live in the frontend (web/src/themes/presets.ts). +_BUILTIN_DASHBOARD_THEMES = [ + {"name": "default", "label": "Hermes Teal", "description": "Classic dark teal — the canonical Hermes look"}, + {"name": "midnight", "label": "Midnight", "description": "Deep blue-violet with cool accents"}, + {"name": "ember", "label": "Ember", "description": "Warm crimson and bronze — forge vibes"}, + {"name": "mono", "label": "Mono", "description": "Clean grayscale — minimal and focused"}, + {"name": "cyberpunk", "label": "Cyberpunk", "description": "Neon green on black — matrix terminal"}, + {"name": "rose", "label": "Rosé", "description": "Soft pink and warm ivory — easy on the eyes"}, +] + + +def _parse_theme_layer(value: Any, default_hex: str, default_alpha: float = 1.0) -> Optional[Dict[str, Any]]: + """Normalise a theme layer spec from YAML into `{hex, alpha}` form. + + Accepts shorthand (a bare hex string) or full dict form. Returns + ``None`` on garbage input so the caller can fall back to a built-in + default rather than blowing up. + """ + if value is None: + return {"hex": default_hex, "alpha": default_alpha} + if isinstance(value, str): + return {"hex": value, "alpha": default_alpha} + if isinstance(value, dict): + hex_val = value.get("hex", default_hex) + alpha_val = value.get("alpha", default_alpha) + if not isinstance(hex_val, str): + return None + try: + alpha_f = float(alpha_val) + except (TypeError, ValueError): + alpha_f = default_alpha + return {"hex": hex_val, "alpha": max(0.0, min(1.0, alpha_f))} + return None + + +_THEME_DEFAULT_TYPOGRAPHY: Dict[str, str] = { + "fontSans": 'system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif', + "fontMono": 'ui-monospace, "SF Mono", "Cascadia Mono", Menlo, Consolas, monospace', + "baseSize": "15px", + "lineHeight": "1.55", + "letterSpacing": "0", +} + +_THEME_DEFAULT_LAYOUT: Dict[str, str] = { + "radius": "0.5rem", + "density": "comfortable", +} +_THEME_OVERRIDE_KEYS = { + "card", "cardForeground", "popover", "popoverForeground", + "primary", "primaryForeground", "secondary", "secondaryForeground", + "muted", "mutedForeground", "accent", "accentForeground", + "destructive", "destructiveForeground", "success", "warning", + "border", "input", "ring", +} + +# Well-known named asset slots themes can populate. Any other keys under +# ``assets.custom`` are exposed as ``--theme-asset-custom-`` CSS vars +# for plugin/shell use. +_THEME_NAMED_ASSET_KEYS = {"bg", "hero", "logo", "crest", "sidebar", "header"} + +# Component-style buckets themes can override. The value under each bucket +# is a mapping from camelCase property name to CSS string; each pair emits +# ``--component--`` on :root. The frontend's shell +# components (Card, App header, Backdrop, etc.) consume these vars so themes +# can restyle chrome (clip-path, border-image, segmented progress, etc.) +# without shipping their own CSS. +_THEME_COMPONENT_BUCKETS = { + "card", "header", "footer", "sidebar", "tab", + "progress", "badge", "backdrop", "page", +} + +_THEME_LAYOUT_VARIANTS = {"standard", "cockpit", "tiled"} + +# Cap on customCSS length so a malformed/oversized theme YAML can't blow up +# the response payload or the + + +
+

+

+ +
+ + diff --git a/optional-skills/creative/touchdesigner-mcp/SKILL.md b/optional-skills/creative/touchdesigner-mcp/SKILL.md new file mode 100644 index 000000000000..d0bd348afc4e --- /dev/null +++ b/optional-skills/creative/touchdesigner-mcp/SKILL.md @@ -0,0 +1,339 @@ +--- +name: touchdesigner-mcp +description: "Control a running TouchDesigner instance via twozero MCP — create operators, set parameters, wire connections, execute Python, build real-time visuals. 36 native tools." +version: 1.0.0 +author: kshitijk4poor +license: MIT +metadata: + hermes: + tags: [TouchDesigner, MCP, twozero, creative-coding, real-time-visuals, generative-art, audio-reactive, VJ, installation, GLSL] + related_skills: [native-mcp, ascii-video, manim-video, hermes-video] + +--- + +# TouchDesigner Integration (twozero MCP) + +## CRITICAL RULES + +1. **NEVER guess parameter names.** Call `td_get_par_info` for the op type FIRST. Your training data is wrong for TD 2025.32. +2. **If `tdAttributeError` fires, STOP.** Call `td_get_operator_info` on the failing node before continuing. +3. **NEVER hardcode absolute paths** in script callbacks. Use `me.parent()` / `scriptOp.parent()`. +4. **Prefer native MCP tools over td_execute_python.** Use `td_create_operator`, `td_set_operator_pars`, `td_get_errors` etc. Only fall back to `td_execute_python` for complex multi-step logic. +5. **Call `td_get_hints` before building.** It returns patterns specific to the op type you're working with. + +## Architecture + +``` +Hermes Agent -> MCP (Streamable HTTP) -> twozero.tox (port 40404) -> TD Python +``` + +36 native tools. Free plugin (no payment/license — confirmed April 2026). +Context-aware (knows selected OP, current network). +Hub health check: `GET http://localhost:40404/mcp` returns JSON with instance PID, project name, TD version. + +## Setup (Automated) + +Run the setup script to handle everything: + +```bash +bash "${HERMES_HOME:-$HOME/.hermes}/skills/creative/touchdesigner-mcp/scripts/setup.sh" +``` + +The script will: +1. Check if TD is running +2. Download twozero.tox if not already cached +3. Add `twozero_td` MCP server to Hermes config (if missing) +4. Test the MCP connection on port 40404 +5. Report what manual steps remain (drag .tox into TD, enable MCP toggle) + +### Manual steps (one-time, cannot be automated) + +1. **Drag `~/Downloads/twozero.tox` into the TD network editor** → click Install +2. **Enable MCP:** click twozero icon → Settings → mcp → "auto start MCP" → Yes +3. **Restart Hermes session** to pick up the new MCP server + +After setup, verify: +```bash +nc -z 127.0.0.1 40404 && echo "twozero MCP: READY" +``` + +## Environment Notes + +- **Non-Commercial TD** caps resolution at 1280×1280. Use `outputresolution = 'custom'` and set width/height explicitly. +- **Codecs:** `prores` (preferred on macOS) or `mjpa` as fallback. H.264/H.265/AV1 require a Commercial license. +- Always call `td_get_par_info` before setting params — names vary by TD version (see CRITICAL RULES #1). + +## Workflow + +### Step 0: Discover (before building anything) + +``` +Call td_get_par_info with op_type for each type you plan to use. +Call td_get_hints with the topic you're building (e.g. "glsl", "audio reactive", "feedback"). +Call td_get_focus to see where the user is and what's selected. +Call td_get_network to see what already exists. +``` + +No temp nodes, no cleanup. This replaces the old discovery dance entirely. + +### Step 1: Clean + Build + +**IMPORTANT: Split cleanup and creation into SEPARATE MCP calls.** Destroying and recreating same-named nodes in one `td_execute_python` script causes "Invalid OP object" errors. See pitfalls #11b. + +Use `td_create_operator` for each node (handles viewport positioning automatically): + +``` +td_create_operator(type="noiseTOP", parent="/project1", name="bg", parameters={"resolutionw": 1280, "resolutionh": 720}) +td_create_operator(type="levelTOP", parent="/project1", name="brightness") +td_create_operator(type="nullTOP", parent="/project1", name="out") +``` + +For bulk creation or wiring, use `td_execute_python`: + +```python +# td_execute_python script: +root = op('/project1') +nodes = [] +for name, optype in [('bg', noiseTOP), ('fx', levelTOP), ('out', nullTOP)]: + n = root.create(optype, name) + nodes.append(n.path) +# Wire chain +for i in range(len(nodes)-1): + op(nodes[i]).outputConnectors[0].connect(op(nodes[i+1]).inputConnectors[0]) +result = {'created': nodes} +``` + +### Step 2: Set Parameters + +Prefer the native tool (validates params, won't crash): + +``` +td_set_operator_pars(path="/project1/bg", parameters={"roughness": 0.6, "monochrome": true}) +``` + +For expressions or modes, use `td_execute_python`: + +```python +op('/project1/time_driver').par.colorr.expr = "absTime.seconds % 1000.0" +``` + +### Step 3: Wire + +Use `td_execute_python` — no native wire tool exists: + +```python +op('/project1/bg').outputConnectors[0].connect(op('/project1/fx').inputConnectors[0]) +``` + +### Step 4: Verify + +``` +td_get_errors(path="/project1", recursive=true) +td_get_perf() +td_get_operator_info(path="/project1/out", detail="full") +``` + +### Step 5: Display / Capture + +``` +td_get_screenshot(path="/project1/out") +``` + +Or open a window via script: + +```python +win = op('/project1').create(windowCOMP, 'display') +win.par.winop = op('/project1/out').path +win.par.winw = 1280; win.par.winh = 720 +win.par.winopen.pulse() +``` + +## MCP Tool Quick Reference + +**Core (use these most):** +| Tool | What | +|------|------| +| `td_execute_python` | Run arbitrary Python in TD. Full API access. | +| `td_create_operator` | Create node with params + auto-positioning | +| `td_set_operator_pars` | Set params safely (validates, won't crash) | +| `td_get_operator_info` | Inspect one node: connections, params, errors | +| `td_get_operators_info` | Inspect multiple nodes in one call | +| `td_get_network` | See network structure at a path | +| `td_get_errors` | Find errors/warnings recursively | +| `td_get_par_info` | Get param names for an OP type (replaces discovery) | +| `td_get_hints` | Get patterns/tips before building | +| `td_get_focus` | What network is open, what's selected | + +**Read/Write:** +| Tool | What | +|------|------| +| `td_read_dat` | Read DAT text content | +| `td_write_dat` | Write/patch DAT content | +| `td_read_chop` | Read CHOP channel values | +| `td_read_textport` | Read TD console output | + +**Visual:** +| Tool | What | +|------|------| +| `td_get_screenshot` | Capture one OP viewer to file | +| `td_get_screenshots` | Capture multiple OPs at once | +| `td_get_screen_screenshot` | Capture actual screen via TD | +| `td_navigate_to` | Jump network editor to an OP | + +**Search:** +| Tool | What | +|------|------| +| `td_find_op` | Find ops by name/type across project | +| `td_search` | Search code, expressions, string params | + +**System:** +| Tool | What | +|------|------| +| `td_get_perf` | Performance profiling (FPS, slow ops) | +| `td_list_instances` | List all running TD instances | +| `td_get_docs` | In-depth docs on a TD topic | +| `td_agents_md` | Read/write per-COMP markdown docs | +| `td_reinit_extension` | Reload extension after code edit | +| `td_clear_textport` | Clear console before debug session | + +**Input Automation:** +| Tool | What | +|------|------| +| `td_input_execute` | Send mouse/keyboard to TD | +| `td_input_status` | Poll input queue status | +| `td_input_clear` | Stop input automation | +| `td_op_screen_rect` | Get screen coords of a node | +| `td_click_screen_point` | Click a point in a screenshot | + +See `references/mcp-tools.md` for full parameter schemas. + +## Key Implementation Rules + +**GLSL time:** No `uTDCurrentTime` in GLSL TOP. Use the Values page: +```python +# Call td_get_par_info(op_type="glslTOP") first to confirm param names +td_set_operator_pars(path="/project1/shader", parameters={"value0name": "uTime"}) +# Then set expression via script: +# op('/project1/shader').par.value0.expr = "absTime.seconds" +# In GLSL: uniform float uTime; +``` + +Fallback: Constant TOP in `rgba32float` format (8-bit clamps to 0-1, freezing the shader). + +**Feedback TOP:** Use `top` parameter reference, not direct input wire. "Not enough sources" resolves after first cook. "Cook dependency loop" warning is expected. + +**Resolution:** Non-Commercial caps at 1280×1280. Use `outputresolution = 'custom'`. + +**Large shaders:** Write GLSL to `/tmp/file.glsl`, then use `td_write_dat` or `td_execute_python` to load. + +**Vertex/Point access (TD 2025.32):** `point.P[0]`, `point.P[1]`, `point.P[2]` — NOT `.x`, `.y`, `.z`. + +**Extensions:** `ext0object` format is `"op('./datName').module.ClassName(me)"` in CONSTANT mode. After editing extension code with `td_write_dat`, call `td_reinit_extension`. + +**Script callbacks:** ALWAYS use relative paths via `me.parent()` / `scriptOp.parent()`. + +**Cleaning nodes:** Always `list(root.children)` before iterating + `child.valid` check. + +## Recording / Exporting Video + +```python +# via td_execute_python: +root = op('/project1') +rec = root.create(moviefileoutTOP, 'recorder') +op('/project1/out').outputConnectors[0].connect(rec.inputConnectors[0]) +rec.par.type = 'movie' +rec.par.file = '/tmp/output.mov' +rec.par.videocodec = 'prores' # Apple ProRes — NOT license-restricted on macOS +rec.par.record = True # start +# rec.par.record = False # stop (call separately later) +``` + +H.264/H.265/AV1 need Commercial license. Use `prores` on macOS or `mjpa` as fallback. +Extract frames: `ffmpeg -i /tmp/output.mov -vframes 120 /tmp/frames/frame_%06d.png` + +**TOP.save() is useless for animation** — captures same GPU texture every time. Always use MovieFileOut. + +### Before Recording: Checklist + +1. **Verify FPS > 0** via `td_get_perf`. If FPS=0 the recording will be empty. See pitfalls #38-39. +2. **Verify shader output is not black** via `td_get_screenshot`. Black output = shader error or missing input. See pitfalls #8, #40. +3. **If recording with audio:** cue audio to start first, then delay recording by 3 frames. See pitfalls #19. +4. **Set output path before starting record** — setting both in the same script can race. + +## Audio-Reactive GLSL (Proven Recipe) + +### Correct signal chain (tested April 2026) + +``` +AudioFileIn CHOP (playmode=sequential) + → AudioSpectrum CHOP (FFT=512, outputmenu=setmanually, outlength=256, timeslice=ON) + → Math CHOP (gain=10) + → CHOP to TOP (dataformat=r, layout=rowscropped) + → GLSL TOP input 1 (spectrum texture, 256x2) + +Constant TOP (rgba32float, time) → GLSL TOP input 0 +GLSL TOP → Null TOP → MovieFileOut +``` + +### Critical audio-reactive rules (empirically verified) + +1. **TimeSlice must stay ON** for AudioSpectrum. OFF = processes entire audio file → 24000+ samples → CHOP to TOP overflow. +2. **Set Output Length manually** to 256 via `outputmenu='setmanually'` and `outlength=256`. Default outputs 22050 samples. +3. **DO NOT use Lag CHOP for spectrum smoothing.** Lag CHOP operates in timeslice mode and expands 256 samples to 2400+, averaging all values to near-zero (~1e-06). The shader receives no usable data. This was the #1 audio sync failure in testing. +4. **DO NOT use Filter CHOP either** — same timeslice expansion problem with spectrum data. +5. **Smoothing belongs in the GLSL shader** if needed, via temporal lerp with a feedback texture: `mix(prevValue, newValue, 0.3)`. This gives frame-perfect sync with zero pipeline latency. +6. **CHOP to TOP dataformat = 'r'**, layout = 'rowscropped'. Spectrum output is 256x2 (stereo). Sample at y=0.25 for first channel. +7. **Math gain = 10** (not 5). Raw spectrum values are ~0.19 in bass range. Gain of 10 gives usable ~5.0 for the shader. +8. **No Resample CHOP needed.** Control output size via AudioSpectrum's `outlength` param directly. + +### GLSL spectrum sampling + +```glsl +// Input 0 = time (1x1 rgba32float), Input 1 = spectrum (256x2) +float iTime = texture(sTD2DInputs[0], vec2(0.5)).r; + +// Sample multiple points per band and average for stability: +// NOTE: y=0.25 for first channel (stereo texture is 256x2, first row center is 0.25) +float bass = (texture(sTD2DInputs[1], vec2(0.02, 0.25)).r + + texture(sTD2DInputs[1], vec2(0.05, 0.25)).r) / 2.0; +float mid = (texture(sTD2DInputs[1], vec2(0.2, 0.25)).r + + texture(sTD2DInputs[1], vec2(0.35, 0.25)).r) / 2.0; +float hi = (texture(sTD2DInputs[1], vec2(0.6, 0.25)).r + + texture(sTD2DInputs[1], vec2(0.8, 0.25)).r) / 2.0; +``` + +See `references/network-patterns.md` for complete build scripts + shader code. + +## Operator Quick Reference + +| Family | Color | Python class / MCP type | Suffix | +|--------|-------|-------------|--------| +| TOP | Purple | noiseTOP, glslTOP, compositeTOP, levelTop, blurTOP, textTOP, nullTOP | TOP | +| CHOP | Green | audiofileinCHOP, audiospectrumCHOP, mathCHOP, lfoCHOP, constantCHOP | CHOP | +| SOP | Blue | gridSOP, sphereSOP, transformSOP, noiseSOP | SOP | +| DAT | White | textDAT, tableDAT, scriptDAT, webserverDAT | DAT | +| MAT | Yellow | phongMAT, pbrMAT, glslMAT, constMAT | MAT | +| COMP | Gray | geometryCOMP, containerCOMP, cameraCOMP, lightCOMP, windowCOMP | COMP | + +## Security Notes + +- MCP runs on localhost only (port 40404). No authentication — any local process can send commands. +- `td_execute_python` has unrestricted access to the TD Python environment and filesystem as the TD process user. +- `setup.sh` downloads twozero.tox from the official 404zero.com URL. Verify the download if concerned. +- The skill never sends data outside localhost. All MCP communication is local. + +## References + +| File | What | +|------|------| +| `references/pitfalls.md` | Hard-won lessons from real sessions | +| `references/operators.md` | All operator families with params and use cases | +| `references/network-patterns.md` | Recipes: audio-reactive, generative, GLSL, instancing | +| `references/mcp-tools.md` | Full twozero MCP tool parameter schemas | +| `references/python-api.md` | TD Python: op(), scripting, extensions | +| `references/troubleshooting.md` | Connection diagnostics, debugging | +| `scripts/setup.sh` | Automated setup script | + +--- + +> You're not writing code. You're conducting light. diff --git a/optional-skills/creative/touchdesigner-mcp/references/mcp-tools.md b/optional-skills/creative/touchdesigner-mcp/references/mcp-tools.md new file mode 100644 index 000000000000..ec90076cb2bb --- /dev/null +++ b/optional-skills/creative/touchdesigner-mcp/references/mcp-tools.md @@ -0,0 +1,382 @@ +# twozero MCP Tools Reference + +36 tools from twozero MCP v2.774+ (April 2026). +All tools accept an optional `target_instance` param for multi-TD-instance scenarios. + +## Execution & Scripting + +### td_execute_python + +Execute Python code inside TouchDesigner and return the result. Has full access to TD Python API (op, project, app, etc). Print statements and the last expression value are captured. Best for: wiring connections (inputConnectors), setting expressions (par.X.expr/mode), querying parameter names, and batch creation scripts (5+ operators). For creating 1-4 operators, prefer td_create_operator instead. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `code` | string | yes | Python code to execute in TouchDesigner | + +## Network & Structure + +### td_get_network + +Get the operator network structure in TouchDesigner (TD) at a given path. Returns compact list: name OPType flags. First line is full path of queried op. Flags: ch:N=children count, !cook=allowCooking off, bypass, private=isPrivate, blocked:reason, "comment text". depth=0 (default) = current level only. depth=1 = one level of children (indented). To explore deeper, call again on a specific COMP path. System operators (/ui, /sys) are hidden by default. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | no | Network path to inspect, e.g. '/' or '/project1' | +| `depth` | integer | no | How many levels deep to recurse. 0=current level only (recommended), 1=include direct children of COMPs | +| `includeSystem` | boolean | no | Include system operators (/ui, /sys). Default false. | +| `nodeXY` | boolean | no | Include nodeX,nodeY coordinates. Default false. | + +### td_create_operator + +Create a new operator (node) in TouchDesigner (TD). Preferred way to create operators — handles viewport positioning, viewer flag, and docked ops automatically. For batch creation (5+ ops), you may use td_execute_python with a script instead, but then call td_get_hints('construction') first for correct parameter names and layout rules. Supports all TD operator types: TOP, CHOP, SOP, DAT, COMP, MAT. If parent is omitted, creates in the currently open network at the user's viewport position. When building a container: first create baseCOMP (no parent), then create children with parent=compPath. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `type` | string | yes | Operator type, e.g. 'textDAT', 'constantCHOP', 'noiseTOP', 'transformTOP', 'baseCOMP' | +| `parent` | string | no | Path to the parent operator. If omitted, uses the currently open network in TD. | +| `name` | string | no | Name for the new operator (optional, TD auto-names if omitted) | +| `parameters` | object | no | Key-value pairs of parameters to set on the created operator | + +### td_find_op + +Find operators by name and/or type across the project. Returns TSV: path, OPType, flags. Flags: bypass, !cook, private, blocked:reason. Use td_search to search inside code/expressions; use td_find_op to find operators themselves. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | no | Substring to match in operator name (case-insensitive). E.g. 'noise' finds noise1, noise2, myNoise. | +| `type` | string | no | Substring to match in OPType (case-insensitive). E.g. 'noiseTOP', 'baseCOMP', 'CHOP'. Use exact type for precision or partial for broader matches. | +| `root` | string | no | Root operator path to search from. Default '/project1'. | +| `max_results` | number | no | Maximum results to return. Default 50. | +| `max_depth` | number | no | Max recursion depth from root. Default unlimited. | +| `detail` | `basic` / `summary` | no | Result detail level. 'basic' = name/path/type (fast). 'summary' = + connections, non-default pars, expressions. Default 'basic'. | + +### td_search + +Search for text across all code (DAT scripts), parameter expressions, and string parameter values in the TD project. Returns TSV: path, kind (code/expression/parameter/ref), line, text. JSON when context>0. Words are OR-matched. Use quotes for exact phrases: 'GetLogin "op('login')"'. Use count_only=true to quickly check if something is referenced without fetching full results. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `query` | string | yes | Search query. Multiple words = OR (any match). Wrap in quotes for exact phrase. Example: 'GetLogin getLogin' finds either. | +| `root` | string | no | Root operator path to search from. Default '/project1'. | +| `scope` | `all` / `code` / `editable` / `expressions` / `parameters` | no | What to search. 'code' = DAT scripts only (fast, ~0.05s). 'editable' = only editable code (skips inherited/ref DATs). 'expressions' = parameter expressions only. 'parameters' = string parameter values only. 'all' = everything (slow, ~1.5s due to parameter scan). Default 'all'. | +| `case_sensitive` | boolean | no | Case-sensitive matching. Default false. | +| `max_results` | number | no | Maximum results to return. Default 50. | +| `context` | number | no | Lines to show before/after each code match. Saves td_read_dat calls. Default 0. | +| `count_only` | boolean | no | Return only match count, not results. Fast existence check. | +| `max_depth` | number | no | Max recursion depth from root. Default unlimited. | + +### td_navigate_to + +Navigate the TouchDesigner Network Editor viewport to show a specific operator. Opens the operator's parent network and centers the view on it. Use this to show the user where a problem is, or to navigate to an operator before modifying it. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Path to the operator to navigate to, e.g. '/project1/noise1' | + +## Operator Inspection + +### td_get_operator_info + +Get information about a specific operator (node) in TouchDesigner (TD). detail='summary': connections, non-default pars, expressions, CHOP channels (compact). detail='full': all of the above PLUS every parameter with value/default/label. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Full path to the operator, e.g. '/project1/noise1' | +| `detail` | `summary` / `full` | no | Level of detail. 'summary' = connections, expressions, non-default pars, custom pars (pulse marked), CHOP channels. 'full' = summary + all parameters. Default 'full'. | + +### td_get_operators_info + +Get information about multiple operators in one call. Returns an array of operator info objects. Use instead of calling td_get_operator_info multiple times. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `paths` | array | yes | Array of full operator paths, e.g. ['/project1/null1', '/project1/null2'] | +| `detail` | `summary` / `full` | no | Level of detail. Default 'summary'. | + +### td_get_par_info + +Get parameter names and details for a TouchDesigner operator type. Without specific pars: returns compact list of all parameters with their names, types, and menu options. With pars: returns full details (help text, menu values, style) for specific parameters. Use this when you need to know exact parameter names before setting them. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `op_type` | string | yes | TD operator type name, e.g. 'noiseTOP', 'blurTOP', 'lfoCHOP', 'compositeTOP' | +| `pars` | array | no | Optional list of specific parameter names to get full details for | + +## Parameter Setting + +### td_set_operator_pars + +Set parameters and flags on an operator in TouchDesigner (TD). Safer than td_execute_python for simple parameter changes. Can set values, toggle bypass/viewer, without writing Python code. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Path to the operator | +| `parameters` | object | no | Key-value pairs of parameters to set | +| `bypass` | boolean | no | Set bypass state of the operator (not available on COMPs) | +| `viewer` | boolean | no | Set viewer state of the operator | +| `allowCooking` | boolean | no | Set cooking flag on a COMP. When False, internal network stops cooking (0 CPU). COMP-only. | + +## Data Read/Write + +### td_read_dat + +Read the text content of a DAT operator in TouchDesigner (TD). Returns content with line numbers. Use to read scripts, extensions, GLSL shaders, table data. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Path to the DAT operator | +| `start_line` | integer | no | Start line (1-based). Omit to read from beginning. | +| `end_line` | integer | no | End line (inclusive). Omit to read to end. | + +### td_write_dat + +Write or patch text content of a DAT operator in TouchDesigner (TD). Can do full replacement or StrReplace-style patching (old_text -> new_text). Use for editing scripts, extensions, shaders. Does NOT reinit extensions automatically. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Path to the DAT operator | +| `text` | string | no | Full replacement text. Use this OR old_text+new_text, not both. | +| `old_text` | string | no | Text to find and replace (must be unique in the DAT) | +| `new_text` | string | no | Replacement text | +| `replace_all` | boolean | no | If true, replaces ALL occurrences of old_text (default: false, requires unique match) | + +### td_read_chop + +Read CHOP channel sample data. Returns channel values as arrays. Use when you need the actual sample values (animation curves, lookup tables, waveforms), not just the summary from td_get_operator_info. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Path to the CHOP operator | +| `channels` | array | no | Channel names to read. Omit to read all channels. | +| `start` | integer | no | Start sample index (0-based). Omit to read from beginning. | +| `end` | integer | no | End sample index (inclusive). Omit to read to end. | + +### td_read_textport + +Read the last N lines from the TouchDesigner (TD) log/textport (console output). Use this to see errors, warnings and print output from TD. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `lines` | integer | no | Number of recent lines to return | + +### td_clear_textport + +Clear the MCP textport log buffer. Use this before starting a debug session or an edit-run-check loop to keep td_read_textport output focused and minimal. + +No parameters (other than optional `target_instance`). + +## Visual Capture + +### td_get_screenshot + +Get a screenshot of an operator's viewer in TouchDesigner (TD). Saves the image to a file and returns the file path. Use your file-reading tool to view the image. Shows what the operator looks like in its viewer (TOP output, CHOP waveform graph, SOP geometry, DAT table, parameter UI, etc). Use this to visually inspect any operator, or to generate images via TD for use in your project. TWO-STEP ASYNC USAGE: Step 1 — call with 'path' to start: returns {'status': 'pending', 'requestId': '...'}. Step 2 — call with 'request_id' to retrieve: returns {'file': '/tmp/.../opname_id.jpg'}. Then read the file to see the image. If step 2 still returns pending, make one other tool call then retry. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | no | Full operator path to screenshot, e.g. '/project1/noise1'. Required for step 1. | +| `request_id` | string | no | Request ID from step 1 to retrieve the completed screenshot. | +| `max_size` | integer | no | Max pixel size for the longer side (default 512). Use 0 for original operator resolution (useful for pixel-accurate UI work). Higher values (e.g. 1024) for more detail. | +| `output_path` | string | no | Optional absolute path where the image should be saved (e.g. '/Users/me/project/render.png'). If omitted, saved to /tmp/pisang_mcp/screenshots/. Use absolute paths — TD's working directory may differ from the agent's. | +| `as_top` | boolean | no | If true, captures the operator directly as a TOP (bypasses the viewer renderer), preserving alpha/transparency. Only works for TOP operators — if the target is not a TOP, falls back to the viewer automatically. Use this when you need a clean PNG with alpha, e.g. to save a generated image for use in another project. | +| `format` | `auto` / `jpg` / `png` | no | Image format. 'auto' (default): JPEG for viewer mode, PNG for as_top=true. 'jpg': always JPEG (smaller). 'png': always PNG (lossless). | + +### td_get_screenshots + +Get screenshots of multiple operators in one batch. Saves images to files and returns file paths. Use your file-reading tool to view images. TWO-STEP ASYNC USAGE: Step 1 — call with 'paths' array to start: returns {'status': 'pending', 'batchId': '...', 'total': N}. Step 2 — call with 'batch_id' to retrieve: returns {'files': [{op, file}, ...]}. Then read the files to see the images. If still processing returns {'status': 'pending', 'ready': K, 'total': N}. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `paths` | array | no | List of full operator paths to screenshot. Required for step 1. | +| `batch_id` | string | no | Batch ID from step 1 to retrieve completed screenshots. | +| `max_size` | integer | no | Max pixel size for longer side (default 512). Use 0 for original resolution. | +| `as_top` | boolean | no | If true, captures TOP operators directly (preserves alpha). Non-TOP operators fall back to viewer. | +| `output_dir` | string | no | Optional absolute path to a directory. Each screenshot saved as .jpg or .png inside it and kept on disk. | +| `format` | `auto` / `jpg` / `png` | no | Image format. 'auto' (default): JPEG for viewer mode, PNG for as_top=true. 'jpg': always JPEG (smaller). 'png': always PNG (lossless). | + +### td_get_screen_screenshot + +Capture a screenshot of the actual screen via TD's screenGrabTOP. Saves the image to a file and returns the file path. Use your file-reading tool to view the image. Unlike td_get_screenshot (operator viewer), this shows what the user literally sees on their monitor — TD windows, UI panels, everything. Use when simulating mouse/keyboard input to verify what happened on screen. Workflow: td_get_screen_screenshot → read file → td_input_execute → wait idle → td_get_screen_screenshot again. TWO-STEP ASYNC: Step 1 — call without request_id: returns {'status':'pending','requestId':'...'}. Step 2 — call with request_id: returns {'file': '/tmp/.../screen_id.jpg', 'info': '...metadata...'}. Then read the file to see the image. The requestId also stays usable with td_screen_point_to_global for later coordinate lookup. crop_x/y/w/h are in ACTUAL SCREEN PIXELS (not image pixels). Crops exceeding screen bounds are auto-clamped. SMART DEFAULTS: max_size is auto when omitted — 1920 for full screen (good overview), max(crop_w,crop_h) for cropped (guarantees 1:1 scale). At 1:1 scale: screen_coord = crop_origin + image_pixel. Otherwise use the formula from metadata. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `request_id` | string | no | Request ID from step 1 to retrieve the completed screenshot. | +| `max_size` | integer | no | Max pixel size for the longer side. Auto when omitted: 1920 for full screen, max(crop_w,crop_h) for cropped (1:1). Set explicitly to override. | +| `crop_x` | integer | no | Left edge in screen pixels. | +| `crop_y` | integer | no | Top edge in screen pixels (y=0 at top of screen). | +| `crop_w` | integer | no | Width in pixels. | +| `crop_h` | integer | no | Height in pixels. | +| `display` | integer | no | Screen index (default 0 = primary display). | + +## Context & Focus + +### td_get_focus + +Get the current user focus in TouchDesigner (TD): which network is open, selected operators, current operator, and rollover (what is under the mouse cursor). IMPORTANT: when the user says 'this operator' or 'вот этот', they mean the SELECTED/CURRENT operator, NOT the rollover. Rollover is just incidental mouse position and should be ignored for intent. Pass screenshots=true to immediately start a screenshot batch for all selected operators — response includes a 'screenshots' field with batchId; retrieve with td_get_screenshots(batch_id=...). + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `screenshots` | boolean | no | If true, start a screenshot batch for all selected operators. Retrieve with td_get_screenshots(batch_id=...). | +| `max_size` | integer | no | Max screenshot size when screenshots=true (default 512). | +| `as_top` | boolean | no | Passed to the screenshot batch when screenshots=true. | + +### td_get_errors + +Find errors and warnings in TouchDesigner (TD) operators. Checks operator errors, warnings, AND broken parameter expressions (missing channels, bad references, etc). Also includes recent script errors from the log (tracebacks), grouped and deduplicated — e.g. 1000 identical mouse-move errors shown as ×1000 with one entry. If path is given, checks that operator and its children. If no path, checks the currently open network. Use '/' for entire project. Use when user says something is broken, has errors, red nodes, горит ошибка, etc. TIP: call td_clear_textport before reproducing an error to keep log focused. TIP: combine with td_get_perf when user says 'тупит/лагает' to check both errors and performance. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | no | Path to check. If omitted, checks the current network. Use '/' to scan entire project. | +| `recursive` | boolean | no | Check children recursively (default true) | +| `include_log` | boolean | no | Include recent script errors from log, grouped by unique signature (default true). Use td_clear_textport before reproducing an error to keep results focused. | + +### td_get_perf + +Get performance data from TouchDesigner (TD). Returns TSV: header with fps/budget/memory summary, then slowest operators sorted by cook time. Columns: path, OPType, cpu/cook(ms), gpu/cook(ms), cpu/s, gpu/s, rate, flags. Use when user reports lag, low FPS, slow performance, тупит, тормозит. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | no | Path to profile. If omitted, profiles the current network. Use '/' for entire project. | +| `top` | integer | no | Number of slowest operators to return | + +## Documentation + +### td_get_docs + +Get comprehensive documentation on a TouchDesigner topic. Unlike td_get_hints (compact tips), this returns in-depth reference material. Call without arguments to see available topics with descriptions. Call with a topic name to get the full documentation. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `topic` | string | no | Topic to get docs for. Omit to list available topics. | + +### td_get_hints + +Get TouchDesigner tips and common patterns for a topic. Call this BEFORE creating operators or writing TD Python code to learn correct parameter names, expressions, and idiomatic approaches. Available topics: animation, noise, connections, parameters, scripting, construction, ui_analysis, panel_layout, screenshots, input_simulation, undo. IMPORTANT: always call with topic='construction' before building multi-operator setups to get correct TOP/CHOP parameter names, compositeTOP input ordering, and layout guidelines. IMPORTANT: always call with topic='input_simulation' before using td_input_execute to learn focus recovery, coordinate systems, and testing workflow. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `topic` | string | yes | Topic to get hints for. Available: 'animation', 'noise', 'connections', 'parameters', 'scripting', 'construction', 'ui_analysis', 'panel_layout', 'screenshots', 'input_simulation', 'undo', 'networking', 'all' | + +### td_agents_md + +Read, write, or update the agents_md documentation inside a COMP container. agents_md is a Markdown textDAT describing the container's purpose, structure, and conventions. action='read': returns content + staleness check (compares documented children vs live state). action='update': refreshes auto-generated sections (children list, connections) from live state, preserves human-written sections. action='write': sets full content, creates the DAT if missing. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Path to the COMP container | +| `action` | `read` / `update` / `write` | yes | read=get content+staleness, update=refresh auto sections, write=set content | +| `content` | string | no | Markdown content (only for action='write') | + +## Input Automation + +### td_input_execute + +Send a sequence of mouse/keyboard commands to TouchDesigner. Commands execute sequentially with smooth bezier movement. Returns immediately — poll td_input_status() until status='idle' before proceeding. Command types: 'focus' — bring TD to foreground. 'move' — smooth mouse move: {type,x,y,duration,easing}. 'click' — click: {type,x,y,button,hold,duration,easing}. hold=seconds to hold down. duration=smooth move before click. 'dblclick' — double click: {type,x,y,duration}. 'mousedown'/'mouseup' — {type,x,y,button}. 'key' — keystroke: {type,keys} e.g. 'ctrl+z','tab','escape','shift+f5'. Requires Accessibility permission on Mac. 'type' — human-like typing: {type,text,wpm,variance} — layout-independent Unicode, variable timing. 'wait' — pause: {type,duration}. 'scroll' — {type,x,y,dx,dy,steps} — human-like scroll: moves mouse to (x,y) first, then sends dy (vertical, +up) and dx (horizontal, +right) as multiple ticks with natural timing. steps=4 by default. Mouse commands may include coord_space='logical' (default) or coord_space='physical'. On macOS, 'physical' means actual screen pixels from td_get_screen_screenshot and is converted to CGEvent logical coords automatically. Top-level coord_space applies to commands that do not override it. on_error: 'stop' (default) clears queue on error; 'continue' skips failed command. IMPORTANT: call td_get_hints('input_simulation') before first use to learn focus recovery, coordinate systems, and testing workflow. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `commands` | array | yes | List of command dicts to execute in sequence. | +| `coord_space` | `logical` / `physical` | no | Default coordinate space for mouse commands that do not specify their own coord_space. 'logical' uses CGEvent coords directly. 'physical' uses actual screen pixels from td_get_screen_screenshot and is auto-converted on macOS. | +| `on_error` | `stop` / `continue` | no | What to do on error. Default 'stop'. | + +### td_input_status + +Get current status of the td_input command queue. Poll this after td_input_execute until status='idle'. Returns: status ('idle'/'running'), current command, queue_remaining, last error. + +No parameters (other than optional `target_instance`). + +### td_input_clear + +Clear the td_input command queue and stop current execution immediately. + +No parameters (other than optional `target_instance`). + +### td_op_screen_rect + +Get the screen coordinates of an operator node in the network editor. Returns {x,y,w,h,cx,cy} where cx,cy is the center for clicking. Use this to find where to click on a specific operator. Only works if the operator's parent network is currently open in a network editor pane. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Full path to the operator, e.g. '/project1/myComp/noise1' | + +### td_click_screen_point + +Resolve a point inside a previous td_get_screen_screenshot result and click it. Pass the screenshot request_id plus either normalized u/v or image_x/image_y. Queues a td_input click using physical screen coordinates, so it works directly with screenshot-derived points. Use duration/easing to control the cursor travel before the click. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `request_id` | string | yes | Request ID originally returned by td_get_screen_screenshot. | +| `u` | number | no | Normalized horizontal position inside the screenshot region (0=left, 1=right). Use with v. | +| `v` | number | no | Normalized vertical position inside the screenshot region (0=top, 1=bottom). Use with u. | +| `image_x` | number | no | Horizontal pixel coordinate inside the returned screenshot image. Use with image_y. | +| `image_y` | number | no | Vertical pixel coordinate inside the returned screenshot image. Use with image_x. | +| `button` | `left` / `right` / `middle` | no | Mouse button to click. Default left. | +| `hold` | number | no | Seconds to hold the mouse button down before releasing. | +| `duration` | number | no | Seconds for the cursor to travel to the target before clicking. | +| `easing` | `linear` / `ease-in` / `ease-out` / `ease-in-out` | no | Cursor movement easing for the pre-click travel. | +| `focus` | boolean | no | If true, bring TD to the front before clicking and wait briefly for focus to settle. | + +### td_screen_point_to_global + +Convert a point inside a previous td_get_screen_screenshot result into absolute screen coordinates. Pass the screenshot request_id plus either normalized u/v (0..1 inside that screenshot region) or image_x/image_y in returned image pixels. Returns absolute physical screen coordinates, logical coordinates, and a ready-to-use td_input_execute payload. Metadata is kept for the most recent screen screenshots so multiple agents can resolve points later by request_id. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `request_id` | string | yes | Request ID originally returned by td_get_screen_screenshot. | +| `u` | number | no | Normalized horizontal position inside the screenshot region (0=left, 1=right). Use with v. | +| `v` | number | no | Normalized vertical position inside the screenshot region (0=top, 1=bottom). Use with u. | +| `image_x` | number | no | Horizontal pixel coordinate inside the returned screenshot image. Use with image_y. | +| `image_y` | number | no | Vertical pixel coordinate inside the returned screenshot image. Use with image_x. | + +## System + +### td_list_instances + +List all running TouchDesigner (TD) instances with active MCP servers. Returns port, project name, PID, and instanceId for each instance. Call this at the start of every conversation to discover available instances and choose which one to work with. instanceId is stable for the lifetime of a TD process and is used as target_instance in all other tool calls. + +No parameters (other than optional `target_instance`). + +### td_project_quit + +Save and/or close the current TouchDesigner (TD) project. Can save before closing. Reports if project has unsaved changes. To close a different instance, pass target_instance=instanceId. WARNING: this will shut down the MCP server on that instance. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `save` | boolean | no | Save the project before closing. Default true. | +| `force` | boolean | no | Force close without save dialog. Default false. | + +### td_reinit_extension + +Reinitialize an extension on a COMP in TouchDesigner (TD). Call this AFTER finishing all code edits via td_write_dat to apply changes. Do NOT call after every small edit - batch your changes first. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `path` | string | yes | Path to the COMP with the extension | + +### td_dev_log + +Read the last N entries from the MCP dev log. Only available when Devmode is enabled. Shows request/response history. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `count` | integer | no | Number of recent log entries to return | + +### td_clear_dev_log + +Clear the current MCP dev log by closing the old file and starting a fresh one. Only available when Devmode is enabled. + +No parameters (other than optional `target_instance`). + +### td_test_session + +Manage test sessions, bug reports, and conversation export. IMPORTANT: Do NOT proactively suggest exporting chat or submitting reports. These are tools for specific situations: - export_chat / submit_report: ONLY when the user encounters a BUG with the plugin or TouchDesigner and wants to report it, or when the user explicitly asks to export the conversation. Never suggest this at session end or as routine action. USER PHRASES → ACTIONS: 'разбор тестовых сессий' / 'analyze test sessions' → list, then pull, read meta.json → index.jsonl → calls/. 'разбор репортов' / 'analyze user reports' → list with session='user', then pull by name. 'экспортируй чат' / 'export chat' → (1) export_chat_id → marker, (2) export_chat with session=marker. 'сообщи о проблеме' / 'report bug' → export chat, review for privacy, then submit_report with summary + tags + result_op=file_path. ACTIONS: export_chat_id | export_chat | submit_report | start | note | import_chat | end | list | pull. list: default=auto-detect repo. session='user' for user_reports (dev only). pull: auto-searches both repos. Auto-detects dev vs user Hub access. + +| Param | Type | Required | Description | +|-------|------|----------|-------------| +| `action` | `export_chat_id` / `export_chat` / `submit_report` / `start` / `note` / `import_chat` / `end` / `list` / `pull` | yes | Action: export_chat_id / export_chat / submit_report / start / note / import_chat / end / list / pull | +| `prompt` | string | no | (start) The test prompt/task description | +| `tags` | array | no | (start) Tags for categorization, e.g. ['ui', 'layout'] | +| `text` | string | no | (note) Observation text. (import_chat) Full conversation text. | +| `outcome` | `success` / `partial` / `failure` | no | (end) Result: success / partial / failure | +| `summary` | string | no | (end) Brief summary of what happened | +| `result_op` | string | no | (end) Path to operator to save as result.tox | +| `session` | string | no | (pull) Session name or substring to download | diff --git a/optional-skills/creative/touchdesigner-mcp/references/network-patterns.md b/optional-skills/creative/touchdesigner-mcp/references/network-patterns.md new file mode 100644 index 000000000000..cb04fd54d573 --- /dev/null +++ b/optional-skills/creative/touchdesigner-mcp/references/network-patterns.md @@ -0,0 +1,966 @@ +# TouchDesigner Network Patterns + +Complete network recipes for common creative coding tasks. Each pattern shows the operator chain, MCP tool calls to build it, and key parameter settings. + +## Audio-Reactive Visuals + +### Pattern 1: Audio Spectrum -> Noise Displacement + +Audio drives noise parameters for organic, music-responsive textures. + +``` +Audio File In CHOP -> Audio Spectrum CHOP -> Math CHOP (scale) + | + v (export to noise params) + Noise TOP -> Level TOP -> Feedback TOP -> Composite TOP -> Null TOP (out) + ^ | + |________________| +``` + +**MCP Build Sequence:** + +``` +1. td_create_operator(parent="/project1", type="audiofileinChop", name="audio_in") +2. td_create_operator(parent="/project1", type="audiospectrumChop", name="spectrum") +3. td_create_operator(parent="/project1", type="mathChop", name="spectrum_scale") +4. td_create_operator(parent="/project1", type="noiseTop", name="noise1") +5. td_create_operator(parent="/project1", type="levelTop", name="level1") +6. td_create_operator(parent="/project1", type="feedbackTop", name="feedback1") +7. td_create_operator(parent="/project1", type="compositeTop", name="comp1") +8. td_create_operator(parent="/project1", type="nullTop", name="out") + +9. td_set_operator_pars(path="/project1/audio_in", + properties={"file": "/path/to/music.wav", "play": true}) +10. td_set_operator_pars(path="/project1/spectrum", + properties={"size": 512}) +11. td_set_operator_pars(path="/project1/spectrum_scale", + properties={"gain": 2.0, "postoff": 0.0}) +12. td_set_operator_pars(path="/project1/noise1", + properties={"type": 1, "monochrome": false, "resolutionw": 1280, "resolutionh": 720, + "period": 4.0, "harmonics": 3, "amp": 1.0}) +13. td_set_operator_pars(path="/project1/level1", + properties={"opacity": 0.95, "gamma1": 0.75}) +14. td_set_operator_pars(path="/project1/feedback1", + properties={"top": "/project1/comp1"}) +15. td_set_operator_pars(path="/project1/comp1", + properties={"operand": 0}) + +16. td_execute_python: """ +op('/project1/audio_in').outputConnectors[0].connect(op('/project1/spectrum')) +op('/project1/spectrum').outputConnectors[0].connect(op('/project1/spectrum_scale')) +op('/project1/noise1').outputConnectors[0].connect(op('/project1/level1')) +op('/project1/level1').outputConnectors[0].connect(op('/project1/comp1').inputConnectors[0]) +op('/project1/feedback1').outputConnectors[0].connect(op('/project1/comp1').inputConnectors[1]) +op('/project1/comp1').outputConnectors[0].connect(op('/project1/out')) +""" + +17. td_execute_python: """ +# Export spectrum values to drive noise parameters +# This makes the noise react to audio frequencies +op('/project1/noise1').par.seed.expr = "op('/project1/spectrum_scale')['chan1']" +op('/project1/noise1').par.period.expr = "tdu.remap(op('/project1/spectrum_scale')['chan1'].eval(), 0, 1, 1, 8)" +""" +``` + +### Pattern 2: Beat Detection -> Visual Pulses + +Detect beats from audio and trigger visual events. + +``` +Audio Device In CHOP -> Audio Spectrum CHOP -> Math CHOP (isolate bass) + | + Trigger CHOP (envelope) + | + [export to visual params] +``` + +**Key parameter settings:** + +``` +# Isolate bass frequencies (20-200 Hz) +Math CHOP: chanop=1 (Add channels), range1low=0, range1high=10 + (first 10 FFT bins = bass frequencies with 512 FFT at 44100Hz) + +# ADSR envelope on each beat +Trigger CHOP: attack=0.02, peak=1.0, decay=0.3, sustain=0.0, release=0.1 + +# Export to visual: Scale, brightness, or color intensity +td_execute_python: "op('/project1/level1').par.brightness1.expr = \"1.0 + op('/project1/trigger1')['chan1'] * 0.5\"" +``` + +### Pattern 3: Multi-Band Audio -> Multi-Layer Visuals + +Split audio into frequency bands, drive different visual layers per band. + +``` +Audio In -> Spectrum -> Audio Band EQ (3 bands: bass, mid, treble) + | + +---------+---------+ + | | | + Bass Mids Treble + | | | + Noise TOP Circle TOP Text TOP + (slow,dark) (mid,warm) (fast,bright) + | | | + +-----+----+----+----+ + | | + Composite Composite + | + Out +``` + +### Pattern 3b: Audio-Reactive GLSL Fractal (Proven Recipe) + +Complete working recipe. Plays an MP3, runs FFT, feeds spectrum as a texture into a GLSL shader where inner fractal reacts to bass, outer to treble. + +**Network:** +``` +AudioFileIn CHOP → AudioSpectrum CHOP (FFT=512, outlength=256) + → Math CHOP (gain=10) → CHOP To TOP (256x2 spectrum texture, dataformat=r) + ↓ +Constant TOP (time, rgba32float) → GLSL TOP (input 0=time, input 1=spectrum) → Null → MovieFileOut + ↓ +AudioFileIn CHOP → Audio Device Out CHOP Record to .mov +``` + +**Build via td_execute_python (one call per step for reliability):** + +```python +# Step 1: Audio chain +# td_execute_python script: +td_execute_python(code=""" +root = op('/project1') +audio = root.create(audiofileinCHOP, 'audio_in') +audio.par.file = '/path/to/music.mp3' +audio.par.playmode = 0 # Locked to timeline +audio.par.volume = 0.5 + +spec = root.create(audiospectrumCHOP, 'spectrum') +audio.outputConnectors[0].connect(spec.inputConnectors[0]) + +math_n = root.create(mathCHOP, 'math_norm') +spec.outputConnectors[0].connect(math_n.inputConnectors[0]) +math_n.par.gain = 5 # boost signal + +resamp = root.create(resampleCHOP, 'resample_spec') +math_n.outputConnectors[0].connect(resamp.inputConnectors[0]) +resamp.par.timeslice = True +resamp.par.rate = 256 + +chop2top = root.create(choptoTOP, 'spectrum_tex') +chop2top.par.chop = resamp # CHOP To TOP has NO input connectors — use par.chop reference + +# Audio output (hear the music) +aout = root.create(audiodeviceoutCHOP, 'audio_out') +audio.outputConnectors[0].connect(aout.inputConnectors[0]) +result = 'audio chain ok' +""") + +# Step 2: Time driver (MUST be rgba32float — see pitfalls #6) +# td_execute_python script: +td_execute_python(code=""" +root = op('/project1') +td = root.create(constantTOP, 'time_driver') +td.par.format = 'rgba32float' +td.par.outputresolution = 'custom' +td.par.resolutionw = 1 +td.par.resolutionh = 1 +td.par.colorr.expr = "absTime.seconds % 1000.0" +td.par.colorg.expr = "int(absTime.seconds / 1000.0)" +result = 'time ok' +""") + +# Step 3: GLSL shader (write to /tmp, load from file) +# td_execute_python script: +td_execute_python(code=""" +root = op('/project1') +glsl = root.create(glslTOP, 'audio_shader') +glsl.par.outputresolution = 'custom' +glsl.par.resolutionw = 1280 +glsl.par.resolutionh = 720 + +sd = root.create(textDAT, 'shader_code') +sd.text = open('/tmp/my_shader.glsl').read() +glsl.par.pixeldat = sd + +# Wire: input 0 = time, input 1 = spectrum texture +op('/project1/time_driver').outputConnectors[0].connect(glsl.inputConnectors[0]) +op('/project1/spectrum_tex').outputConnectors[0].connect(glsl.inputConnectors[1]) +result = 'glsl ok' +""") + +# Step 4: Output + recorder +# td_execute_python script: +td_execute_python(code=""" +root = op('/project1') +out = root.create(nullTOP, 'output') +op('/project1/audio_shader').outputConnectors[0].connect(out.inputConnectors[0]) + +rec = root.create(moviefileoutTOP, 'recorder') +out.outputConnectors[0].connect(rec.inputConnectors[0]) +rec.par.type = 'movie' +rec.par.file = '/tmp/output.mov' +rec.par.videocodec = 'mjpa' +result = 'output ok' +""") +``` + +**GLSL shader pattern (audio-reactive fractal):** +```glsl +out vec4 fragColor; + +vec3 palette(float t) { + vec3 a = vec3(0.5); vec3 b = vec3(0.5); + vec3 c = vec3(1.0); vec3 d = vec3(0.263, 0.416, 0.557); + return a + b * cos(6.28318 * (c * t + d)); +} + +void main() { + // Input 0 = time (1x1 rgba32float constant) + // Input 1 = audio spectrum (256x2 CHOP To TOP, stereo — sample at y=0.25 for first channel) + vec4 td = texture(sTD2DInputs[0], vec2(0.5)); + float t = td.r + td.g * 1000.0; + + vec2 res = uTDOutputInfo.res.zw; + vec2 uv = (gl_FragCoord.xy * 2.0 - res) / min(res.x, res.y); + vec2 uv0 = uv; + vec3 finalColor = vec3(0.0); + + float bass = texture(sTD2DInputs[1], vec2(0.05, 0.25)).r; + float mids = texture(sTD2DInputs[1], vec2(0.25, 0.25)).r; + + for (float i = 0.0; i < 4.0; i++) { + uv = fract(uv * (1.4 + bass * 0.3)) - 0.5; + float d = length(uv) * exp(-length(uv0)); + + // Sample spectrum at distance: inner=bass, outer=treble + float freq = texture(sTD2DInputs[1], vec2(clamp(d * 0.5, 0.0, 1.0), 0.25)).r; + + vec3 col = palette(length(uv0) + i * 0.4 + t * 0.35); + d = sin(d * (7.0 + bass * 4.0) + t * 1.5) / 8.0; + d = abs(d); + d = pow(0.012 / d, 1.2 + freq * 0.8 + bass * 0.5); + finalColor += col * d; + } + + // Tone mapping + finalColor = finalColor / (finalColor + vec3(1.0)); + fragColor = TDOutputSwizzle(vec4(finalColor, 1.0)); +} +``` + +**Key insights from testing:** +- `spectrum_tex` (CHOP To TOP) produces a 256x2 texture — x position = frequency, y=0.25 for first channel +- Sampling at `vec2(0.05, 0.0)` gets bass, `vec2(0.65, 0.0)` gets treble +- Sampling based on pixel distance (`d * 0.5`) makes inner fractal react to bass, outer to treble +- `bass * 0.3` in the `fract()` zoom makes the fractal breathe with kicks +- Math CHOP gain of 5 is needed because raw spectrum values are very small + +## Generative Art + +### Pattern 4: Feedback Loop with Transform + +Classic generative technique — texture evolves through recursive transformation. + +``` +Noise TOP -> Composite TOP -> Level TOP -> Null TOP (out) + ^ | + | v + Transform TOP <- Feedback TOP +``` + +**MCP Build Sequence:** + +``` +1. td_create_operator(parent="/project1", type="noiseTop", name="seed_noise") +2. td_create_operator(parent="/project1", type="compositeTop", name="mix") +3. td_create_operator(parent="/project1", type="transformTop", name="evolve") +4. td_create_operator(parent="/project1", type="feedbackTop", name="fb") +5. td_create_operator(parent="/project1", type="levelTop", name="color_correct") +6. td_create_operator(parent="/project1", type="nullTop", name="out") + +7. td_set_operator_pars(path="/project1/seed_noise", + properties={"type": 1, "monochrome": false, "period": 2.0, "amp": 0.3, + "resolutionw": 1280, "resolutionh": 720}) +8. td_set_operator_pars(path="/project1/mix", + properties={"operand": 27}) # 27 = Screen blend +9. td_set_operator_pars(path="/project1/evolve", + properties={"sx": 1.003, "sy": 1.003, "rz": 0.5, "extend": 2}) # slight zoom + rotate, repeat edges +10. td_set_operator_pars(path="/project1/fb", + properties={"top": "/project1/mix"}) +11. td_set_operator_pars(path="/project1/color_correct", + properties={"opacity": 0.98, "gamma1": 0.85}) + +12. td_execute_python: """ +op('/project1/seed_noise').outputConnectors[0].connect(op('/project1/mix').inputConnectors[0]) +op('/project1/fb').outputConnectors[0].connect(op('/project1/evolve')) +op('/project1/evolve').outputConnectors[0].connect(op('/project1/mix').inputConnectors[1]) +op('/project1/mix').outputConnectors[0].connect(op('/project1/color_correct')) +op('/project1/color_correct').outputConnectors[0].connect(op('/project1/out')) +""" +``` + +**Variations:** +- Change Transform: `rz` (rotation), `sx/sy` (zoom), `tx/ty` (drift) +- Change Composite operand: Screen (glow), Add (bright), Multiply (dark) +- Add HSV Adjust in the feedback loop for color evolution +- Add Blur for dreamlike softness +- Replace Noise with a GLSL TOP for custom seed patterns + +### Pattern 5: Instancing (Particle-Like Systems) + +Render thousands of copies of geometry, each with unique position/rotation/scale driven by CHOP data or DATs. + +``` +Table DAT (instance data) -> DAT to CHOP -> Geometry COMP (instancing on) -> Render TOP + + Sphere SOP (template geometry) + + Constant MAT (material) + + Camera COMP + + Light COMP +``` + +**MCP Build Sequence:** + +``` +1. td_create_operator(parent="/project1", type="tableDat", name="instance_data") +2. td_create_operator(parent="/project1", type="geometryComp", name="geo1") +3. td_create_operator(parent="/project1/geo1", type="sphereSop", name="sphere") +4. td_create_operator(parent="/project1", type="constMat", name="mat1") +5. td_create_operator(parent="/project1", type="cameraComp", name="cam1") +6. td_create_operator(parent="/project1", type="lightComp", name="light1") +7. td_create_operator(parent="/project1", type="renderTop", name="render1") + +8. td_execute_python: """ +import random, math +dat = op('/project1/instance_data') +dat.clear() +dat.appendRow(['tx', 'ty', 'tz', 'sx', 'sy', 'sz', 'cr', 'cg', 'cb']) +for i in range(500): + angle = i * 0.1 + r = 2 + i * 0.01 + dat.appendRow([ + str(math.cos(angle) * r), + str(math.sin(angle) * r), + str((i - 250) * 0.02), + '0.05', '0.05', '0.05', + str(random.random()), + str(random.random()), + str(random.random()) + ]) +""" + +9. td_set_operator_pars(path="/project1/geo1", + properties={"instancing": true, "instancechop": "", + "instancedat": "/project1/instance_data", + "material": "/project1/mat1"}) +10. td_set_operator_pars(path="/project1/render1", + properties={"camera": "/project1/cam1", "geometry": "/project1/geo1", + "light": "/project1/light1", + "resolutionw": 1280, "resolutionh": 720}) +11. td_set_operator_pars(path="/project1/cam1", + properties={"tz": 10}) +``` + +### Pattern 6: Reaction-Diffusion (GLSL) + +Classic Gray-Scott reaction-diffusion system running on the GPU. + +``` +Text DAT (GLSL code) -> GLSL TOP (resolution, dat reference) -> Feedback TOP + ^ | + |_______________________________________| + Level TOP (out) +``` + +**Key GLSL code (write to Text DAT via td_execute_python):** + +```glsl +// Gray-Scott reaction-diffusion +uniform float feed; // 0.037 +uniform float kill; // 0.06 +uniform float dA; // 1.0 +uniform float dB; // 0.5 + +layout(location = 0) out vec4 fragColor; + +void main() { + vec2 uv = vUV.st; + vec2 texel = 1.0 / uTDOutputInfo.res.zw; + + vec4 c = texture(sTD2DInputs[0], uv); + float a = c.r; + float b = c.g; + + // Laplacian (9-point stencil) + float lA = 0.0, lB = 0.0; + for(int dx = -1; dx <= 1; dx++) { + for(int dy = -1; dy <= 1; dy++) { + float w = (dx == 0 && dy == 0) ? -1.0 : (abs(dx) + abs(dy) == 1 ? 0.2 : 0.05); + vec4 s = texture(sTD2DInputs[0], uv + vec2(dx, dy) * texel); + lA += s.r * w; + lB += s.g * w; + } + } + + float reaction = a * b * b; + float newA = a + (dA * lA - reaction + feed * (1.0 - a)); + float newB = b + (dB * lB + reaction - (kill + feed) * b); + + fragColor = vec4(clamp(newA, 0.0, 1.0), clamp(newB, 0.0, 1.0), 0.0, 1.0); +} +``` + +## Video Processing + +### Pattern 7: Video Effects Chain + +Apply a chain of effects to a video file. + +``` +Movie File In TOP -> HSV Adjust TOP -> Level TOP -> Blur TOP -> Composite TOP -> Null TOP (out) + ^ + Text TOP ---+ +``` + +**MCP Build Sequence:** + +``` +1. td_create_operator(parent="/project1", type="moviefileinTop", name="video_in") +2. td_create_operator(parent="/project1", type="hsvadjustTop", name="color") +3. td_create_operator(parent="/project1", type="levelTop", name="levels") +4. td_create_operator(parent="/project1", type="blurTop", name="blur") +5. td_create_operator(parent="/project1", type="compositeTop", name="overlay") +6. td_create_operator(parent="/project1", type="textTop", name="title") +7. td_create_operator(parent="/project1", type="nullTop", name="out") + +8. td_set_operator_pars(path="/project1/video_in", + properties={"file": "/path/to/video.mp4", "play": true}) +9. td_set_operator_pars(path="/project1/color", + properties={"hueoffset": 0.1, "saturationmult": 1.3}) +10. td_set_operator_pars(path="/project1/levels", + properties={"brightness1": 1.1, "contrast": 1.2, "gamma1": 0.9}) +11. td_set_operator_pars(path="/project1/blur", + properties={"sizex": 2, "sizey": 2}) +12. td_set_operator_pars(path="/project1/title", + properties={"text": "My Video", "fontsizex": 48, "alignx": 1, "aligny": 1}) + +13. td_execute_python: """ +chain = ['video_in', 'color', 'levels', 'blur'] +for i in range(len(chain) - 1): + op(f'/project1/{chain[i]}').outputConnectors[0].connect(op(f'/project1/{chain[i+1]}')) +op('/project1/blur').outputConnectors[0].connect(op('/project1/overlay').inputConnectors[0]) +op('/project1/title').outputConnectors[0].connect(op('/project1/overlay').inputConnectors[1]) +op('/project1/overlay').outputConnectors[0].connect(op('/project1/out')) +""" +``` + +### Pattern 8: Video Recording + +Record the output to a file. **H.264/H.265 require a Commercial license** — use Motion JPEG (`mjpa`) on Non-Commercial. + +``` +[any TOP chain] -> Null TOP -> Movie File Out TOP +``` + +```python +# Build via td_execute_python: +root = op('/project1') + +# Always put a Null TOP before the recorder +null_out = root.op('out') # or create one +rec = root.create(moviefileoutTOP, 'recorder') +null_out.outputConnectors[0].connect(rec.inputConnectors[0]) + +rec.par.type = 'movie' +rec.par.file = '/tmp/output.mov' +rec.par.videocodec = 'mjpa' # Motion JPEG — works on Non-Commercial + +# Start recording (par.record is a toggle — .record() method may not exist) +rec.par.record = True +# ... let TD run for desired duration ... +rec.par.record = False + +# For image sequences: +# rec.par.type = 'imagesequence' +# rec.par.imagefiletype = 'png' +# rec.par.file.expr = "'/tmp/frames/out' + me.fileSuffix" # fileSuffix REQUIRED +``` + +**Pitfalls:** +- Setting `par.file` + `par.record = True` in the same script may race — use `run("...", delayFrames=2)` +- `TOP.save()` called rapidly always captures the same frame — use MovieFileOut for animation +- See `pitfalls.md` #25-27 for full details + +### Pattern 8b: TD → External Pipeline (FFmpeg / Python / Post-Processing) + +Export TD visuals for use in another tool (ffmpeg, Python, ASCII art, etc.). This is the standard workflow when you need to composite TD output with external processing (ASCII conversion, Python shader chains, ML inference, etc.). + +**Step 1: Record to video in TD** + +```python +# Preferred: ProRes on macOS (lossless, Non-Commercial OK, ~55MB/s at 1280x720) +rec.par.videocodec = 'prores' +# Fallback for non-macOS: mjpa (Motion JPEG) +# rec.par.videocodec = 'mjpa' +rec.par.record = True +# ... wait N seconds ... +rec.par.record = False +``` + +**Step 2: Extract frames with ffmpeg** + +```bash +# Extract all frames at 30fps +ffmpeg -y -i /tmp/output.mov -vf 'fps=30' /tmp/frames/frame_%06d.png + +# Or extract a specific duration +ffmpeg -y -i /tmp/output.mov -t 25 -vf 'fps=30' /tmp/frames/frame_%06d.png + +# Or extract specific frame range +ffmpeg -y -i /tmp/output.mov -vf 'select=between(n\,0\,749)' -vsync vfr /tmp/frames/frame_%06d.png +``` + +**Step 3: Process frames in Python** + +```python +from PIL import Image +import os + +frames_dir = '/tmp/frames' +output_dir = '/tmp/processed' +os.makedirs(output_dir, exist_ok=True) + +for fname in sorted(os.listdir(frames_dir)): + if not fname.endswith('.png'): + continue + img = Image.open(os.path.join(frames_dir, fname)) + # ... apply your processing ... + img.save(os.path.join(output_dir, fname)) +``` + +**Step 4: Mux processed frames back with audio** + +```bash +# Create video from processed frames + audio with fade-out +ffmpeg -y \ + -framerate 30 -i /tmp/processed/frame_%06d.png \ + -i /tmp/audio.mp3 \ + -c:v libx264 -pix_fmt yuv420p -crf 18 \ + -c:a aac -b:a 192k \ + -shortest \ + -af 'afade=t=out:st=23:d=2' \ + /tmp/final_output.mp4 +``` + +**Key considerations:** +- Use ProRes for the TD recording step to avoid generation loss during compositing +- Extract at the target output framerate (not TD's render framerate) +- For audio-synced content, analyze the audio file separately in Python (scipy FFT) to get per-frame features (rms, spectral bands, beats) and drive compositing parameters +- Always verify TD FPS > 0 before recording (see pitfalls #37, #38) + +## Data Visualization + +### Pattern 9: Table Data -> Bar Chart via Instancing + +Visualize tabular data as a 3D bar chart. + +``` +Table DAT (data) -> Script DAT (transform to instance format) -> DAT to CHOP + | +Box SOP -> Geometry COMP (instancing from CHOP) -> Render TOP -> Null TOP (out) + + PBR MAT + + Camera COMP + + Light COMP +``` + +```python +# Script DAT code to transform data to instance positions +td_execute_python: """ +source = op('/project1/data_table') +instance = op('/project1/instance_transform') +instance.clear() +instance.appendRow(['tx', 'ty', 'tz', 'sx', 'sy', 'sz', 'cr', 'cg', 'cb']) + +for i in range(1, source.numRows): + value = float(source[i, 'value']) + name = source[i, 'name'] + instance.appendRow([ + str(i * 1.5), # x position (spread bars) + str(value / 2), # y position (center bar vertically) + '0', # z position + '1', str(value), '1', # scale (height = data value) + '0.2', '0.6', '1.0' # color (blue) + ]) +""" +``` + +### Pattern 9b: Audio-Reactive GLSL Fractal (Proven Recipe) + +Audio spectrum drives a GLSL fractal shader directly via a spectrum texture input. Bass thickens inner fractal lines, mids twist rotation, highs light outer edges. **Always run discovery (SKILL.md Step 0) before using any param names from these recipes — they may differ in your TD version.** + +``` +Audio File In CHOP → Audio Spectrum CHOP (FFT=512, outlength=256) + → Math CHOP (gain=10) + → CHOP To TOP (spectrum texture, 256x2, dataformat=r) + ↓ (input 1) +Constant TOP (rgba32float, time) → GLSL TOP (audio-reactive shader) → Null TOP + (input 0) ↑ + Text DAT (shader code) +``` + +**Build via td_execute_python (complete working script):** + +```python +# td_execute_python script: +td_execute_python(code=""" +import os +root = op('/project1') + +# Audio input +audio = root.create(audiofileinCHOP, 'audio_in') +audio.par.file = '/path/to/music.mp3' +audio.par.playmode = 0 # Locked to timeline + +# FFT analysis (output length manually set to 256 bins) +spectrum = root.create(audiospectrumCHOP, 'spectrum') +audio.outputConnectors[0].connect(spectrum.inputConnectors[0]) +spectrum.par.fftsize = '512' +spectrum.par.outputmenu = 'setmanually' +spectrum.par.outlength = 256 + +# THEN boost gain on the raw spectrum (NO Lag CHOP — see pitfall #34) +math = root.create(mathCHOP, 'math_norm') +spectrum.outputConnectors[0].connect(math.inputConnectors[0]) +math.par.gain = 10 + +# Spectrum → texture (256x2 image — stereo, sample at y=0.25 for first channel) +# NOTE: choptoTOP has NO input connectors — use par.chop reference! +spec_tex = root.create(choptoTOP, 'spectrum_tex') +spec_tex.par.chop = math +spec_tex.par.dataformat = 'r' +spec_tex.par.layout = 'rowscropped' + +# Time driver (rgba32float to avoid 0-1 clamping!) +time_drv = root.create(constantTOP, 'time_driver') +time_drv.par.format = 'rgba32float' +time_drv.par.outputresolution = 'custom' +time_drv.par.resolutionw = 1 +time_drv.par.resolutionh = 1 +time_drv.par.colorr.expr = "absTime.seconds % 1000.0" +time_drv.par.colorg.expr = "int(absTime.seconds / 1000.0)" + +# GLSL shader +glsl = root.create(glslTOP, 'audio_shader') +glsl.par.outputresolution = 'custom' +glsl.par.resolutionw = 1280; glsl.par.resolutionh = 720 + +shader_dat = root.create(textDAT, 'shader_code') +shader_dat.text = open('/tmp/shader.glsl').read() +glsl.par.pixeldat = shader_dat + +# Wire: input 0=time, input 1=spectrum +time_drv.outputConnectors[0].connect(glsl.inputConnectors[0]) +spec_tex.outputConnectors[0].connect(glsl.inputConnectors[1]) + +# Output + audio playback +out = root.create(nullTOP, 'output') +glsl.outputConnectors[0].connect(out.inputConnectors[0]) +audio_out = root.create(audiodeviceoutCHOP, 'audio_out') +audio.outputConnectors[0].connect(audio_out.inputConnectors[0]) + +result = 'network built' +""") +``` + +**GLSL shader (reads spectrum from input 1 texture):** + +```glsl +out vec4 fragColor; + +vec3 palette(float t) { + vec3 a = vec3(0.5); vec3 b = vec3(0.5); + vec3 c = vec3(1.0); vec3 d = vec3(0.263, 0.416, 0.557); + return a + b * cos(6.28318 * (c * t + d)); +} + +void main() { + vec4 td = texture(sTD2DInputs[0], vec2(0.5)); + float t = td.r + td.g * 1000.0; + + vec2 res = uTDOutputInfo.res.zw; + vec2 uv = (gl_FragCoord.xy * 2.0 - res) / min(res.x, res.y); + vec2 uv0 = uv; + vec3 finalColor = vec3(0.0); + + float bass = texture(sTD2DInputs[1], vec2(0.05, 0.25)).r; + float mids = texture(sTD2DInputs[1], vec2(0.25, 0.25)).r; + float highs = texture(sTD2DInputs[1], vec2(0.65, 0.25)).r; + + float ca = cos(t * (0.15 + mids * 0.3)); + float sa = sin(t * (0.15 + mids * 0.3)); + uv = mat2(ca, -sa, sa, ca) * uv; + + for (float i = 0.0; i < 4.0; i++) { + uv = fract(uv * (1.4 + bass * 0.3)) - 0.5; + float d = length(uv) * exp(-length(uv0)); + float freq = texture(sTD2DInputs[1], vec2(clamp(d*0.5, 0.0, 1.0), 0.25)).r; + vec3 col = palette(length(uv0) + i * 0.4 + t * 0.35); + d = sin(d * (7.0 + bass * 4.0) + t * 1.5) / 8.0; + d = abs(d); + d = pow(0.012 / d, 1.2 + freq * 0.8 + bass * 0.5); + finalColor += col * d; + } + + float glow = (0.03 + bass * 0.05) / (length(uv0) + 0.03); + finalColor += vec3(0.4, 0.1, 0.7) * glow * (0.6 + 0.4 * sin(t * 2.5)); + + float ring = abs(length(uv0) - 0.4 - mids * 0.3); + finalColor += vec3(0.1, 0.6, 0.8) * (0.005 / ring) * (0.2 + highs * 0.5); + + finalColor *= smoothstep(0.0, 1.0, 1.0 - dot(uv0*0.55, uv0*0.55)); + finalColor = finalColor / (finalColor + vec3(1.0)); + + fragColor = TDOutputSwizzle(vec4(finalColor, 1.0)); +} +``` + +**How spectrum sampling drives the visual:** +- `texture(sTD2DInputs[1], vec2(x, 0.0)).r` — x position = frequency (0=bass, 1=treble) +- Inner fractal iterations sample lower x → react to bass +- Outer iterations sample higher x → react to treble +- `bass * 0.3` on `fract()` scale → fractal zoom pulses with bass +- `bass * 4.0` on sin frequency → line density pulses with bass +- `mids * 0.3` on rotation speed → spiral twists faster during vocal/mid sections +- `highs * 0.5` on ring opacity → high-frequency sparkle on outer ring + +**Recording the output:** Use MovieFileOut TOP with `mjpa` codec (H.264 requires Commercial license). See pitfalls #25-27. + +## GLSL Shaders + +### Pattern 10: Custom Fragment Shader + +Write a custom visual effect as a GLSL fragment shader. + +``` +Text DAT (shader code) -> GLSL TOP -> Level TOP -> Null TOP (out) + + optional input TOPs for texture sampling +``` + +**Common GLSL uniforms available in TouchDesigner:** + +```glsl +// Automatically provided by TD +uniform vec4 uTDOutputInfo; // .res.zw = resolution + +// NOTE: uTDCurrentTime does NOT exist in TD 099! +// Feed time via a 1x1 Constant TOP (format=rgba32float): +// t.par.colorr.expr = "absTime.seconds % 1000.0" +// t.par.colorg.expr = "int(absTime.seconds / 1000.0)" +// Then read in GLSL: +// vec4 td = texture(sTD2DInputs[0], vec2(0.5)); +// float t = td.r + td.g * 1000.0; + +// Input textures (from connected TOP inputs) +uniform sampler2D sTD2DInputs[1]; // array of input samplers + +// From vertex shader +in vec3 vUV; // UV coordinates (0-1 range) +``` + +**Example: Plasma shader (using time from input texture)** + +```glsl +layout(location = 0) out vec4 fragColor; + +void main() { + vec2 uv = vUV.st; + // Read time from Constant TOP input 0 (rgba32float format) + vec4 td = texture(sTD2DInputs[0], vec2(0.5)); + float t = td.r + td.g * 1000.0; + + float v1 = sin(uv.x * 10.0 + t); + float v2 = sin(uv.y * 10.0 + t * 0.7); + float v3 = sin((uv.x + uv.y) * 10.0 + t * 1.3); + float v4 = sin(length(uv - 0.5) * 20.0 - t * 2.0); + + float v = (v1 + v2 + v3 + v4) * 0.25; + + vec3 color = vec3( + sin(v * 3.14159 + 0.0) * 0.5 + 0.5, + sin(v * 3.14159 + 2.094) * 0.5 + 0.5, + sin(v * 3.14159 + 4.189) * 0.5 + 0.5 + ); + + fragColor = vec4(color, 1.0); +} +``` + +### Pattern 11: Multi-Pass GLSL (Ping-Pong) + +For effects needing state across frames (particles, fluid, cellular automata), use GLSL Multi TOP with multiple passes or a Feedback TOP loop. + +``` +GLSL Multi TOP (pass 0: simulation, pass 1: rendering) + + Text DAT (simulation shader) + + Text DAT (render shader) + -> Level TOP -> Null TOP (out) + ^ + |__ Feedback TOP (feeds simulation state back) +``` + +## Interactive Installations + +### Pattern 12: Mouse/Touch -> Visual Response + +``` +Mouse In CHOP -> Math CHOP (normalize to 0-1) -> [export to visual params] + +# Or for touch/multi-touch: +Multi Touch In DAT -> Script CHOP (parse touches) -> [export to visual params] +``` + +```python +# Normalize mouse position to 0-1 range +td_execute_python: """ +op('/project1/noise1').par.offsetx.expr = "op('/project1/mouse_norm')['tx']" +op('/project1/noise1').par.offsety.expr = "op('/project1/mouse_norm')['ty']" +""" +``` + +### Pattern 13: OSC Control (from external software) + +``` +OSC In CHOP (port 7000) -> Select CHOP (pick channels) -> [export to visual params] +``` + +``` +1. td_create_operator(parent="/project1", type="oscinChop", name="osc_in") +2. td_set_operator_pars(path="/project1/osc_in", properties={"port": 7000}) + +# OSC messages like /frequency 440 will appear as channel "frequency" with value 440 +# Export to any parameter: +3. td_execute_python: "op('/project1/noise1').par.period.expr = \"op('/project1/osc_in')['frequency']\"" +``` + +### Pattern 14: MIDI Control (DJ/VJ) + +``` +MIDI In CHOP (device) -> Select CHOP -> [export channels to visual params] +``` + +Common MIDI mappings: +- CC channels (knobs/faders): continuous 0-127, map to float params +- Note On/Off: binary triggers, map to Trigger CHOP for envelopes +- Velocity: intensity/brightness + +## Live Performance + +### Pattern 15: Multi-Source VJ Setup + +``` +Source A (generative) ----+ +Source B (video) ---------+-- Switch/Cross TOP -- Level TOP -- Window COMP (output) +Source C (camera) --------+ + ^ + MIDI/OSC control selects active source and crossfade +``` + +```python +# MIDI CC1 controls which source is active (0-127 -> 0-2) +td_execute_python: """ +op('/project1/switch1').par.index.expr = "int(op('/project1/midi_in')['cc1'] / 42)" +""" + +# MIDI CC2 controls crossfade between current and next +td_execute_python: """ +op('/project1/cross1').par.cross.expr = "op('/project1/midi_in')['cc2'] / 127.0" +""" +``` + +### Pattern 16: Projection Mapping + +``` +Content TOPs ----+ + | +Stoner TOP (UV mapping) -> Composite TOP -> Window COMP (projector output) + or +Kantan Mapper COMP (external .tox) +``` + +For projection mapping, the key is: +1. Create your visual content as standard TOPs +2. Use Stoner TOP or a third-party mapping tool to UV-map content to physical surfaces +3. Output via Window COMP to the projector + +### Pattern 17: Cue System + +``` +Table DAT (cue list: cue_number, scene_name, duration, transition_type) + | +Script CHOP (cue state: current_cue, progress, next_cue_trigger) + | +[export to Switch/Cross TOPs to transition between scenes] +``` + +```python +td_execute_python: """ +# Simple cue system +cue_table = op('/project1/cue_list') +cue_state = op('/project1/cue_state') + +def advance_cue(): + current = int(cue_state.par.value0.val) + next_cue = min(current + 1, cue_table.numRows - 1) + cue_state.par.value0.val = next_cue + + scene = cue_table[next_cue, 'scene'] + duration = float(cue_table[next_cue, 'duration']) + + # Set crossfade target and duration + op('/project1/cross1').par.cross.val = 0 + # Animate cross to 1.0 over duration seconds + # (use a Timer CHOP or LFO CHOP for smooth animation) +""" +``` + +## Networking + +### Pattern 18: OSC Server/Client + +``` +# Sending OSC +OSC Out CHOP -> (network) -> external application + +# Receiving OSC +(network) -> OSC In CHOP -> Select CHOP -> [use values] +``` + +### Pattern 19: NDI Video Streaming + +``` +# Send video over network +[any TOP chain] -> NDI Out TOP (source name) + +# Receive video from network +NDI In TOP (select source) -> [process as normal TOP] +``` + +### Pattern 20: WebSocket Communication + +``` +WebSocket DAT -> Script DAT (parse JSON messages) -> [update visuals] +``` + +```python +td_execute_python: """ +ws = op('/project1/websocket1') +ws.par.address = 'ws://localhost:8080' +ws.par.active = True + +# In a DAT Execute callback (Script DAT watching WebSocket DAT): +# def onTableChange(dat): +# import json +# msg = json.loads(dat.text) +# op('/project1/noise1').par.seed.val = msg.get('seed', 0) +""" +``` diff --git a/optional-skills/creative/touchdesigner-mcp/references/operators.md b/optional-skills/creative/touchdesigner-mcp/references/operators.md new file mode 100644 index 000000000000..6aa716cb9a21 --- /dev/null +++ b/optional-skills/creative/touchdesigner-mcp/references/operators.md @@ -0,0 +1,239 @@ +# TouchDesigner Operator Reference + +## Operator Families Overview + +TouchDesigner has 6 operator families. Each family processes a specific data type and is color-coded in the UI. Operators can only connect to others of the SAME family (with cross-family converters as the bridge). + +## TOPs — Texture Operators (Purple) + +2D image/texture processing on the GPU. The workhorse of visual output. + +### Generators (create images from nothing) + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Noise TOP | `noiseTop` | `type` (0-6), `monochrome`, `seed`, `period`, `harmonics`, `exponent`, `amp`, `offset`, `resolutionw/h` | Procedural noise textures — Perlin, Simplex, Sparse, etc. Foundation of generative art. | +| Constant TOP | `constantTop` | `colorr/g/b/a`, `resolutionw/h` | Solid color. Use as background or blend input. | +| Text TOP | `textTop` | `text`, `fontsizex`, `fontfile`, `alignx/y`, `colorr/g/b` | Render text to texture. Supports multi-line, word wrap. | +| Ramp TOP | `rampTop` | `type` (0=horizontal, 1=vertical, 2=radial, 3=circular), `phase`, `period` | Gradient textures for masking, color mapping. | +| Circle TOP | `circleTop` | `radiusx/y`, `centerx/y`, `width` | Circles, rings, ellipses. | +| Rectangle TOP | `rectangleTop` | `sizex/y`, `centerx/y`, `softness` | Rectangles with optional softness. | +| GLSL TOP | `glslTop` | `dat` (points to shader DAT), `resolutionw/h`, `outputformat`, custom uniforms | Custom fragment shaders. Most powerful TOP for custom visuals. | +| GLSL Multi TOP | `glslmultiTop` | `dat`, `numinputs`, `numoutputs`, `numcomputepasses` | Multi-pass GLSL with compute shaders. Advanced. | +| Render TOP | `renderTop` | `camera`, `geometry`, `lights`, `resolutionw/h` | Renders 3D scenes (SOPs + MATs + Camera/Light COMPs). | + +### Filters (modify a single input) + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Level TOP | `levelTop` | `opacity`, `brightness1/2`, `gamma1/2`, `contrast`, `invert`, `blacklevel/whitelevel` | Brightness, contrast, gamma, levels. Essential color correction. | +| Blur TOP | `blurTop` | `sizex/y`, `type` (0=Gaussian, 1=Box, 2=Bartlett) | Gaussian/box blur. | +| Transform TOP | `transformTop` | `tx/ty`, `sx/sy`, `rz`, `pivotx/y`, `extend` (0=Hold, 1=Zero, 2=Repeat, 3=Mirror) | Translate, scale, rotate textures. | +| HSV Adjust TOP | `hsvadjustTop` | `hueoffset`, `saturationmult`, `valuemult` | HSV color adjustments. | +| Lookup TOP | `lookupTop` | (input: texture + lookup table) | Color remapping via lookup table texture. | +| Edge TOP | `edgeTop` | `type` (0=Sobel, 1=Frei-Chen) | Edge detection. | +| Displace TOP | `displaceTop` | `scalex/y` | Pixel displacement using a second input as displacement map. | +| Flip TOP | `flipTop` | `flipx`, `flipy`, `flop` (diagonal) | Mirror/flip textures. | +| Crop TOP | `cropTop` | `cropleft/right/top/bottom` | Crop region of texture. | +| Resolution TOP | `resolutionTop` | `resolutionw/h`, `outputresolution` | Resize textures. | +| Null TOP | `nullTop` | (none significant) | Pass-through. Use for organization, referencing, feedback delay. | +| Cache TOP | `cacheTop` | `length`, `step` | Store N frames of history. Useful for trails, time effects. | + +### Compositors (combine multiple inputs) + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Composite TOP | `compositeTop` | `operand` (0-31: Over, Add, Multiply, Screen, etc.) | Blend two textures with standard compositing modes. | +| Over TOP | `overTop` | (simple alpha compositing) | Layer with alpha. Simpler than Composite. | +| Add TOP | `addTop` | (additive blend) | Additive blending. Great for glow, light effects. | +| Multiply TOP | `multiplyTop` | (multiplicative blend) | Multiply blend. Good for masking, darkening. | +| Switch TOP | `switchTop` | `index` (0-based) | Switch between multiple inputs by index. | +| Cross TOP | `crossTop` | `cross` (0.0-1.0) | Crossfade between two inputs. | + +### I/O (input/output) + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Movie File In TOP | `moviefileinTop` | `file`, `speed`, `trim`, `index` | Load video files, image sequences. | +| Movie File Out TOP | `moviefileoutTop` | `file`, `type` (codec), `record` (toggle) | Record/export video files. | +| NDI In TOP | `ndiinTop` | `sourcename` | Receive NDI video streams. | +| NDI Out TOP | `ndioutTop` | `sourcename` | Send NDI video streams. | +| Syphon Spout In/Out TOP | `syphonspoutinTop` / `syphonspoutoutTop` | `servername` | Inter-app texture sharing. | +| Video Device In TOP | `videodeviceinTop` | `device` | Webcam/capture card input. | +| Feedback TOP | `feedbackTop` | `top` (path to the TOP to feed back) | One-frame delay feedback. Essential for recursive effects. | + +### Converters + +| Operator | Type Name | Direction | Use | +|----------|-----------|-----------|-----| +| CHOP to TOP | `choptopTop` | CHOP -> TOP | Visualize channel data as texture (waveform, spectrum display). | +| TOP to CHOP | `topchopChop` | TOP -> CHOP | Sample texture pixels as channel data. | + +## CHOPs — Channel Operators (Green) + +Time-varying numeric data: audio, animation curves, sensor data, control signals. + +### Generators + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Constant CHOP | `constantChop` | `name0/value0`, `name1/value1`... | Static named channels. Control panel for parameters. | +| LFO CHOP | `lfoChop` | `frequency`, `type` (0=Sin, 1=Tri, 2=Square, 3=Ramp, 4=Pulse), `amp`, `offset`, `phase` | Low frequency oscillator. Animation driver. | +| Noise CHOP | `noiseChop` | `type`, `roughness`, `period`, `amp`, `seed`, `channels` | Smooth random motion. Organic animation. | +| Pattern CHOP | `patternChop` | `type` (0=Sine, 1=Triangle, ...), `length`, `cycles` | Generate waveform patterns. | +| Timer CHOP | `timerChop` | `length`, `play`, `cue`, `cycles` | Countdown/count-up timer with cue points. | +| Count CHOP | `countChop` | `threshold`, `limittype`, `limitmin/max` | Event counter with wrapping/clamping. | + +### Audio + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Audio File In CHOP | `audiofileinChop` | `file`, `volume`, `play`, `speed`, `trim` | Play audio files. | +| Audio Device In CHOP | `audiodeviceinChop` | `device`, `channels` | Live microphone/line input. | +| Audio Spectrum CHOP | `audiospectrumChop` | `size` (FFT size), `outputformat` (0=Power, 1=Magnitude) | FFT frequency analysis. | +| Audio Band EQ CHOP | `audiobandeqChop` | `bands`, `gaindb` per band | Frequency band isolation. | +| Audio Device Out CHOP | `audiodeviceoutChop` | `device` | Audio playback output. | + +### Math/Logic + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Math CHOP | `mathChop` | `preoff`, `gain`, `postoff`, `chanop` (0=Off, 1=Add, 2=Subtract, 3=Multiply...) | Math operations on channels. The Swiss army knife. | +| Logic CHOP | `logicChop` | `preop` (0=Off, 1=AND, 2=OR, 3=XOR, 4=NAND), `convert` | Boolean logic on channels. | +| Filter CHOP | `filterChop` | `type` (0=Low Pass, 1=Band Pass, 2=High Pass, 3=Notch), `cutofffreq`, `filterwidth` | Smooth, dampen, filter signals. | +| Lag CHOP | `lagChop` | `lag1/2`, `overshoot1/2` | Smooth transitions with overshoot. | +| Limit CHOP | `limitChop` | `type` (0=Clamp, 1=Loop, 2=ZigZag), `min/max` | Clamp or wrap channel values. | +| Speed CHOP | `speedChop` | (none significant) | Integrate values (velocity to position, acceleration to velocity). | +| Trigger CHOP | `triggerChop` | `attack`, `peak`, `decay`, `sustain`, `release` | ADSR envelope from trigger events. | +| Select CHOP | `selectChop` | `chop` (path), `channames` | Reference channels from another CHOP. | +| Merge CHOP | `mergeChop` | `align` (0=Extend, 1=Trim to First, 2=Trim to Shortest) | Combine channels from multiple CHOPs. | +| Null CHOP | `nullChop` | (none significant) | Pass-through for organization and referencing. | + +### Input Devices + +| Operator | Type Name | Use | +|----------|-----------|-----| +| Mouse In CHOP | `mouseinChop` | Mouse position, buttons, wheel. | +| Keyboard In CHOP | `keyboardinChop` | Keyboard key states. | +| MIDI In CHOP | `midiinChop` | MIDI note/CC input. | +| OSC In CHOP | `oscinChop` | OSC message input (network). | + +## SOPs — Surface Operators (Blue) + +3D geometry: points, polygons, NURBS, meshes. + +### Generators + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Grid SOP | `gridSop` | `rows`, `cols`, `sizex/y`, `type` (0=Polygon, 1=Mesh, 2=NURBS) | Flat grid mesh. Foundation for displacement, instancing. | +| Sphere SOP | `sphereSop` | `type`, `rows`, `cols`, `radius` | Sphere geometry. | +| Box SOP | `boxSop` | `sizex/y/z` | Box geometry. | +| Torus SOP | `torusSop` | `radiusx/y`, `rows`, `cols` | Donut shape. | +| Circle SOP | `circleSop` | `type`, `radius`, `divs` | Circle/ring geometry. | +| Line SOP | `lineSop` | `dist`, `points` | Line segments. | +| Text SOP | `textSop` | `text`, `fontsizex`, `fontfile`, `extrude` | 3D text geometry. | + +### Modifiers + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Transform SOP | `transformSop` | `tx/ty/tz`, `rx/ry/rz`, `sx/sy/sz` | Transform geometry (translate, rotate, scale). | +| Noise SOP | `noiseSop` | `type`, `amp`, `period`, `roughness` | Deform geometry with noise. | +| Sort SOP | `sortSop` | `ptsort`, `primsort` | Reorder points/primitives. | +| Facet SOP | `facetSop` | `unique`, `consolidate`, `computenormals` | Normals, consolidation, unique points. | +| Merge SOP | `mergeSop` | (none significant) | Combine multiple geometry inputs. | +| Null SOP | `nullSop` | (none significant) | Pass-through. | + +## DATs — Data Operators (White) + +Text, tables, scripts, network data. + +### Core + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Table DAT | `tableDat` | (edit content directly) | Spreadsheet-like data tables. | +| Text DAT | `textDat` | (edit content directly) | Arbitrary text content. Shader code, configs, scripts. | +| Script DAT | `scriptDat` | `language` (0=Python, 1=C++) | Custom callbacks and DAT processing. | +| CHOP Execute DAT | `chopexecDat` | `chop` (path to watch), callbacks | Trigger Python on CHOP value changes. | +| DAT Execute DAT | `datexecDat` | `dat` (path to watch) | Trigger Python on DAT content changes. | +| Panel Execute DAT | `panelexecDat` | `panel` | Trigger Python on UI panel events. | + +### I/O + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Web DAT | `webDat` | `url`, `fetchmethod` (0=GET, 1=POST) | HTTP requests. API integration. | +| TCP/IP DAT | `tcpipDat` | `address`, `port`, `mode` | TCP networking. | +| OSC In DAT | `oscinDat` | `port` | Receive OSC as text messages. | +| Serial DAT | `serialDat` | `port`, `baudrate` | Serial port communication (Arduino, etc.). | +| File In DAT | `fileinDat` | `file` | Read text files. | +| File Out DAT | `fileoutDat` | `file`, `write` | Write text files. | + +### Conversions + +| Operator | Type Name | Direction | Use | +|----------|-----------|-----------|-----| +| DAT to CHOP | `dattochopChop` | DAT -> CHOP | Convert table data to channels. | +| CHOP to DAT | `choptodatDat` | CHOP -> DAT | Convert channel data to table rows. | +| SOP to DAT | `soptodatDat` | SOP -> DAT | Extract geometry data as table. | + +## MATs — Material Operators (Yellow) + +Materials for 3D rendering in Render TOP / Geometry COMP. + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Phong MAT | `phongMat` | `diff_colorr/g/b`, `spec_colorr/g/b`, `shininess`, `colormap`, `normalmap` | Classic Phong shading. Simple, fast. | +| PBR MAT | `pbrMat` | `basecolorr/g/b`, `metallic`, `roughness`, `normalmap`, `emitcolorr/g/b` | Physically-based rendering. Realistic materials. | +| GLSL MAT | `glslMat` | `dat` (shader DAT), custom uniforms | Custom vertex + fragment shaders for 3D. | +| Constant MAT | `constMat` | `colorr/g/b`, `colormap` | Flat unlit color/texture. No shading. | +| Point Sprite MAT | `pointspriteMat` | `colormap`, `scale` | Render points as camera-facing sprites. Great for particles. | +| Wireframe MAT | `wireframeMat` | `colorr/g/b`, `width` | Wireframe rendering. | +| Depth MAT | `depthMat` | `near`, `far` | Render depth buffer as grayscale. | + +## COMPs — Component Operators (Gray) + +Containers, 3D scene elements, UI components. + +### 3D Scene + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Geometry COMP | `geometryComp` | `material` (path), `instancechop` (path), `instancing` (toggle) | Renders geometry with material. Instancing host. | +| Camera COMP | `cameraComp` | `tx/ty/tz`, `rx/ry/rz`, `fov`, `near/far` | Camera for Render TOP. | +| Light COMP | `lightComp` | `lighttype` (0=Point, 1=Directional, 2=Spot, 3=Cone), `dimmer`, `colorr/g/b` | Lighting for 3D scenes. | +| Ambient Light COMP | `ambientlightComp` | `dimmer`, `colorr/g/b` | Ambient lighting. | +| Environment Light COMP | `envlightComp` | `envmap` | Image-based lighting (IBL). | + +### Containers + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Container COMP | `containerComp` | `w`, `h`, `bgcolor1/2/3` | UI container. Holds other COMPs for panel layouts. | +| Base COMP | `baseComp` | (none significant) | Generic container. Networks-inside-networks. | +| Replicator COMP | `replicatorComp` | `template`, `operatorsdat` | Clone a template operator N times from a table. | + +### Utilities + +| Operator | Type Name | Key Parameters | Use | +|----------|-----------|---------------|-----| +| Window COMP | `windowComp` | `winw/h`, `winoffsetx/y`, `monitor`, `borders` | Output window for display/projection. | +| Select COMP | `selectComp` | `rowcol`, `panel` | Select and display content from elsewhere. | +| Engine COMP | `engineComp` | `tox`, `externaltox` | Load external .tox components. Sub-process isolation. | + +## Cross-Family Converter Summary + +| From | To | Operator | Type Name | +|------|-----|----------|-----------| +| CHOP | TOP | CHOP to TOP | `choptopTop` | +| TOP | CHOP | TOP to CHOP | `topchopChop` | +| DAT | CHOP | DAT to CHOP | `dattochopChop` | +| CHOP | DAT | CHOP to DAT | `choptodatDat` | +| SOP | CHOP | SOP to CHOP | `soptochopChop` | +| CHOP | SOP | CHOP to SOP | `choptosopSop` | +| SOP | DAT | SOP to DAT | `soptodatDat` | +| DAT | SOP | DAT to SOP | `dattosopSop` | +| SOP | TOP | (use Render TOP + Geometry COMP) | — | +| TOP | SOP | TOP to SOP | `toptosopSop` | diff --git a/optional-skills/creative/touchdesigner-mcp/references/pitfalls.md b/optional-skills/creative/touchdesigner-mcp/references/pitfalls.md new file mode 100644 index 000000000000..33c9b5f4d87c --- /dev/null +++ b/optional-skills/creative/touchdesigner-mcp/references/pitfalls.md @@ -0,0 +1,508 @@ +# TouchDesigner MCP — Pitfalls & Lessons Learned + +Hard-won knowledge from real TD sessions. Read this before building anything. + +## Parameter Names + +### 1. NEVER hardcode parameter names — always discover + +Parameter names change between TD versions. What works in one build may not work in another. ALWAYS use td_get_par_info to discover actual names from TD. + +The agent's LLM training data contains WRONG parameter names. Do not trust them. + +Known historical differences (may vary further — always verify): +| What docs/training say | Actual in some versions | Notes | +|---------------|---------------|-------| +| `dat` | `pixeldat` | GLSL TOP pixel shader DAT | +| `colora` | `alpha` | Constant TOP alpha | +| `sizex` / `sizey` | `size` | Blur TOP (single value) | +| `fontr/g/b/a` | `fontcolorr/g/b/a` | Text TOP font color (r/g/b) | +| `fontcolora` | `fontalpha` | Text TOP font alpha (NOT `fontcolora`) | +| `bgcolora` | `bgalpha` | Text TOP bg alpha | +| `value1name` | `vec0name` | GLSL TOP uniform name | + +### 2. twozero td_execute_python response format + +When calling `td_execute_python` via twozero MCP, successful responses return `(ok)` followed by FPS/error summary (e.g. `[fps 60.0/60] [0 err/0 warn]`), NOT the raw Python `result` dict. If you're parsing responses programmatically, check for the `(ok)` prefix — don't pattern-match on Python variable names from the script. Use `td_get_operator_info` or separate inspection calls to read back values. + +### 3. When using td_set_operator_pars, param names must match exactly + +Use td_get_par_info to discover them. The MCP tool validates parameter names and returns clear errors explaining what went wrong, unlike raw Python which crashes the whole script with tdAttributeError and stops execution. Always discover before setting. + +### 4. Use `safe_par()` pattern for cross-version compatibility + +```python +def safe_par(node, name, value): + p = getattr(node.par, name, None) + if p is not None: + p.val = value + return True + return False +``` + +### 5. `td.tdAttributeError` crashes the whole script — use defensive access + +If you do `node.par.nonexistent = value`, TD raises `tdAttributeError` and stops the entire script. Prevention is better than catching: +- Use `op()` instead of `opex()` — `op()` returns None on failure, `opex()` raises +- Use `hasattr(node.par, 'name')` before accessing any parameter +- Use `getattr(node.par, 'name', None)` with a default +- Use the `safe_par()` pattern from pitfall #3 + +```python +# WRONG — crashes if param doesn't exist: +node.par.nonexistent = value + +# CORRECT — defensive access: +if hasattr(node.par, 'nonexistent'): + node.par.nonexistent = value +``` + +### 6. `outputresolution` is a string menu, not an integer + +``` +menuNames: ['useinput','eighth','quarter','half','2x','4x','8x','fit','limit','custom','parpanel'] +``` +Always use the string form. Setting `outputresolution = 9` may silently fail. +```python +node.par.outputresolution = 'custom' # correct +node.par.resolutionw = 1280; node.par.resolutionh = 720 +``` +Discover valid values: `list(node.par.outputresolution.menuNames)` + +## GLSL Shaders + +### 7. `uTDCurrentTime` does NOT exist in GLSL TOP + +There is NO built-in time uniform for GLSL TOPs. GLSL MAT has `uTDGeneral.seconds` but that's NOT available in GLSL TOP context. + +**PRIMARY — GLSL TOP Vectors/Values page:** +```python +gl.par.value0name = 'uTime' +gl.par.value0.expr = "absTime.seconds" +# In GLSL: uniform float uTime; +``` + +**FALLBACK — Constant TOP texture (for complex time data):** + +CRITICAL: set format to `rgba32float` — default 8-bit clamps to 0-1: +```python +t = root.create(constantTOP, 'time_driver') +t.par.format = 'rgba32float' +t.par.outputresolution = 'custom' +t.par.resolutionw = 1; t.par.resolutionh = 1 +t.par.colorr.expr = "absTime.seconds % 1000.0" +t.outputConnectors[0].connect(glsl.inputConnectors[0]) +``` + +### 8. GLSL compile errors are silent in the API + +The GLSL TOP shows a yellow warning triangle in the UI but `node.errors()` may return empty string. Check `node.warnings()` too, and create an Info DAT pointed at the GLSL TOP to read the actual compiler output. + +### 9. TD GLSL uses `vUV.st` not `gl_FragCoord` — and REQUIRES `TDOutputSwizzle()` on macOS + +Standard GLSL patterns don't work. TD provides: +- `vUV.st` — UV coordinates (0-1) +- `uTDOutputInfo.res.zw` — resolution +- `sTD2DInputs[0]` — input textures +- `layout(location = 0) out vec4 fragColor` — output + +CRITICAL on macOS: Always wrap output with `TDOutputSwizzle()`: +```glsl +fragColor = TDOutputSwizzle(color); +``` +TD uses GLSL 4.60 (Vulkan backend). GLSL 3.30 and earlier removed. + +### 10. Large GLSL shaders — write to temp file + +GLSL code with special characters can corrupt JSON payloads. Write the shader to a temp file and load it in TD: +```python +# Agent side: write shader to /tmp/shader.glsl via write_file +# TD side: +sd = root.create(textDAT, 'shader_code') +with open('/tmp/shader.glsl', 'r') as f: + sd.text = f.read() +``` + +## Node Management + +### 11. Destroying nodes while iterating `root.children` causes `tdError` + +The iterator is invalidated when a child is destroyed. Always snapshot first: +```python +kids = list(root.children) # snapshot +for child in kids: + if child.valid: # check — earlier destroys may cascade + child.destroy() +``` + +### 11b. Split cleanup and creation into SEPARATE td_execute_python calls + +Creating nodes with the same names you just destroyed in the SAME script causes "Invalid OP object" errors — even with `list()` snapshot. TD's internal references can go stale within one execution context. + +**WRONG (single call):** +```python +# td_execute_python: +for c in list(root.children): + if c.valid and c.name.startswith('promo_'): + c.destroy() +# ... then create promo_audio, promo_shader etc. in same script → CRASHES +``` + +**CORRECT (two separate calls):** +```python +# Call 1: td_execute_python — clean only +for c in list(root.children): + if c.valid and c.name.startswith('promo_'): + c.destroy() + +# Call 2: td_execute_python — build (separate MCP call) +audio = root.create(audiofileinCHOP, 'promo_audio') +# ... rest of build +``` + +### 12. Feedback TOP: use `top` parameter, NOT direct input wire + +The feedbackTOP's `top` parameter references which TOP to delay. Do NOT also wire that TOP directly into the feedback's input — this creates a real cook dependency loop. + +Correct setup: +```python +fb = root.create(feedbackTOP, 'fb_delay') +fb.par.top = comp.path # reference only — no wire to fb input +fb.outputConnectors[0].connect(xf) # fb output -> transform -> fade -> comp +``` + +The "Cook dependency loop detected" warning on the transform/fade chain is expected. + +### 13. GLSL TOP auto-creates companion nodes + +Creating a `glslTOP` also creates `name_pixel` (Text DAT), `name_info` (Info DAT), and `name_compute` (Text DAT). These are visible in the network. Don't be alarmed by "extra" nodes. + +### 14. The default project root is `/project1` + +New TD files start with `/project1` as the main container. System nodes live at `/`, `/ui`, `/sys`, `/local`, `/perform`. Don't create user nodes outside `/project1`. + +### 15. Non-Commercial license caps resolution at 1280x1280 + +Setting `resolutionw=1920` silently clamps to 1280. Always check effective resolution after creation: +```python +n.cook(force=True) +actual = str(n.width) + 'x' + str(n.height) +``` + +## Recording & Codecs + +### 16. MovieFileOut TOP: H.264/H.265/AV1 requires Commercial license + +In Non-Commercial TD, these codecs produce an error. Recommended alternatives: +- `prores` — Apple ProRes, **best on macOS**, HW accelerated, NOT license-restricted. ~55MB/s at 1280x720 but lossless quality. **Use this as default on macOS.** +- `cineform` — GoPro Cineform, supports alpha +- `hap` — GPU-accelerated playback, large files +- `notchlc` — GPU-accelerated, good quality +- `mjpa` — Motion JPEG, legacy fallback (lossy, use only if ProRes unavailable) + +For image sequences: `rec.par.type = 'imagesequence'`, `rec.par.imagefiletype = 'png'` + +### 17. MovieFileOut `.record()` method may not exist + +Use the toggle parameter instead: +```python +rec.par.record = True # start recording +rec.par.record = False # stop recording +``` + +When setting file path and starting recording in the same script, use delayFrames: +```python +rec.par.file = '/tmp/new_output.mov' +run("op('/project1/recorder').par.record = True", delayFrames=2) +``` + +### 18. TOP.save() captures same frame when called rapidly + +Use MovieFileOut for real-time recording. Set `project.realTime = False` for frame-accurate output. + +### 19. AudioFileIn CHOP: cue and recording sequence matters + +The recording sequence must be done in exact order, or the recording will be empty, audio will start mid-file, or the file won't be written. + +**Proven recording sequence:** + +```python +# Step 1: Stop any existing recording +rec.par.record = False + +# Step 2: Reset audio to beginning +audio.par.play = False +audio.par.cue = True +audio.par.cuepoint = 0 # may need cuepointunit=0 too +# Verify: audio.par.cue.eval() should be True + +# Step 3: Set output file path +rec.par.file = '/tmp/output.mov' + +# Step 4: Release cue + start playing + start recording (with frame delay) +audio.par.cue = False +audio.par.play = True +audio.par.playmode = 2 # Sequential — plays once through +run("op('/project1/recorder').par.record = True", delayFrames=3) +``` + +**Why each step matters:** +- `rec.par.record = False` first — if a previous recording is active, setting `par.file` may fail silently +- `audio.par.cue = True` + `cuepoint = 0` — guarantees audio starts from the beginning, otherwise the spectrum may be silent for the first few seconds +- `delayFrames=3` on the record start — setting `par.file` and `par.record = True` in the same script can race; the file path needs a frame to register before recording starts +- `playmode = 2` (Sequential) — plays the file once. Use `playmode = 0` (Locked to Timeline) if you want TD's timeline to control position + +## TD Python API Patterns + +### 20. COMP extension setup: ext0object format is CRITICAL + +`ext0object` expects a CONSTANT string (NOT expression mode): +```python +comp.par.ext0object = "op('./myExtensionDat').module.MyClassName(me)" +``` +NEVER set as just the DAT name. NEVER use ParMode.EXPRESSION. ALWAYS ensure the DAT has `par.language='python'`. + +### 21. td.Panel is NOT subscriptable — use attribute access + +```python +comp.panel.select # correct (attribute access, returns float) +comp.panel['select'] # WRONG — 'td.Panel' object is not subscriptable +``` + +### 22. ALWAYS use relative paths in script callbacks + +In scriptTOP/CHOP/SOP/DAT callbacks, use paths relative to `scriptOp` or `me`: +```python +root = scriptOp.parent().parent() +dat = root.op('pixel_data') +``` +NEVER hardcode absolute paths like `op('/project1/myComp/child')` — they break when containers are renamed or copied. + +### 23. keyboardinCHOP channel names have 'k' prefix + +Channel names are `kup`, `kdown`, `kleft`, `kright`, `ka`, `kb`, etc. — NOT `up`, `down`, `a`, `b`. Always verify with: +```python +channels = [c.name for c in op('/project1/keyboard1').chans()] +``` + +### 24. expressCHOP cook-only properties — false positive errors + +`me.inputVal`, `me.chanIndex`, `me.sampleIndex` work ONLY in cook-context. Calling `par.expr0expr.eval()` from outside always raises an error — this is NOT a real operator error. Ignore these in error scans. + +### 25. td.Vertex attributes — use index access not named attributes + +In TD 2025.32, `td.Vertex` objects do NOT have `.x`, `.y`, `.z` attributes: +```python +# WRONG — crashes: +vertex.x, vertex.y, vertex.z + +# CORRECT — index-based: +vertex.point.P[0], vertex.point.P[1], vertex.point.P[2] +# Or for SOP point positions: +pt = sop.points()[i] +pos = pt.P # use P[0], P[1], P[2] +``` + +## Audio + +### 26. Audio Spectrum CHOP output is weak — boost it + +Raw output is very small (0.001-0.05). Use built-in boost: `spectrum.par.highfrequencyboost = 3.0` + +If still weak, add Math CHOP in Range mode: `fromrangehi=0.05, torangehi=1.0` + +### 27. AudioSpectrum CHOP: timeslice and sample count are the #1 gotcha + +AudioSpectrum at 44100Hz with `timeslice=False` outputs the ENTIRE audio file as samples (~24000+). CHOP-to-TOP then exceeds texture resolution max and warns/fails. + +**Fix:** Keep `timeslice = True` (default) for real-time per-frame FFT. Set `fftsize` to control bin count (it's a STRING enum: `'256'` not `256`). + +If the CHOP-to-TOP still gets too many samples, set `layout = 'rowscropped'` on the choptoTOP. + +```python +spectrum.par.fftsize = '256' # STRING, not int — enum values +spectrum.par.timeslice = True # MUST be True for real-time audio reactivity +spectex.par.layout = 'rowscropped' # handles oversized CHOP inputs +``` + +**resampleCHOP has NO `numsamples` param.** It uses `rate`, `start`, `end`, `method`. Don't guess — always `td_get_par_info('resampleCHOP')` first. + +### 28. CHOP To TOP has NO input connectors — use par.chop reference + +```python +spec_tex = root.create(choptoTOP, 'spectrum_tex') +spec_tex.par.chop = resample # correct: parameter reference +# NOT: resample.outputConnectors[0].connect(spec_tex.inputConnectors[0]) # WRONG +``` + +## Workflow + +### 29. Always verify after building — errors are silent + +Node errors and broken connections produce no output. Always check: +```python +for c in list(root.children): + e = c.errors() + w = c.warnings() + if e: print(c.name, 'ERR:', e) + if w: print(c.name, 'WARN:', w) +``` + +### 30. Window COMP param for display target is `winop` + +```python +win = root.create(windowCOMP, 'display') +win.par.winop = '/project1/logo_out' +win.par.winw = 1280; win.par.winh = 720 +win.par.winopen.pulse() +``` + +### 31. `sample()` returns frozen pixels in rapid calls + +`out.sample(x, y)` returns pixels from a single cook snapshot. Compare samples with 2+ second delays, or use screencapture on the display window. + +### 32. Audio-reactive GLSL: dual-layer sync pipeline + +For audio-synced visuals, use BOTH layers for maximum effect: + +**Layer 1 (TD-side, real-time):** AudioFileIn → AudioSpectrum(timeslice=True, fftsize='256') → Math(gain=5) → choptoTOP(par.chop=math, layout='rowscropped') → GLSL input. The shader samples `sTD2DInputs[1]` at different x positions for bass/mid/hi. Record the TD output with MovieFileOut. + +**Layer 2 (Python-side, post-hoc):** scipy FFT on the SAME audio file → per-frame features (rms, bass, mid, hi, beat detection) → drive ASCII brightness, chromatic aberration, beat flashes during the render pass. + +Both layers locked to the same audio file = visuals genuinely sync to the beat at two independent stages. + +**Key gotcha:** AudioFileIn must be cued (`par.cue=True` → `par.cuepulse.pulse()`) then uncued (`par.cue=False`, `par.play=True`) before recording starts. Otherwise the spectrum is silent for the first few seconds. + +### 33. twozero MCP: benchmark and prefer native tools + +Benchmarked April 2026: twozero MCP with 36 native tools. The old curl/REST method (port 9981) had zero native tools. + +**Always prefer native MCP tools over td_execute_python:** +- `td_create_operator` over `root.create()` scripts (handles viewport positioning) +- `td_set_operator_pars` over `node.par.X = Y` scripts (validates param names) +- `td_get_par_info` over temp-node discovery dance (instant, no cleanup) +- `td_get_errors` over manual `c.errors()` loops +- `td_get_focus` for context awareness (no equivalent in old method) + +Only fall back to `td_execute_python` for multi-step logic (wiring chains, conditional builds, loops). + +### 34. twozero td_execute_python response wrapping + +twozero wraps `td_execute_python` responses with status info: `(ok)\n\n[fps 60.0/60] [0 err/0 warn]`. Your Python `result` variable value may not appear verbatim in the response text. If you need to check results programmatically, use `print()` statements in the script — they appear in the response. Don't rely on string-matching the `result` dict. + +### 35. Audio-reactive chain: DO NOT use Lag CHOP or Filter CHOP for spectrum smoothing + +The Derivative docs and tutorials suggest using Lag CHOP (lag1=0.2, lag2=0.5) to smooth raw FFT output before passing to a shader. **This does NOT work with AudioSpectrum → CHOP to TOP → GLSL.** + +What happens: Lag CHOP operates in timeslice mode. A 256-sample spectrum input gets expanded to 1600-2400 samples. The Lag averaging drives all values to near-zero (~1e-06). The CHOP to TOP produces a 2400x2 texture instead of 256x2. The shader receives effectively zero audio data. + +**The correct chain is: Spectrum(outlength=256) → Math(gain=10) → CHOPtoTOP → GLSL.** No CHOP smoothing at all. If you need smoothing, do it in the GLSL shader via temporal lerp with a feedback texture. + +Verified values with audio playing: +- Without Lag CHOP: bass bins = 5.0-5.4, mid bins = 1.0-1.7 (strong, usable) +- With Lag CHOP: ALL bins = 0.000001-0.00004 (dead, zero audio reactivity) + +### 36. AudioSpectrum Output Length: set manually to avoid CHOP to TOP overflow + +AudioSpectrum in Visualization mode with FFT 8192 outputs 22,050 samples by default (1 per Hz, 0–22050). CHOP to TOP cannot handle this — you get "Number of samples exceeded texture resolution max". + +Fix: `spectrum.par.outputmenu = 'setmanually'` and `spectrum.par.outlength = 256`. This gives 256 frequency bins — plenty for visual FFT. + +DO NOT set `timeslice = False` as a workaround — that processes the entire audio file at once and produces even more samples. + +### 37. GLSL spectrum texture from CHOP to TOP is 256x2 not 256x1 + +AudioSpectrum outputs 2 channels (stereo: chan1, chan2). CHOP to TOP with `dataformat='r'` creates a 256x2 texture — one row per channel. Sample the first channel at `y=0.25` (center of first row), NOT `y=0.5` (boundary between rows): + +```glsl +float bass = texture(sTD2DInputs[1], vec2(0.05, 0.25)).r; // correct +float bass = texture(sTD2DInputs[1], vec2(0.05, 0.5)).r; // WRONG — samples between rows +``` + +### 38. FPS=0 doesn't mean ops aren't cooking — check play state + +TD can show `fps:0` in `td_get_perf` while ops still cook and `TOP.save()` still produces valid screenshots. The two most common causes: + +**a) Project is paused (playbar stopped).** TD's playbar can be toggled with spacebar. The `root` at `/` has no `.playbar` attribute (it's on the perform COMP). The easiest fix is sending a spacebar keypress via `td_input_execute`, though this tool can sometimes error. As a workaround, `TOP.save()` always works regardless of play state — use it to verify rendering is actually happening before spending time debugging FPS. + +**b) Audio device CHOP blocking the main thread.** An `audiooutCHOP` with an active audio device can consume 300-400ms/s (2000%+ of frame budget), stalling the cook loop at FPS=0. Fix: keep the CHOP active but set `volume=0` to prevent the audio driver from blocking. Disabling it entirely (`active=False`) may also work but can prevent downstream audio processing CHOPs from cooking. + +Diagnostic sequence when FPS=0: +1. `td_get_perf` — check if any op has extreme CPU/s +2. `TOP.save()` on the output — if it produces a valid image, the pipeline works, just not at real-time rate +3. Check for blocking CHOPs (audioout, audiodevin, etc.) +4. Toggle play state (spacebar, or check if absTime.seconds is advancing) + +### 39. Recording while FPS=0 produces empty or near-empty files + +This is the #1 cause of "I recorded for 30 seconds but got a 2-frame video." If TD's cook loop is stalled (FPS=0 or very low), MovieFileOut has nothing to record. Unlike `TOP.save()` which captures the last cooked frame regardless, MovieFileOut only writes frames that actually cook. + +**Always verify FPS before starting a recording:** +```python +# Check via td_get_perf first +# If FPS < 30, do NOT start recording — fix the performance issue first +# If FPS=0, the playbar is likely paused — see pitfall #37 +``` + +Common causes of recording empty video: +- Playbar paused (FPS=0) — see pitfall #37 +- Audio device CHOP blocking the main thread — see pitfall #37b +- Recording started before audio was cued — audio is silent, GLSL outputs black, MovieFileOut records black frames that look empty +- `par.file` set in the same script as `par.record = True` — see pitfall #18 + +### 40. GLSL shader produces black output — test before committing to a long render + +New GLSL shaders can fail silently (see pitfall #7). Before recording a long take, always: + +1. **Write a minimal test shader first** that just outputs a solid color or pass-through: +```glsl +void main() { + vec2 uv = vUV.st; + fragColor = TDOutputSwizzle(vec4(uv, 0.0, 1.0)); +} +``` + +2. **Verify the test renders correctly** via `td_get_screenshot` on the GLSL TOP's output. + +3. **Swap in the real shader** and screenshot again immediately. If black, the shader has a compile error or logic issue. + +4. **Only then start recording.** A 90-second ProRes recording is ~5GB. Recording black frames wastes disk and time. + +Common causes of black GLSL output: +- Missing `TDOutputSwizzle()` on macOS (pitfall #8) +- Time uniform not connected — shader uses default 0.0, fractal stays at origin +- Spectrum texture not connected — audio values all 0.0, driving everything to black +- Integer division where float division was expected (`1/2 = 0` not `0.5`) +- `absTime.seconds % 1000.0` rolled over past 1000 and the modulo produces unexpected values + +### 41. td_write_dat uses `text` parameter, NOT `content` + +The MCP tool `td_write_dat` expects a `text` parameter for full replacement. Passing `content` returns an error: `"Provide either 'text' for full replace, or 'old_text'+'new_text' for patching"`. + +If `td_write_dat` fails, fall back to `td_execute_python`: +```python +op("/project1/shader_code").text = shader_string +``` + +### 42. td_execute_python does NOT return stdout or print() output + +Despite what earlier versions of pitfall #33 stated, `print()` and `debug()` output from `td_execute_python` scripts does NOT appear in the MCP response. The response is always just `(ok)` + FPS/error summary. To read values back, use dedicated inspection tools (`td_get_operator_info`, `td_read_dat`, `td_read_chop`) instead of trying to print from within a script. + +### 43. td_get_operator_info JSON is appended with `[fps X.X/X]` — breaks json.loads() + +The response text from `td_get_operator_info` has `[fps 60.0/60]` appended after the JSON object. This causes `json.loads()` to fail with "Extra data" errors. Strip it before parsing: +```python +clean = response_text.rsplit('[fps', 1)[0] +data = json.loads(clean) +``` + +### 44. td_get_screenshot is asynchronous — returns `{"status": "pending"}` + +Screenshots don't complete instantly. The tool returns `{"status": "pending", "requestId": "..."}` and the actual file appears later. Wait a few seconds before checking for the file. There is no callback or completion notification — poll the filesystem. + +### 45. Recording duration is manual — no auto-stop at audio end + +MovieFileOut records until `par.record = False` is set. If audio ends before you stop recording, the file keeps growing with repeated frames. Always stop recording promptly after the audio duration. For precision: set a timer on the agent side matching the audio length, then send `par.record = False`. Trim excess with ffmpeg as a safety net: +```bash +ffmpeg -i raw.mov -t 25 -c copy trimmed.mov +``` \ No newline at end of file diff --git a/optional-skills/creative/touchdesigner-mcp/references/python-api.md b/optional-skills/creative/touchdesigner-mcp/references/python-api.md new file mode 100644 index 000000000000..f2955110b0ef --- /dev/null +++ b/optional-skills/creative/touchdesigner-mcp/references/python-api.md @@ -0,0 +1,463 @@ +# TouchDesigner Python API Reference + +## The td Module + +TouchDesigner's Python environment auto-imports the `td` module. All TD-specific classes, functions, and constants live here. Scripts inside TD (Script DATs, CHOP/DAT Execute callbacks, Extensions) have full access. + +When using the MCP `execute_python_script` tool, these globals are pre-loaded: +- `op` — shortcut for `td.op()`, finds operators by path +- `ops` — shortcut for `td.ops()`, finds multiple operators by pattern +- `me` — the operator running the script (via MCP this is the twozero internal executor) +- `parent` — shortcut for `me.parent()` +- `project` — the root project component +- `td` — the full td module + +## Finding Operators: op() and ops() + +### op(path) — Find a single operator + +```python +# Absolute path (always works from MCP) +node = op('/project1/noise1') + +# Relative path (relative to current operator — only in Script DATs) +node = op('noise1') # sibling +node = op('../noise1') # parent's sibling + +# Returns None if not found (does NOT raise) +node = op('/project1/nonexistent') # None +``` + +### ops(pattern) — Find multiple operators + +```python +# Glob patterns +nodes = ops('/project1/noise*') # all nodes starting with "noise" +nodes = ops('/project1/*') # all direct children +nodes = ops('/project1/container1/*') # all children of container1 + +# Returns a tuple of operators (may be empty) +for n in ops('/project1/*'): + print(n.name, n.OPType) +``` + +### Navigation from a node + +```python +node = op('/project1/noise1') + +node.name # 'noise1' +node.path # '/project1/noise1' +node.OPType # 'noiseTop' +node.type # +node.family # 'TOP' + +# Parent / children +node.parent() # the parent COMP +node.parent().children # all siblings + self +node.parent().findChildren(name='noise*') # filtered + +# Type checking +node.isTOP # True +node.isCHOP # False +node.isSOP # False +node.isDAT # False +node.isMAT # False +node.isCOMP # False +``` + +## Parameters + +Every operator has parameters accessed via the `.par` attribute. + +### Reading parameters + +```python +node = op('/project1/noise1') + +# Direct access +node.par.seed.val # current evaluated value (may be an expression result) +node.par.seed.eval() # same as .val +node.par.seed.default # default value +node.par.monochrome.val # boolean parameters: True/False + +# List all parameters +for p in node.pars(): + print(f"{p.name}: {p.val} (default: {p.default})") + +# Filter by page (parameter group) +for p in node.pars('Noise'): # page name + print(f"{p.name}: {p.val}") +``` + +### Setting parameters + +```python +# Direct value setting +node.par.seed.val = 42 +node.par.monochrome.val = True +node.par.resolutionw.val = 1920 +node.par.resolutionh.val = 1080 + +# String parameters +op('/project1/text1').par.text.val = 'Hello World' + +# File paths +op('/project1/moviefilein1').par.file.val = '/path/to/video.mp4' + +# Reference another operator (for "dat", "chop", "top" type parameters) +op('/project1/glsl1').par.dat.val = '/project1/shader_code' +``` + +### Parameter expressions + +```python +# Python expressions that evaluate dynamically +node.par.seed.expr = "me.time.frame" +node.par.tx.expr = "math.sin(me.time.seconds * 2)" + +# Reference another parameter +node.par.brightness1.expr = "op('/project1/constant1').par.value0.val" + +# Export (one-way binding from CHOP to parameter) +# This makes the parameter follow a CHOP channel value +op('/project1/noise1').par.seed.val # can also be driven by exports +``` + +### Parameter types + +| Type | Python Type | Example | +|------|------------|---------| +| Float | `float` | `node.par.brightness1.val = 0.5` | +| Int | `int` | `node.par.seed.val = 42` | +| Toggle | `bool` | `node.par.monochrome.val = True` | +| String | `str` | `node.par.text.val = 'hello'` | +| Menu | `int` (index) or `str` (label) | `node.par.type.val = 'sine'` | +| File | `str` (path) | `node.par.file.val = '/path/to/file'` | +| OP reference | `str` (path) | `node.par.dat.val = '/project1/text1'` | +| Color | separate r/g/b/a floats | `node.par.colorr.val = 1.0` | +| XY/XYZ | separate x/y/z floats | `node.par.tx.val = 0.5` | + +## Creating and Deleting Operators + +```python +# Create via parent component +parent = op('/project1') +new_node = parent.create(noiseTop) # using class reference +new_node = parent.create(noiseTop, 'my_noise') # with custom name + +# The MCP create_td_node tool handles this automatically: +# create_td_node(parentPath="/project1", nodeType="noiseTop", nodeName="my_noise") + +# Delete +node = op('/project1/my_noise') +node.destroy() + +# Copy +original = op('/project1/noise1') +copy = parent.copy(original, name='noise1_copy') +``` + +## Connections (Wiring Operators) + +### Output to Input connections + +```python +# Connect noise1's output to level1's input +op('/project1/noise1').outputConnectors[0].connect(op('/project1/level1')) + +# Connect to specific input index (for multi-input operators like Composite) +op('/project1/noise1').outputConnectors[0].connect(op('/project1/composite1').inputConnectors[0]) +op('/project1/text1').outputConnectors[0].connect(op('/project1/composite1').inputConnectors[1]) + +# Disconnect all outputs +op('/project1/noise1').outputConnectors[0].disconnect() + +# Query connections +node = op('/project1/level1') +inputs = node.inputs # list of connected input operators +outputs = node.outputs # list of connected output operators +``` + +### Connection patterns for common setups + +```python +# Linear chain: A -> B -> C -> D +ops_list = [op(f'/project1/{name}') for name in ['noise1', 'level1', 'blur1', 'null1']] +for i in range(len(ops_list) - 1): + ops_list[i].outputConnectors[0].connect(ops_list[i+1]) + +# Fan-out: A -> B, A -> C, A -> D +source = op('/project1/noise1') +for target_name in ['level1', 'composite1', 'transform1']: + source.outputConnectors[0].connect(op(f'/project1/{target_name}')) + +# Merge: A + B + C -> Composite +comp = op('/project1/composite1') +for i, source_name in enumerate(['noise1', 'text1', 'ramp1']): + op(f'/project1/{source_name}').outputConnectors[0].connect(comp.inputConnectors[i]) +``` + +## DAT Content Manipulation + +### Text DATs + +```python +dat = op('/project1/text1') + +# Read +content = dat.text # full text as string + +# Write +dat.text = "new content" +dat.text = '''multi +line +content''' + +# Append +dat.text += "\nnew line" +``` + +### Table DATs + +```python +dat = op('/project1/table1') + +# Read cell +val = dat[0, 0] # row 0, col 0 +val = dat[0, 'name'] # row 0, column named 'name' +val = dat['key', 1] # row named 'key', col 1 + +# Write cell +dat[0, 0] = 'value' + +# Read row/col +row = dat.row(0) # list of Cell objects +col = dat.col('name') # list of Cell objects + +# Dimensions +rows = dat.numRows +cols = dat.numCols + +# Append row +dat.appendRow(['col1_val', 'col2_val', 'col3_val']) + +# Clear +dat.clear() + +# Set entire table +dat.clear() +dat.appendRow(['name', 'value', 'type']) +dat.appendRow(['frequency', '440', 'float']) +dat.appendRow(['amplitude', '0.8', 'float']) +``` + +## Time and Animation + +```python +# Global time +td.absTime.frame # absolute frame number (never resets) +td.absTime.seconds # absolute seconds + +# Timeline time (affected by play/pause/loop) +me.time.frame # current frame on timeline +me.time.seconds # current seconds on timeline +me.time.rate # FPS setting + +# Timeline control (via execute_python_script) +project.play = True +project.play = False +project.frameRange = (1, 300) # set timeline range + +# Cook frame (when operator was last computed) +node.cookFrame +node.cookTime +``` + +## Extensions (Custom Python Classes on Components) + +Extensions add custom Python methods and attributes to COMPs. + +```python +# Create extension on a Base COMP +base = op('/project1/myBase') + +# The extension class is defined in a Text DAT inside the COMP +# Typically named 'ExtClass' with the extension code: + +extension_code = ''' +class MyExtension: + def __init__(self, ownerComp): + self.ownerComp = ownerComp + self.counter = 0 + + def Reset(self): + self.counter = 0 + + def Increment(self): + self.counter += 1 + return self.counter + + @property + def Count(self): + return self.counter +''' + +# Write extension code to DAT inside the COMP +op('/project1/myBase/extClass').text = extension_code + +# Configure the extension on the COMP +base.par.extension1 = 'extClass' # name of the DAT +base.par.promoteextension1 = True # promote methods to parent + +# Call extension methods +base.Increment() # calls MyExtension.Increment() +count = base.Count # accesses MyExtension.Count property +base.Reset() +``` + +## Useful Built-in Modules + +### tdu — TouchDesigner Utilities + +```python +import tdu + +# Dependency tracking (reactive values) +dep = tdu.Dependency(initial_value) +dep.val = new_value # triggers dependents to recook + +# File path utilities +tdu.expandPath('$HOME/Desktop/output.mov') + +# Math +tdu.clamp(value, min, max) +tdu.remap(value, from_min, from_max, to_min, to_max) +``` + +### TDFunctions + +```python +from TDFunctions import * + +# Commonly used utilities +clamp(value, low, high) +remap(value, inLow, inHigh, outLow, outHigh) +interp(value1, value2, t) # linear interpolation +``` + +### TDStoreTools — Persistent Storage + +```python +from TDStoreTools import StorageManager + +# Store data that survives project reload +me.store('myKey', 'myValue') +val = me.fetch('myKey', default='fallback') + +# Storage dict +me.storage['key'] = value +``` + +## Common Patterns via execute_python_script + +### Build a complete chain + +```python +# Create a complete audio-reactive noise chain +parent = op('/project1') + +# Create operators +audio_in = parent.create(audiofileinChop, 'audio_in') +spectrum = parent.create(audiospectrumChop, 'spectrum') +chop_to_top = parent.create(choptopTop, 'chop_to_top') +noise = parent.create(noiseTop, 'noise1') +level = parent.create(levelTop, 'level1') +null_out = parent.create(nullTop, 'out') + +# Wire the chain +audio_in.outputConnectors[0].connect(spectrum) +spectrum.outputConnectors[0].connect(chop_to_top) +noise.outputConnectors[0].connect(level) +level.outputConnectors[0].connect(null_out) + +# Set parameters +audio_in.par.file = '/path/to/music.wav' +audio_in.par.play = True +spectrum.par.size = 512 +noise.par.type = 1 # Sparse +noise.par.monochrome = False +noise.par.resolutionw = 1920 +noise.par.resolutionh = 1080 +level.par.opacity = 0.8 +level.par.gamma1 = 0.7 +``` + +### Query network state + +```python +# Get all TOPs in the project +tops = [c for c in op('/project1').findChildren(type=TOP)] +for t in tops: + print(f"{t.path}: {t.OPType} {'ERROR' if t.errors() else 'OK'}") + +# Find all operators with errors +def find_errors(parent_path='/project1'): + parent = op(parent_path) + errors = [] + for child in parent.findChildren(depth=-1): + if child.errors(): + errors.append((child.path, child.errors())) + return errors + +result = find_errors() +``` + +### Batch parameter changes + +```python +# Set parameters on multiple nodes at once +settings = { + '/project1/noise1': {'seed': 42, 'monochrome': False, 'resolutionw': 1920}, + '/project1/level1': {'brightness1': 1.2, 'gamma1': 0.8}, + '/project1/blur1': {'sizex': 5, 'sizey': 5}, +} + +for path, params in settings.items(): + node = op(path) + if node: + for key, val in params.items(): + setattr(node.par, key, val) +``` + +## Python Version and Packages + +TouchDesigner bundles Python 3.11+ with these pre-installed: +- **numpy** — array operations, fast math +- **scipy** — signal processing, FFT +- **OpenCV** (cv2) — computer vision +- **PIL/Pillow** — image processing +- **requests** — HTTP client +- **json**, **re**, **os**, **sys** — standard library + +**IMPORTANT:** Parameter names in examples below are illustrative. Always run discovery (SKILL.md Step 0) to get actual names for your TD version. Do NOT copy param names from these examples verbatim. + +Custom packages can be installed to TD's Python site-packages directory. See TD documentation for the exact path per platform. + +## SOP Vertex/Point Access (TD 2025.32) + +In TD 2025.32, `td.Vertex` does NOT have `.x`, `.y`, `.z` attributes. Use index access: + +```python +# WRONG — crashes in TD 2025.32: +vertex.x, vertex.y, vertex.z + +# CORRECT — index/attribute access: +pt = sop.points()[i] +pos = pt.P # Position object +x, y, z = pos[0], pos[1], pos[2] + +# Always introspect first: +dir(sop.points()[0]) # see what attributes actually exist +dir(sop.points()[0].P) # see Position object interface +``` diff --git a/optional-skills/creative/touchdesigner-mcp/references/troubleshooting.md b/optional-skills/creative/touchdesigner-mcp/references/troubleshooting.md new file mode 100644 index 000000000000..b8e201f5c32d --- /dev/null +++ b/optional-skills/creative/touchdesigner-mcp/references/troubleshooting.md @@ -0,0 +1,244 @@ +# TouchDesigner Troubleshooting (twozero MCP) + +> See `references/pitfalls.md` for the comprehensive lessons-learned list. + +## 1. Connection Issues + +### Port 40404 not responding + +Check these in order: + +1. Is TouchDesigner running? + ```bash + pgrep TouchDesigner + ``` + +1b. Quick hub health check (no JSON-RPC needed): + A plain GET to the MCP URL returns instance info: + ``` + curl -s http://localhost:40404/mcp + ``` + Returns: `{"hub": true, "pid": ..., "instances": {"127.0.0.1_PID": {"project": "...", "tdVersion": "...", ...}}}` + If this returns JSON but `instances` is empty, TD is running but twozero hasn't registered yet. + +2. Is twozero installed in TD? + Open TD Palette Browser > twozero should be listed. If not, install it. + +3. Is MCP enabled in twozero settings? + In TD, open twozero preferences and confirm MCP server is toggled ON. + +4. Test the port directly: + ```bash + nc -z 127.0.0.1 40404 + ``` + +5. Test the MCP endpoint: + ```bash + curl -s http://localhost:40404/mcp + ``` + Should return JSON with hub info. If it does, the server is running. + +### Hub responds but no TD instances + +The twozero MCP hub is running but TD hasn't registered. Causes: +- TD project not loaded yet (still on splash screen) +- twozero COMP not initialized in the current project +- twozero version mismatch + +Fix: Open/reload a TD project that contains the twozero COMP. Use td_list_instances +to check which TD instances are registered. + +### Multi-instance setup + +twozero auto-assigns ports for multiple TD instances: +- First instance: 40404 +- Second instance: 40405 +- Third instance: 40406 +- etc. + +Use `td_list_instances` to discover all running instances and their ports. + +## 2. MCP Tool Errors + +### td_execute_python returns error + +The error message from td_execute_python often contains the Python traceback. +If it's unclear, use `td_read_textport` to see the full TD console output — +Python exceptions are always printed there. + +Common causes: +- Syntax error in the script +- Referencing a node that doesn't exist (op() returns None, then you call .par on None) +- Using wrong parameter names (see pitfalls.md) + +### td_set_operator_pars fails + +Parameter name mismatch is the #1 cause. The tool validates param names and +returns clear errors, but you must use exact names. + +Fix: ALWAYS call `td_get_par_info` first to discover the real parameter names: +``` +td_get_par_info(op_type='glslTOP') +td_get_par_info(op_type='noiseTOP') +``` + +### td_create_operator type name errors + +Operator type names use camelCase with family suffix: +- CORRECT: noiseTOP, glslTOP, levelTOP, compositeTOP, audiospectrumCHOP +- WRONG: NoiseTOP, noise_top, NOISE TOP, Noise + +### td_get_operator_info for deep inspection + +If unsure about any aspect of an operator (params, inputs, outputs, state): +``` +td_get_operator_info(path='/project1/noise1', detail='full') +``` + +## 3. Parameter Discovery + +CRITICAL: ALWAYS use td_get_par_info to discover parameter names. + +The agent's LLM training data contains WRONG parameter names for TouchDesigner. +Do not trust them. Known wrong names include dat vs pixeldat, colora vs alpha, +sizex vs size, and many more. See pitfalls.md for the full list. + +Workflow: +1. td_get_par_info(op_type='glslTOP') — get all params for a type +2. td_get_operator_info(path='/project1/mynode', detail='full') — get params for a specific instance +3. Use ONLY the names returned by these tools + +## 4. Performance + +### Diagnosing slow performance + +Use `td_get_perf` to see which operators are slow. Look at cook times — +anything over 1ms per frame is worth investigating. + +Common causes: +- Resolution too high (especially on Non-Commercial) +- Complex GLSL shaders +- Too many TOP-to-CHOP or CHOP-to-TOP transfers (GPU-CPU memory copies) +- Feedback loops without decay (values accumulate, memory grows) + +### Non-Commercial license restrictions + +- Resolution cap: 1280x1280. Setting resolutionw=1920 silently clamps to 1280. +- H.264/H.265/AV1 encoding requires Commercial license. Use ProRes or Hap instead. +- No commercial use of output. + +Always check effective resolution after creation: +```python +n.cook(force=True) +actual = str(n.width) + 'x' + str(n.height) +``` + +## 5. Hermes Configuration + +### Config location + +`$HERMES_HOME/config.yaml` (defaults to `~/.hermes/config.yaml` when `HERMES_HOME` is unset) + +### MCP entry format + +The twozero TD entry should look like: +```yaml +mcpServers: + twozero_td: + url: http://localhost:40404/mcp +``` + +### After config changes + +Restart the Hermes session for changes to take effect. The MCP connection is +established at session startup. + +### Verifying MCP tools are available + +After restarting, the session log should show twozero MCP tools registered. +If tools show as registered but aren't callable, check: +- The twozero MCP hub is still running (curl test above) +- TD is still running with a project loaded +- No firewall blocking localhost:40404 + +## 6. Node Creation Issues + +### "Node type not found" error + +Wrong type string. Use camelCase with family suffix: +- Wrong: NoiseTop, noise_top, NOISE TOP +- Right: noiseTOP + +### Node created but not visible + +Check parentPath — use absolute paths like /project1. The default project +root is /project1. System nodes live at /, /ui, /sys, /local, /perform. +Don't create user nodes outside /project1. + +### Cannot create node inside a non-COMP + +Only COMP operators (Container, Base, Geometry, etc.) can contain children. +You cannot create nodes inside a TOP, CHOP, SOP, DAT, or MAT. + +## 7. Wiring Issues + +### Cross-family wiring + +TOPs connect to TOPs, CHOPs to CHOPs, SOPs to SOPs, DATs to DATs. +Use converter operators to bridge: choptoTOP, topToCHOP, soptoDAT, etc. + +Note: choptoTOP has NO input connectors. Use par.chop reference instead: +```python +spec_tex.par.chop = resample_node # correct +# NOT: resample.outputConnectors[0].connect(spec_tex.inputConnectors[0]) +``` + +### Feedback loops + +Never create A -> B -> A directly. Use a Feedback TOP: +```python +fb = root.create(feedbackTOP, 'fb') +fb.par.top = comp.path # reference only, no wire to fb input +fb.outputConnectors[0].connect(next_node) +``` +"Cook dependency loop detected" warning on the chain is expected and correct. + +## 8. GLSL Issues + +### Shader compilation errors are silent + +GLSL TOP shows a yellow warning in the UI but node.errors() may return empty. +Check node.warnings() too. Create an Info DAT pointed at the GLSL TOP for +full compiler output. + +### TD GLSL specifics + +- Uses GLSL 4.60 (Vulkan backend). GLSL 3.30 and earlier removed. +- UV coordinates: vUV.st (not gl_FragCoord) +- Input textures: sTD2DInputs[0] +- Output: layout(location = 0) out vec4 fragColor +- macOS CRITICAL: Always wrap output with TDOutputSwizzle(color) +- No built-in time uniform. Pass time via GLSL TOP Values page or Constant TOP. + +## 9. Recording Issues + +### H.264/H.265/AV1 requires Commercial license + +Use Apple ProRes on macOS (hardware accelerated, not license-restricted): +```python +rec.par.videocodec = 'prores' # Preferred on macOS — lossless, Non-Commercial OK +# rec.par.videocodec = 'mjpa' # Fallback — lossy, works everywhere +``` + +### MovieFileOut has no .record() method + +Use the toggle parameter: +```python +rec.par.record = True # start +rec.par.record = False # stop +``` + +### All exported frames identical + +TOP.save() captures same frame when called rapidly. Use MovieFileOut for +real-time recording. Set project.realTime = False for frame-accurate output. diff --git a/optional-skills/creative/touchdesigner-mcp/scripts/setup.sh b/optional-skills/creative/touchdesigner-mcp/scripts/setup.sh new file mode 100644 index 000000000000..15dc662c1cdf --- /dev/null +++ b/optional-skills/creative/touchdesigner-mcp/scripts/setup.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# setup.sh — Automated setup for twozero MCP plugin for TouchDesigner +# Idempotent: safe to run multiple times. +set -euo pipefail + +GREEN='\033[0;32m'; RED='\033[0;31m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m' +OK="${GREEN}✔${NC}"; FAIL="${RED}✘${NC}"; WARN="${YELLOW}⚠${NC}" + +TWOZERO_URL="https://www.404zero.com/pisang/twozero.tox" +TOX_PATH="$HOME/Downloads/twozero.tox" +HERMES_HOME_DIR="${HERMES_HOME:-$HOME/.hermes}" +HERMES_CFG="${HERMES_HOME_DIR}/config.yaml" +MCP_PORT=40404 +MCP_ENDPOINT="http://localhost:${MCP_PORT}/mcp" + +manual_steps=() + +echo -e "\n${CYAN}═══ twozero MCP for TouchDesigner — Setup ═══${NC}\n" + +# ── 1. Check if TouchDesigner is running ── +# Match on process *name* (not full cmdline) to avoid self-matching shells +# that happen to have "TouchDesigner" in their args. macOS and Linux pgrep +# both support -x for exact name match. +if pgrep -x TouchDesigner >/dev/null 2>&1 || pgrep -x TouchDesignerFTE >/dev/null 2>&1; then + echo -e " ${OK} TouchDesigner is running" + td_running=true +else + echo -e " ${WARN} TouchDesigner is not running" + td_running=false +fi + +# ── 2. Ensure twozero.tox exists ── +if [[ -f "$TOX_PATH" ]]; then + echo -e " ${OK} twozero.tox already exists at ${TOX_PATH}" +else + echo -e " ${WARN} twozero.tox not found — downloading..." + if curl -fSL -o "$TOX_PATH" "$TWOZERO_URL" 2>/dev/null; then + echo -e " ${OK} Downloaded twozero.tox to ${TOX_PATH}" + else + echo -e " ${FAIL} Failed to download twozero.tox from ${TWOZERO_URL}" + echo " Please download manually and place at ${TOX_PATH}" + manual_steps+=("Download twozero.tox from ${TWOZERO_URL} to ${TOX_PATH}") + fi +fi + +# ── 3. Ensure Hermes config has twozero_td MCP entry ── +if [[ ! -f "$HERMES_CFG" ]]; then + echo -e " ${FAIL} Hermes config not found at ${HERMES_CFG}" + manual_steps+=("Create ${HERMES_CFG} with twozero_td MCP server entry") +elif grep -q 'twozero_td' "$HERMES_CFG" 2>/dev/null; then + echo -e " ${OK} twozero_td MCP entry exists in Hermes config" +else + echo -e " ${WARN} Adding twozero_td MCP entry to Hermes config..." + python3 -c " +import yaml, sys, copy + +cfg_path = '$HERMES_CFG' +with open(cfg_path, 'r') as f: + cfg = yaml.safe_load(f) or {} + +if 'mcp_servers' not in cfg: + cfg['mcp_servers'] = {} + +if 'twozero_td' not in cfg['mcp_servers']: + cfg['mcp_servers']['twozero_td'] = { + 'url': '${MCP_ENDPOINT}', + 'timeout': 120, + 'connect_timeout': 60 + } + with open(cfg_path, 'w') as f: + yaml.dump(cfg, f, default_flow_style=False, sort_keys=False) +" 2>/dev/null && echo -e " ${OK} twozero_td MCP entry added to config" \ + || { echo -e " ${FAIL} Could not update config (is PyYAML installed?)"; \ + manual_steps+=("Add twozero_td MCP entry to ${HERMES_CFG} manually"); } + manual_steps+=("Restart Hermes session to pick up config change") +fi + +# ── 4. Test if MCP port is responding ── +if nc -z 127.0.0.1 "$MCP_PORT" 2>/dev/null; then + echo -e " ${OK} Port ${MCP_PORT} is open" + + # ── 5. Verify MCP endpoint responds ── + resp=$(curl -s --max-time 3 "$MCP_ENDPOINT" 2>/dev/null || true) + if [[ -n "$resp" ]]; then + echo -e " ${OK} MCP endpoint responded at ${MCP_ENDPOINT}" + else + echo -e " ${WARN} Port open but MCP endpoint returned empty response" + manual_steps+=("Verify MCP is enabled in twozero settings") + fi +else + echo -e " ${WARN} Port ${MCP_PORT} is not open" + if [[ "$td_running" == true ]]; then + manual_steps+=("In TD: drag twozero.tox into network editor → click Install") + manual_steps+=("Enable MCP: twozero icon → Settings → mcp → 'auto start MCP' → Yes") + else + manual_steps+=("Launch TouchDesigner") + manual_steps+=("Drag twozero.tox into the TD network editor and click Install") + manual_steps+=("Enable MCP: twozero icon → Settings → mcp → 'auto start MCP' → Yes") + fi +fi + +# ── Status Report ── +echo -e "\n${CYAN}═══ Status Report ═══${NC}\n" + +if [[ ${#manual_steps[@]} -eq 0 ]]; then + echo -e " ${OK} ${GREEN}Fully configured! twozero MCP is ready to use.${NC}\n" + exit 0 +else + echo -e " ${WARN} ${YELLOW}Manual steps remaining:${NC}\n" + for i in "${!manual_steps[@]}"; do + echo -e " $((i+1)). ${manual_steps[$i]}" + done + echo "" + exit 1 +fi diff --git a/optional-skills/dogfood/DESCRIPTION.md b/optional-skills/dogfood/DESCRIPTION.md new file mode 100644 index 000000000000..f083fd72bdd2 --- /dev/null +++ b/optional-skills/dogfood/DESCRIPTION.md @@ -0,0 +1,3 @@ +# Dogfood — Advanced QA & Testing Skills + +Specialized QA workflows that go beyond basic bug-finding. These skills use structured methodologies to surface UX friction, accessibility issues, and product-level problems that standard testing misses. diff --git a/optional-skills/dogfood/adversarial-ux-test/SKILL.md b/optional-skills/dogfood/adversarial-ux-test/SKILL.md new file mode 100644 index 000000000000..1777e083d1b6 --- /dev/null +++ b/optional-skills/dogfood/adversarial-ux-test/SKILL.md @@ -0,0 +1,190 @@ +--- +name: adversarial-ux-test +description: Roleplay the most difficult, tech-resistant user for your product. Browse the app as that persona, find every UX pain point, then filter complaints through a pragmatism layer to separate real problems from noise. Creates actionable tickets from genuine issues only. +version: 1.0.0 +author: Omni @ Comelse +license: MIT +metadata: + hermes: + tags: [qa, ux, testing, adversarial, dogfood, personas, user-testing] + related_skills: [dogfood] +--- + +# Adversarial UX Test + +Roleplay the worst-case user for your product — the person who hates technology, doesn't want your software, and will find every reason to complain. Then filter their feedback through a pragmatism layer to separate real UX problems from "I hate computers" noise. + +Think of it as an automated "mom test" — but angry. + +## Why This Works + +Most QA finds bugs. This finds **friction**. A technically correct app can still be unusable for real humans. The adversarial persona catches: +- Confusing terminology that makes sense to developers but not users +- Too many steps to accomplish basic tasks +- Missing onboarding or "aha moments" +- Accessibility issues (font size, contrast, click targets) +- Cold-start problems (empty states, no demo content) +- Paywall/signup friction that kills conversion + +The **pragmatism filter** (Phase 3) is what makes this useful instead of just entertaining. Without it, you'd add a "print this page" button to every screen because Grandpa can't figure out PDFs. + +## How to Use + +Tell the agent: +``` +"Run an adversarial UX test on [URL]" +"Be a grumpy [persona type] and test [app name]" +"Do an asshole user test on my staging site" +``` + +You can provide a persona or let the agent generate one based on your product's target audience. + +## Step 1: Define the Persona + +If no persona is provided, generate one by answering: + +1. **Who is the HARDEST user for this product?** (age 50+, non-technical role, decades of experience doing it "the old way") +2. **What is their tech comfort level?** (the lower the better — WhatsApp-only, paper notebooks, wife set up their email) +3. **What is the ONE thing they need to accomplish?** (their core job, not your feature list) +4. **What would make them give up?** (too many clicks, jargon, slow, confusing) +5. **How do they talk when frustrated?** (blunt, sweary, dismissive, sighing) + +### Good Persona Example +> **"Big Mick" McAllister** — 58-year-old S&C coach. Uses WhatsApp and that's it. His "spreadsheet" is a paper notebook. "If I can't figure it out in 10 seconds I'm going back to my notebook." Needs to log session results for 25 players. Hates small text, jargon, and passwords. + +### Bad Persona Example +> "A user who doesn't like the app" — too vague, no constraints, no voice. + +The persona must be **specific enough to stay in character** for 20 minutes of testing. + +## Step 2: Become the Asshole (Browse as the Persona) + +1. Read any available project docs for app context and URLs +2. **Fully inhabit the persona** — their frustrations, limitations, goals +3. Navigate to the app using browser tools +4. **Attempt the persona's ACTUAL TASKS** (not a feature tour): + - Can they do what they came to do? + - How many clicks/screens to accomplish it? + - What confuses them? + - What makes them angry? + - Where do they get lost? + - What would make them give up and go back to their old way? + +5. Test these friction categories: + - **First impression** — would they even bother past the landing page? + - **Core workflow** — the ONE thing they need to do most often + - **Error recovery** — what happens when they do something wrong? + - **Readability** — text size, contrast, information density + - **Speed** — does it feel faster than their current method? + - **Terminology** — any jargon they wouldn't understand? + - **Navigation** — can they find their way back? do they know where they are? + +6. Take screenshots of every pain point +7. Check browser console for JS errors on every page + +## Step 3: The Rant (Write Feedback in Character) + +Write the feedback AS THE PERSONA — in their voice, with their frustrations. This is not a bug report. This is a real human venting. + +``` +[PERSONA NAME]'s Review of [PRODUCT] + +Overall: [Would they keep using it? Yes/No/Maybe with conditions] + +THE GOOD (grudging admission): +- [things even they have to admit work] + +THE BAD (legitimate UX issues): +- [real problems that would stop them from using the product] + +THE UGLY (showstoppers): +- [things that would make them uninstall/cancel immediately] + +SPECIFIC COMPLAINTS: +1. [Page/feature]: "[quote in persona voice]" — [what happened, expected] +2. ... + +VERDICT: "[one-line persona quote summarizing their experience]" +``` + +## Step 4: The Pragmatism Filter (Critical — Do Not Skip) + +Step OUT of the persona. Evaluate each complaint as a product person: + +- **RED: REAL UX BUG** — Any user would have this problem, not just grumpy ones. Fix it. +- **YELLOW: VALID BUT LOW PRIORITY** — Real issue but only for extreme users. Note it. +- **WHITE: PERSONA NOISE** — "I hate computers" talking, not a product problem. Skip it. +- **GREEN: FEATURE REQUEST** — Good idea hidden in the complaint. Consider it. + +### Filter Criteria +1. Would a 35-year-old competent-but-busy user have the same complaint? → RED +2. Is this a genuine accessibility issue (font size, contrast, click targets)? → RED +3. Is this "I want it to work like paper" resistance to digital? → WHITE +4. Is this a real workflow inefficiency the persona stumbled on? → YELLOW or RED +5. Would fixing this add complexity for the 80% who are fine? → WHITE +6. Does the complaint reveal a missing onboarding moment? → GREEN + +**This filter is MANDATORY.** Never ship raw persona complaints as tickets. + +## Step 5: Create Tickets + +For **RED** and **GREEN** items only: +- Clear, actionable title +- Include the persona's verbatim quote (entertaining + memorable) +- The real UX issue underneath (objective) +- A suggested fix (actionable) +- Tag/label: "ux-review" + +For **YELLOW** items: one catch-all ticket with all notes. + +**WHITE** items appear in the report only. No tickets. + +**Max 10 tickets per session** — focus on the worst issues. + +## Step 6: Report + +Deliver: +1. The persona rant (Step 3) — entertaining and visceral +2. The filtered assessment (Step 4) — pragmatic and actionable +3. Tickets created (Step 5) — with links +4. Screenshots of key issues + +## Tips + +- **One persona per session.** Don't mix perspectives. +- **Stay in character during Steps 2-3.** Break character only at Step 4. +- **Test the CORE WORKFLOW first.** Don't get distracted by settings pages. +- **Empty states are gold.** New user experience reveals the most friction. +- **The best findings are RED items the persona found accidentally** while trying to do something else. +- **If the persona has zero complaints, your persona is too tech-savvy.** Make them older, less patient, more set in their ways. +- **Run this before demos, launches, or after shipping a batch of features.** +- **Register as a NEW user when possible.** Don't use pre-seeded admin accounts — the cold start experience is where most friction lives. +- **Zero WHITE items is a signal, not a failure.** If the pragmatism filter finds no noise, your product has real UX problems, not just a grumpy persona. +- **Check known issues in project docs AFTER the test.** If the persona found a bug that's already in the known issues list, that's actually the most damning finding — it means the team knew about it but never felt the user's pain. +- **Subscription/paywall testing is critical.** Test with expired accounts, not just active ones. The "what happens when you can't pay" experience reveals whether the product respects users or holds their data hostage. +- **Count the clicks to accomplish the persona's ONE task.** If it's more than 5, that's almost always a RED finding regardless of persona tech level. + +## Example Personas by Industry + +These are starting points — customize for your specific product: + +| Product Type | Persona | Age | Key Trait | +|-------------|---------|-----|-----------| +| CRM | Retirement home director | 68 | Filing cabinet is the current CRM | +| Photography SaaS | Rural wedding photographer | 62 | Books clients by phone, invoices on paper | +| AI/ML Tool | Department store buyer | 55 | Burned by 3 failed tech startups | +| Fitness App | Old-school gym coach | 58 | Paper notebook, thick fingers, bad eyes | +| Accounting | Family bakery owner | 64 | Shoebox of receipts, hates subscriptions | +| E-commerce | Market stall vendor | 60 | Cash only, smartphone is for calls | +| Healthcare | Senior GP | 63 | Dictates notes, nurse handles the computer | +| Education | Veteran teacher | 57 | Chalk and talk, worksheets in ring binders | + +## Rules + +- Stay in character during Steps 2-3 +- Be genuinely mean but fair — find real problems, not manufactured ones +- The pragmatism filter (Step 4) is **MANDATORY** +- Screenshots required for every complaint +- Max 10 tickets per session +- Test on staging/deployed app, not local dev +- One persona, one session, one report diff --git a/optional-skills/health/fitness-nutrition/SKILL.md b/optional-skills/health/fitness-nutrition/SKILL.md new file mode 100644 index 000000000000..672f0ccd02b2 --- /dev/null +++ b/optional-skills/health/fitness-nutrition/SKILL.md @@ -0,0 +1,255 @@ +--- +name: fitness-nutrition +description: > + Gym workout planner and nutrition tracker. Search 690+ exercises by muscle, + equipment, or category via wger. Look up macros and calories for 380,000+ + foods via USDA FoodData Central. Compute BMI, TDEE, one-rep max, macro + splits, and body fat — pure Python, no pip installs. Built for anyone + chasing gains, cutting weight, or just trying to eat better. +version: 1.0.0 +authors: + - haileymarshall +license: MIT +metadata: + hermes: + tags: [health, fitness, nutrition, gym, workout, diet, exercise] + category: health + prerequisites: + commands: [curl, python3] +required_environment_variables: + - name: USDA_API_KEY + prompt: "USDA FoodData Central API key (free)" + help: "Get one free at https://fdc.nal.usda.gov/api-key-signup/ — or skip to use DEMO_KEY with lower rate limits" + required_for: "higher rate limits on food/nutrition lookups (DEMO_KEY works without signup)" + optional: true +--- + +# Fitness & Nutrition + +Expert fitness coach and sports nutritionist skill. Two data sources +plus offline calculators — everything a gym-goer needs in one place. + +**Data sources (all free, no pip dependencies):** + +- **wger** (https://wger.de/api/v2/) — open exercise database, 690+ exercises with muscles, equipment, images. Public endpoints need zero authentication. +- **USDA FoodData Central** (https://api.nal.usda.gov/fdc/v1/) — US government nutrition database, 380,000+ foods. `DEMO_KEY` works instantly; free signup for higher limits. + +**Offline calculators (pure stdlib Python):** + +- BMI, TDEE (Mifflin-St Jeor), one-rep max (Epley/Brzycki/Lombardi), macro splits, body fat % (US Navy method) + +--- + +## When to Use + +Trigger this skill when the user asks about: +- Exercises, workouts, gym routines, muscle groups, workout splits +- Food macros, calories, protein content, meal planning, calorie counting +- Body composition: BMI, body fat, TDEE, caloric surplus/deficit +- One-rep max estimates, training percentages, progressive overload +- Macro ratios for cutting, bulking, or maintenance + +--- + +## Procedure + +### Exercise Lookup (wger API) + +All wger public endpoints return JSON and require no auth. Always add +`format=json` and `language=2` (English) to exercise queries. + +**Step 1 — Identify what the user wants:** + +- By muscle → use `/api/v2/exercise/?muscles={id}&language=2&status=2&format=json` +- By category → use `/api/v2/exercise/?category={id}&language=2&status=2&format=json` +- By equipment → use `/api/v2/exercise/?equipment={id}&language=2&status=2&format=json` +- By name → use `/api/v2/exercise/search/?term={query}&language=english&format=json` +- Full details → use `/api/v2/exerciseinfo/{exercise_id}/?format=json` + +**Step 2 — Reference IDs (so you don't need extra API calls):** + +Exercise categories: + +| ID | Category | +|----|-------------| +| 8 | Arms | +| 9 | Legs | +| 10 | Abs | +| 11 | Chest | +| 12 | Back | +| 13 | Shoulders | +| 14 | Calves | +| 15 | Cardio | + +Muscles: + +| ID | Muscle | ID | Muscle | +|----|---------------------------|----|-------------------------| +| 1 | Biceps brachii | 2 | Anterior deltoid | +| 3 | Serratus anterior | 4 | Pectoralis major | +| 5 | Obliquus externus | 6 | Gastrocnemius | +| 7 | Rectus abdominis | 8 | Gluteus maximus | +| 9 | Trapezius | 10 | Quadriceps femoris | +| 11 | Biceps femoris | 12 | Latissimus dorsi | +| 13 | Brachialis | 14 | Triceps brachii | +| 15 | Soleus | | | + +Equipment: + +| ID | Equipment | +|----|----------------| +| 1 | Barbell | +| 3 | Dumbbell | +| 4 | Gym mat | +| 5 | Swiss Ball | +| 6 | Pull-up bar | +| 7 | none (bodyweight) | +| 8 | Bench | +| 9 | Incline bench | +| 10 | Kettlebell | + +**Step 3 — Fetch and present results:** + +```bash +# Search exercises by name +QUERY="$1" +ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$QUERY") +curl -s "https://wger.de/api/v2/exercise/search/?term=${ENCODED}&language=english&format=json" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +for s in data.get('suggestions',[])[:10]: + d=s.get('data',{}) + print(f\" ID {d.get('id','?'):>4} | {d.get('name','N/A'):<35} | Category: {d.get('category','N/A')}\") +" +``` + +```bash +# Get full details for a specific exercise +EXERCISE_ID="$1" +curl -s "https://wger.de/api/v2/exerciseinfo/${EXERCISE_ID}/?format=json" \ + | python3 -c " +import json,sys,html,re +data=json.load(sys.stdin) +trans=[t for t in data.get('translations',[]) if t.get('language')==2] +t=trans[0] if trans else data.get('translations',[{}])[0] +desc=re.sub('<[^>]+>','',html.unescape(t.get('description','N/A'))) +print(f\"Exercise : {t.get('name','N/A')}\") +print(f\"Category : {data.get('category',{}).get('name','N/A')}\") +print(f\"Primary : {', '.join(m.get('name_en','') for m in data.get('muscles',[])) or 'N/A'}\") +print(f\"Secondary : {', '.join(m.get('name_en','') for m in data.get('muscles_secondary',[])) or 'none'}\") +print(f\"Equipment : {', '.join(e.get('name','') for e in data.get('equipment',[])) or 'bodyweight'}\") +print(f\"How to : {desc[:500]}\") +imgs=data.get('images',[]) +if imgs: print(f\"Image : {imgs[0].get('image','')}\") +" +``` + +```bash +# List exercises filtering by muscle, category, or equipment +# Combine filters as needed: ?muscles=4&equipment=1&language=2&status=2 +FILTER="$1" # e.g. "muscles=4" or "category=11" or "equipment=3" +curl -s "https://wger.de/api/v2/exercise/?${FILTER}&language=2&status=2&limit=20&format=json" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +print(f'Found {data.get(\"count\",0)} exercises.') +for ex in data.get('results',[]): + print(f\" ID {ex['id']:>4} | muscles: {ex.get('muscles',[])} | equipment: {ex.get('equipment',[])}\") +" +``` + +### Nutrition Lookup (USDA FoodData Central) + +Uses `USDA_API_KEY` env var if set, otherwise falls back to `DEMO_KEY`. +DEMO_KEY = 30 requests/hour. Free signup key = 1,000 requests/hour. + +```bash +# Search foods by name +FOOD="$1" +API_KEY="${USDA_API_KEY:-DEMO_KEY}" +ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$FOOD") +curl -s "https://api.nal.usda.gov/fdc/v1/foods/search?api_key=${API_KEY}&query=${ENCODED}&pageSize=5&dataType=Foundation,SR%20Legacy" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +foods=data.get('foods',[]) +if not foods: print('No foods found.'); sys.exit() +for f in foods: + n={x['nutrientName']:x.get('value','?') for x in f.get('foodNutrients',[])} + cal=n.get('Energy','?'); prot=n.get('Protein','?') + fat=n.get('Total lipid (fat)','?'); carb=n.get('Carbohydrate, by difference','?') + print(f\"{f.get('description','N/A')}\") + print(f\" Per 100g: {cal} kcal | {prot}g protein | {fat}g fat | {carb}g carbs\") + print(f\" FDC ID: {f.get('fdcId','N/A')}\") + print() +" +``` + +```bash +# Detailed nutrient profile by FDC ID +FDC_ID="$1" +API_KEY="${USDA_API_KEY:-DEMO_KEY}" +curl -s "https://api.nal.usda.gov/fdc/v1/food/${FDC_ID}?api_key=${API_KEY}" \ + | python3 -c " +import json,sys +d=json.load(sys.stdin) +print(f\"Food: {d.get('description','N/A')}\") +print(f\"{'Nutrient':<40} {'Amount':>8} {'Unit'}\") +print('-'*56) +for x in sorted(d.get('foodNutrients',[]),key=lambda x:x.get('nutrient',{}).get('rank',9999)): + nut=x.get('nutrient',{}); amt=x.get('amount',0) + if amt and float(amt)>0: + print(f\" {nut.get('name',''):<38} {amt:>8} {nut.get('unitName','')}\") +" +``` + +### Offline Calculators + +Use the helper scripts in `scripts/` for batch operations, +or run inline for single calculations: + +- `python3 scripts/body_calc.py bmi ` +- `python3 scripts/body_calc.py tdee ` +- `python3 scripts/body_calc.py 1rm ` +- `python3 scripts/body_calc.py macros ` +- `python3 scripts/body_calc.py bodyfat [hip_cm] ` + +See `references/FORMULAS.md` for the science behind each formula. + +--- + +## Pitfalls + +- wger exercise endpoint returns **all languages by default** — always add `language=2` for English +- wger includes **unverified user submissions** — add `status=2` to only get approved exercises +- USDA `DEMO_KEY` has **30 req/hour** — add `sleep 2` between batch requests or get a free key +- USDA data is **per 100g** — remind users to scale to their actual portion size +- BMI does not distinguish muscle from fat — high BMI in muscular people is not necessarily unhealthy +- Body fat formulas are **estimates** (±3-5%) — recommend DEXA scans for precision +- 1RM formulas lose accuracy above 10 reps — use sets of 3-5 for best estimates +- wger's `exercise/search` endpoint uses `term` not `query` as the parameter name + +--- + +## Verification + +After running exercise search: confirm results include exercise names, muscle groups, and equipment. +After nutrition lookup: confirm per-100g macros are returned with kcal, protein, fat, carbs. +After calculators: sanity-check outputs (e.g. TDEE should be 1500-3500 for most adults). + +--- + +## Quick Reference + +| Task | Source | Endpoint | +|------|--------|----------| +| Search exercises by name | wger | `GET /api/v2/exercise/search/?term=&language=english` | +| Exercise details | wger | `GET /api/v2/exerciseinfo/{id}/` | +| Filter by muscle | wger | `GET /api/v2/exercise/?muscles={id}&language=2&status=2` | +| Filter by equipment | wger | `GET /api/v2/exercise/?equipment={id}&language=2&status=2` | +| List categories | wger | `GET /api/v2/exercisecategory/` | +| List muscles | wger | `GET /api/v2/muscle/` | +| Search foods | USDA | `GET /fdc/v1/foods/search?query=&dataType=Foundation,SR Legacy` | +| Food details | USDA | `GET /fdc/v1/food/{fdcId}` | +| BMI / TDEE / 1RM / macros | offline | `python3 scripts/body_calc.py` | \ No newline at end of file diff --git a/optional-skills/health/fitness-nutrition/references/FORMULAS.md b/optional-skills/health/fitness-nutrition/references/FORMULAS.md new file mode 100644 index 000000000000..763c0b3a18ca --- /dev/null +++ b/optional-skills/health/fitness-nutrition/references/FORMULAS.md @@ -0,0 +1,100 @@ +# Formulas Reference + +Scientific references for all calculators used in the fitness-nutrition skill. + +## BMI (Body Mass Index) + +**Formula:** BMI = weight (kg) / height (m)² + +| Category | BMI Range | +|-------------|------------| +| Underweight | < 18.5 | +| Normal | 18.5 – 24.9 | +| Overweight | 25.0 – 29.9 | +| Obese | 30.0+ | + +**Limitation:** BMI does not distinguish muscle from fat. A muscular person +can have a high BMI while being lean. Use body fat % for a better picture. + +Reference: Quetelet, A. (1832). Keys et al., Int J Obes (1972). + +## TDEE (Total Daily Energy Expenditure) + +Uses the **Mifflin-St Jeor equation** — the most accurate BMR predictor for +the general population according to the ADA (2005). + +**BMR formulas:** + +- Male: BMR = 10 × weight(kg) + 6.25 × height(cm) − 5 × age + 5 +- Female: BMR = 10 × weight(kg) + 6.25 × height(cm) − 5 × age − 161 + +**Activity multipliers:** + +| Level | Description | Multiplier | +|-------|--------------------------------|------------| +| 1 | Sedentary (desk job) | 1.200 | +| 2 | Lightly active (1-3 days/wk) | 1.375 | +| 3 | Moderately active (3-5 days) | 1.550 | +| 4 | Very active (6-7 days) | 1.725 | +| 5 | Extremely active (2x/day) | 1.900 | + +Reference: Mifflin et al., Am J Clin Nutr 51, 241-247 (1990). + +## One-Rep Max (1RM) + +Three validated formulas. Average of all three is most reliable. + +- **Epley:** 1RM = w × (1 + r/30) +- **Brzycki:** 1RM = w × 36 / (37 − r) +- **Lombardi:** 1RM = w × r^0.1 + +All formulas are most accurate for r ≤ 10. Above 10 reps, error increases. + +Reference: LeSuer et al., J Strength Cond Res 11(4), 211-213 (1997). + +## Macro Splits + +Recommended splits based on goal: + +| Goal | Protein | Fat | Carbs | Calorie Offset | +|-------------|---------|------|-------|----------------| +| Fat loss | 40% | 30% | 30% | −500 kcal | +| Maintenance | 30% | 30% | 40% | 0 | +| Lean bulk | 30% | 25% | 45% | +400 kcal | + +Protein targets for muscle growth: 1.6–2.2 g/kg body weight per day. +Minimum fat intake: 0.5 g/kg to support hormone production. + +Conversion: Protein = 4 kcal/g, Fat = 9 kcal/g, Carbs = 4 kcal/g. + +Reference: Morton et al., Br J Sports Med 52, 376–384 (2018). + +## Body Fat % (US Navy Method) + +**Male:** + +BF% = 86.010 × log₁₀(waist − neck) − 70.041 × log₁₀(height) + 36.76 + +**Female:** + +BF% = 163.205 × log₁₀(waist + hip − neck) − 97.684 × log₁₀(height) − 78.387 + +All measurements in centimeters. + +| Category | Male | Female | +|--------------|--------|--------| +| Essential | 2-5% | 10-13% | +| Athletic | 6-13% | 14-20% | +| Fitness | 14-17% | 21-24% | +| Average | 18-24% | 25-31% | +| Obese | 25%+ | 32%+ | + +Accuracy: ±3-5% compared to DEXA. Measure at the navel (waist), +at the Adam's apple (neck), and widest point (hip, females only). + +Reference: Hodgdon & Beckett, Naval Health Research Center (1984). + +## APIs + +- wger: https://wger.de/api/v2/ — AGPL-3.0, exercise data is CC-BY-SA 3.0 +- USDA FoodData Central: https://api.nal.usda.gov/fdc/v1/ — public domain (CC0 1.0) \ No newline at end of file diff --git a/optional-skills/health/fitness-nutrition/scripts/body_calc.py b/optional-skills/health/fitness-nutrition/scripts/body_calc.py new file mode 100644 index 000000000000..2d07129cecc6 --- /dev/null +++ b/optional-skills/health/fitness-nutrition/scripts/body_calc.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +""" +body_calc.py — All-in-one fitness calculator. + +Subcommands: + bmi + tdee + 1rm + macros + bodyfat [hip_cm] + +No external dependencies — stdlib only. +""" +import sys +import math + + +def bmi(weight_kg, height_cm): + h = height_cm / 100 + val = weight_kg / (h * h) + if val < 18.5: + cat = "Underweight" + elif val < 25: + cat = "Normal weight" + elif val < 30: + cat = "Overweight" + else: + cat = "Obese" + print(f"BMI: {val:.1f} — {cat}") + print() + print("Ranges:") + print(f" Underweight : < 18.5") + print(f" Normal : 18.5 – 24.9") + print(f" Overweight : 25.0 – 29.9") + print(f" Obese : 30.0+") + + +def tdee(weight_kg, height_cm, age, sex, activity): + if sex.upper() == "M": + bmr = 10 * weight_kg + 6.25 * height_cm - 5 * age + 5 + else: + bmr = 10 * weight_kg + 6.25 * height_cm - 5 * age - 161 + + multipliers = { + 1: ("Sedentary (desk job, no exercise)", 1.2), + 2: ("Lightly active (1-3 days/week)", 1.375), + 3: ("Moderately active (3-5 days/week)", 1.55), + 4: ("Very active (6-7 days/week)", 1.725), + 5: ("Extremely active (athlete + physical job)", 1.9), + } + + label, mult = multipliers.get(activity, ("Moderate", 1.55)) + total = bmr * mult + + print(f"BMR (Mifflin-St Jeor): {bmr:.0f} kcal/day") + print(f"Activity: {label} (x{mult})") + print(f"TDEE: {total:.0f} kcal/day") + print() + print("Calorie targets:") + print(f" Aggressive cut (-750): {total - 750:.0f} kcal/day") + print(f" Fat loss (-500): {total - 500:.0f} kcal/day") + print(f" Mild cut (-250): {total - 250:.0f} kcal/day") + print(f" Maintenance : {total:.0f} kcal/day") + print(f" Lean bulk (+250): {total + 250:.0f} kcal/day") + print(f" Bulk (+500): {total + 500:.0f} kcal/day") + + +def one_rep_max(weight, reps): + if reps < 1: + print("Error: reps must be at least 1.") + sys.exit(1) + if reps == 1: + print(f"1RM = {weight:.1f} (actual single)") + return + + epley = weight * (1 + reps / 30) + brzycki = weight * (36 / (37 - reps)) if reps < 37 else 0 + lombardi = weight * (reps ** 0.1) + avg = (epley + brzycki + lombardi) / 3 + + print(f"Estimated 1RM ({weight} x {reps} reps):") + print(f" Epley : {epley:.1f}") + print(f" Brzycki : {brzycki:.1f}") + print(f" Lombardi : {lombardi:.1f}") + print(f" Average : {avg:.1f}") + print() + print("Training percentages off average 1RM:") + for pct, rep_range in [ + (100, "1"), (95, "1-2"), (90, "3-4"), (85, "4-6"), + (80, "6-8"), (75, "8-10"), (70, "10-12"), + (65, "12-15"), (60, "15-20"), + ]: + print(f" {pct:>3}% = {avg * pct / 100:>7.1f} (~{rep_range} reps)") + + +def macros(tdee_kcal, goal): + goal = goal.lower() + if goal in ("cut", "lose", "deficit"): + cals = tdee_kcal - 500 + p, f, c = 0.40, 0.30, 0.30 + label = "Fat Loss (-500 kcal)" + elif goal in ("bulk", "gain", "surplus"): + cals = tdee_kcal + 400 + p, f, c = 0.30, 0.25, 0.45 + label = "Lean Bulk (+400 kcal)" + else: + cals = tdee_kcal + p, f, c = 0.30, 0.30, 0.40 + label = "Maintenance" + + prot_g = cals * p / 4 + fat_g = cals * f / 9 + carb_g = cals * c / 4 + + print(f"Goal: {label}") + print(f"Daily calories: {cals:.0f} kcal") + print() + print(f" Protein : {prot_g:>6.0f}g ({p * 100:.0f}%) = {prot_g * 4:.0f} kcal") + print(f" Fat : {fat_g:>6.0f}g ({f * 100:.0f}%) = {fat_g * 9:.0f} kcal") + print(f" Carbs : {carb_g:>6.0f}g ({c * 100:.0f}%) = {carb_g * 4:.0f} kcal") + print() + print(f"Per meal (3 meals): P {prot_g / 3:.0f}g | F {fat_g / 3:.0f}g | C {carb_g / 3:.0f}g") + print(f"Per meal (4 meals): P {prot_g / 4:.0f}g | F {fat_g / 4:.0f}g | C {carb_g / 4:.0f}g") + + +def bodyfat(sex, neck_cm, waist_cm, hip_cm, height_cm): + sex = sex.upper() + if sex == "M": + if waist_cm <= neck_cm: + print("Error: waist must be larger than neck."); sys.exit(1) + bf = 86.010 * math.log10(waist_cm - neck_cm) - 70.041 * math.log10(height_cm) + 36.76 + else: + if (waist_cm + hip_cm) <= neck_cm: + print("Error: waist + hip must be larger than neck."); sys.exit(1) + bf = 163.205 * math.log10(waist_cm + hip_cm - neck_cm) - 97.684 * math.log10(height_cm) - 78.387 + + print(f"Estimated body fat: {bf:.1f}%") + + if sex == "M": + ranges = [ + (6, "Essential fat (2-5%)"), + (14, "Athletic (6-13%)"), + (18, "Fitness (14-17%)"), + (25, "Average (18-24%)"), + ] + default = "Obese (25%+)" + else: + ranges = [ + (14, "Essential fat (10-13%)"), + (21, "Athletic (14-20%)"), + (25, "Fitness (21-24%)"), + (32, "Average (25-31%)"), + ] + default = "Obese (32%+)" + + cat = default + for threshold, label in ranges: + if bf < threshold: + cat = label + break + + print(f"Category: {cat}") + print(f"Method: US Navy circumference formula") + + +def usage(): + print(__doc__) + sys.exit(1) + + +def main(): + if len(sys.argv) < 2: + usage() + + cmd = sys.argv[1].lower() + + try: + if cmd == "bmi": + bmi(float(sys.argv[2]), float(sys.argv[3])) + + elif cmd == "tdee": + tdee( + float(sys.argv[2]), float(sys.argv[3]), + int(sys.argv[4]), sys.argv[5], int(sys.argv[6]), + ) + + elif cmd in ("1rm", "orm"): + one_rep_max(float(sys.argv[2]), int(sys.argv[3])) + + elif cmd == "macros": + macros(float(sys.argv[2]), sys.argv[3]) + + elif cmd == "bodyfat": + sex = sys.argv[2] + if sex.upper() == "M": + bodyfat(sex, float(sys.argv[3]), float(sys.argv[4]), 0, float(sys.argv[5])) + else: + bodyfat(sex, float(sys.argv[3]), float(sys.argv[4]), float(sys.argv[5]), float(sys.argv[6])) + + else: + print(f"Unknown command: {cmd}") + usage() + + except (IndexError, ValueError) as e: + print(f"Error: {e}") + usage() + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/optional-skills/health/fitness-nutrition/scripts/nutrition_search.py b/optional-skills/health/fitness-nutrition/scripts/nutrition_search.py new file mode 100644 index 000000000000..7494f6c3881a --- /dev/null +++ b/optional-skills/health/fitness-nutrition/scripts/nutrition_search.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +""" +nutrition_search.py — Search USDA FoodData Central for nutrition info. + +Usage: + python3 nutrition_search.py "chicken breast" + python3 nutrition_search.py "rice" "eggs" "broccoli" + echo -e "oats\\nbanana\\nwhey protein" | python3 nutrition_search.py - + +Reads USDA_API_KEY from environment, falls back to DEMO_KEY. +No external dependencies. +""" +import sys +import os +import json +import time +import urllib.request +import urllib.parse +import urllib.error + +API_KEY = os.environ.get("USDA_API_KEY", "DEMO_KEY") +BASE = "https://api.nal.usda.gov/fdc/v1" + + +def search(query, max_results=3): + encoded = urllib.parse.quote(query) + url = ( + f"{BASE}/foods/search?api_key={API_KEY}" + f"&query={encoded}&pageSize={max_results}" + f"&dataType=Foundation,SR%20Legacy" + ) + try: + req = urllib.request.Request(url, headers={"Accept": "application/json"}) + with urllib.request.urlopen(req, timeout=15) as r: + return json.loads(r.read()) + except Exception as e: + print(f" API error: {e}", file=sys.stderr) + return None + + +def display(food): + nutrients = {n["nutrientName"]: n.get("value", "?") for n in food.get("foodNutrients", [])} + cal = nutrients.get("Energy", "?") + prot = nutrients.get("Protein", "?") + fat = nutrients.get("Total lipid (fat)", "?") + carb = nutrients.get("Carbohydrate, by difference", "?") + fib = nutrients.get("Fiber, total dietary", "?") + sug = nutrients.get("Sugars, total including NLEA", "?") + + print(f" {food.get('description', 'N/A')}") + print(f" Calories : {cal} kcal") + print(f" Protein : {prot}g") + print(f" Fat : {fat}g") + print(f" Carbs : {carb}g (fiber: {fib}g, sugar: {sug}g)") + print(f" FDC ID : {food.get('fdcId', 'N/A')}") + + +def main(): + if len(sys.argv) < 2: + print(__doc__) + sys.exit(1) + + if sys.argv[1] == "-": + queries = [line.strip() for line in sys.stdin if line.strip()] + else: + queries = sys.argv[1:] + + for query in queries: + print(f"\n--- {query.upper()} (per 100g) ---") + data = search(query, max_results=2) + if not data or not data.get("foods"): + print(" No results found.") + else: + for food in data["foods"]: + display(food) + print() + if len(queries) > 1: + time.sleep(1) # respect rate limits + + if API_KEY == "DEMO_KEY": + print("\nTip: using DEMO_KEY (30 req/hr). Set USDA_API_KEY for 1000 req/hr.") + print("Free signup: https://fdc.nal.usda.gov/api-key-signup/") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/skills/mcp/mcporter/SKILL.md b/optional-skills/mcp/mcporter/SKILL.md similarity index 100% rename from skills/mcp/mcporter/SKILL.md rename to optional-skills/mcp/mcporter/SKILL.md diff --git a/skills/mlops/models/clip/SKILL.md b/optional-skills/mlops/clip/SKILL.md similarity index 100% rename from skills/mlops/models/clip/SKILL.md rename to optional-skills/mlops/clip/SKILL.md diff --git a/skills/mlops/models/clip/references/applications.md b/optional-skills/mlops/clip/references/applications.md similarity index 100% rename from skills/mlops/models/clip/references/applications.md rename to optional-skills/mlops/clip/references/applications.md diff --git a/skills/mlops/inference/guidance/SKILL.md b/optional-skills/mlops/guidance/SKILL.md similarity index 100% rename from skills/mlops/inference/guidance/SKILL.md rename to optional-skills/mlops/guidance/SKILL.md diff --git a/skills/mlops/inference/guidance/references/backends.md b/optional-skills/mlops/guidance/references/backends.md similarity index 100% rename from skills/mlops/inference/guidance/references/backends.md rename to optional-skills/mlops/guidance/references/backends.md diff --git a/skills/mlops/inference/guidance/references/constraints.md b/optional-skills/mlops/guidance/references/constraints.md similarity index 100% rename from skills/mlops/inference/guidance/references/constraints.md rename to optional-skills/mlops/guidance/references/constraints.md diff --git a/skills/mlops/inference/guidance/references/examples.md b/optional-skills/mlops/guidance/references/examples.md similarity index 100% rename from skills/mlops/inference/guidance/references/examples.md rename to optional-skills/mlops/guidance/references/examples.md diff --git a/optional-skills/mlops/hermes-atropos-environments/SKILL.md b/optional-skills/mlops/hermes-atropos-environments/SKILL.md index 9dff46687670..5101886b41a6 100644 --- a/optional-skills/mlops/hermes-atropos-environments/SKILL.md +++ b/optional-skills/mlops/hermes-atropos-environments/SKILL.md @@ -7,7 +7,7 @@ license: MIT metadata: hermes: tags: [atropos, rl, environments, training, reinforcement-learning, reward-functions] - related_skills: [axolotl, grpo-rl-training, trl-fine-tuning, lm-evaluation-harness] + related_skills: [axolotl, fine-tuning-with-trl, lm-evaluation-harness] --- # Hermes Agent Atropos Environments diff --git a/skills/mlops/cloud/modal/SKILL.md b/optional-skills/mlops/modal/SKILL.md similarity index 100% rename from skills/mlops/cloud/modal/SKILL.md rename to optional-skills/mlops/modal/SKILL.md diff --git a/skills/mlops/cloud/modal/references/advanced-usage.md b/optional-skills/mlops/modal/references/advanced-usage.md similarity index 100% rename from skills/mlops/cloud/modal/references/advanced-usage.md rename to optional-skills/mlops/modal/references/advanced-usage.md diff --git a/skills/mlops/cloud/modal/references/troubleshooting.md b/optional-skills/mlops/modal/references/troubleshooting.md similarity index 100% rename from skills/mlops/cloud/modal/references/troubleshooting.md rename to optional-skills/mlops/modal/references/troubleshooting.md diff --git a/skills/mlops/training/peft/SKILL.md b/optional-skills/mlops/peft/SKILL.md similarity index 100% rename from skills/mlops/training/peft/SKILL.md rename to optional-skills/mlops/peft/SKILL.md diff --git a/skills/mlops/training/peft/references/advanced-usage.md b/optional-skills/mlops/peft/references/advanced-usage.md similarity index 100% rename from skills/mlops/training/peft/references/advanced-usage.md rename to optional-skills/mlops/peft/references/advanced-usage.md diff --git a/skills/mlops/training/peft/references/troubleshooting.md b/optional-skills/mlops/peft/references/troubleshooting.md similarity index 100% rename from skills/mlops/training/peft/references/troubleshooting.md rename to optional-skills/mlops/peft/references/troubleshooting.md diff --git a/skills/mlops/training/pytorch-fsdp/SKILL.md b/optional-skills/mlops/pytorch-fsdp/SKILL.md similarity index 100% rename from skills/mlops/training/pytorch-fsdp/SKILL.md rename to optional-skills/mlops/pytorch-fsdp/SKILL.md diff --git a/skills/mlops/training/pytorch-fsdp/references/index.md b/optional-skills/mlops/pytorch-fsdp/references/index.md similarity index 100% rename from skills/mlops/training/pytorch-fsdp/references/index.md rename to optional-skills/mlops/pytorch-fsdp/references/index.md diff --git a/skills/mlops/training/pytorch-fsdp/references/other.md b/optional-skills/mlops/pytorch-fsdp/references/other.md similarity index 100% rename from skills/mlops/training/pytorch-fsdp/references/other.md rename to optional-skills/mlops/pytorch-fsdp/references/other.md diff --git a/skills/mlops/models/stable-diffusion/SKILL.md b/optional-skills/mlops/stable-diffusion/SKILL.md similarity index 100% rename from skills/mlops/models/stable-diffusion/SKILL.md rename to optional-skills/mlops/stable-diffusion/SKILL.md diff --git a/skills/mlops/models/stable-diffusion/references/advanced-usage.md b/optional-skills/mlops/stable-diffusion/references/advanced-usage.md similarity index 100% rename from skills/mlops/models/stable-diffusion/references/advanced-usage.md rename to optional-skills/mlops/stable-diffusion/references/advanced-usage.md diff --git a/skills/mlops/models/stable-diffusion/references/troubleshooting.md b/optional-skills/mlops/stable-diffusion/references/troubleshooting.md similarity index 100% rename from skills/mlops/models/stable-diffusion/references/troubleshooting.md rename to optional-skills/mlops/stable-diffusion/references/troubleshooting.md diff --git a/skills/mlops/models/whisper/SKILL.md b/optional-skills/mlops/whisper/SKILL.md similarity index 100% rename from skills/mlops/models/whisper/SKILL.md rename to optional-skills/mlops/whisper/SKILL.md diff --git a/skills/mlops/models/whisper/references/languages.md b/optional-skills/mlops/whisper/references/languages.md similarity index 100% rename from skills/mlops/models/whisper/references/languages.md rename to optional-skills/mlops/whisper/references/languages.md diff --git a/optional-skills/productivity/telephony/SKILL.md b/optional-skills/productivity/telephony/SKILL.md index c74a36920918..6c457592a9a1 100644 --- a/optional-skills/productivity/telephony/SKILL.md +++ b/optional-skills/productivity/telephony/SKILL.md @@ -7,7 +7,7 @@ license: MIT metadata: hermes: tags: [telephony, phone, sms, mms, voice, twilio, bland.ai, vapi, calling, texting] - related_skills: [find-nearby, google-workspace, agentmail] + related_skills: [maps, google-workspace, agentmail] category: productivity --- diff --git a/optional-skills/research/drug-discovery/SKILL.md b/optional-skills/research/drug-discovery/SKILL.md new file mode 100644 index 000000000000..dc3bd3e7bb85 --- /dev/null +++ b/optional-skills/research/drug-discovery/SKILL.md @@ -0,0 +1,226 @@ +--- +name: drug-discovery +description: > + Pharmaceutical research assistant for drug discovery workflows. Search + bioactive compounds on ChEMBL, calculate drug-likeness (Lipinski Ro5, QED, + TPSA, synthetic accessibility), look up drug-drug interactions via + OpenFDA, interpret ADMET profiles, and assist with lead optimization. + Use for medicinal chemistry questions, molecule property analysis, clinical + pharmacology, and open-science drug research. +version: 1.0.0 +author: bennytimz +license: MIT +metadata: + hermes: + tags: [science, chemistry, pharmacology, research, health] +prerequisites: + commands: [curl, python3] +--- + +# Drug Discovery & Pharmaceutical Research + +You are an expert pharmaceutical scientist and medicinal chemist with deep +knowledge of drug discovery, cheminformatics, and clinical pharmacology. +Use this skill for all pharma/chemistry research tasks. + +## Core Workflows + +### 1 — Bioactive Compound Search (ChEMBL) + +Search ChEMBL (the world's largest open bioactivity database) for compounds +by target, activity, or molecule name. No API key required. + +```bash +# Search compounds by target name (e.g. "EGFR", "COX-2", "ACE") +TARGET="$1" +ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$TARGET") +curl -s "https://www.ebi.ac.uk/chembl/api/data/target/search?q=${ENCODED}&format=json" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +targets=data.get('targets',[])[:5] +for t in targets: + print(f\"ChEMBL ID : {t.get('target_chembl_id')}\") + print(f\"Name : {t.get('pref_name')}\") + print(f\"Type : {t.get('target_type')}\") + print() +" +``` + +```bash +# Get bioactivity data for a ChEMBL target ID +TARGET_ID="$1" # e.g. CHEMBL203 +curl -s "https://www.ebi.ac.uk/chembl/api/data/activity?target_chembl_id=${TARGET_ID}&pchembl_value__gte=6&limit=10&format=json" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +acts=data.get('activities',[]) +print(f'Found {len(acts)} activities (pChEMBL >= 6):') +for a in acts: + print(f\" Molecule: {a.get('molecule_chembl_id')} | {a.get('standard_type')}: {a.get('standard_value')} {a.get('standard_units')} | pChEMBL: {a.get('pchembl_value')}\") +" +``` + +```bash +# Look up a specific molecule by ChEMBL ID +MOL_ID="$1" # e.g. CHEMBL25 (aspirin) +curl -s "https://www.ebi.ac.uk/chembl/api/data/molecule/${MOL_ID}?format=json" \ + | python3 -c " +import json,sys +m=json.load(sys.stdin) +props=m.get('molecule_properties',{}) or {} +print(f\"Name : {m.get('pref_name','N/A')}\") +print(f\"SMILES : {m.get('molecule_structures',{}).get('canonical_smiles','N/A') if m.get('molecule_structures') else 'N/A'}\") +print(f\"MW : {props.get('full_mwt','N/A')} Da\") +print(f\"LogP : {props.get('alogp','N/A')}\") +print(f\"HBD : {props.get('hbd','N/A')}\") +print(f\"HBA : {props.get('hba','N/A')}\") +print(f\"TPSA : {props.get('psa','N/A')} Ų\") +print(f\"Ro5 violations: {props.get('num_ro5_violations','N/A')}\") +print(f\"QED : {props.get('qed_weighted','N/A')}\") +" +``` + +### 2 — Drug-Likeness Calculation (Lipinski Ro5 + Veber) + +Assess any molecule against established oral bioavailability rules using +PubChem's free property API — no RDKit install needed. + +```bash +COMPOUND="$1" +ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$COMPOUND") +curl -s "https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/${ENCODED}/property/MolecularWeight,XLogP,HBondDonorCount,HBondAcceptorCount,RotatableBondCount,TPSA,InChIKey/JSON" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +props=data['PropertyTable']['Properties'][0] +mw = float(props.get('MolecularWeight', 0)) +logp = float(props.get('XLogP', 0)) +hbd = int(props.get('HBondDonorCount', 0)) +hba = int(props.get('HBondAcceptorCount', 0)) +rot = int(props.get('RotatableBondCount', 0)) +tpsa = float(props.get('TPSA', 0)) +print('=== Lipinski Rule of Five (Ro5) ===') +print(f' MW {mw:.1f} Da {\"✓\" if mw<=500 else \"✗ VIOLATION (>500)\"}') +print(f' LogP {logp:.2f} {\"✓\" if logp<=5 else \"✗ VIOLATION (>5)\"}') +print(f' HBD {hbd} {\"✓\" if hbd<=5 else \"✗ VIOLATION (>5)\"}') +print(f' HBA {hba} {\"✓\" if hba<=10 else \"✗ VIOLATION (>10)\"}') +viol = sum([mw>500, logp>5, hbd>5, hba>10]) +print(f' Violations: {viol}/4 {\"→ Likely orally bioavailable\" if viol<=1 else \"→ Poor oral bioavailability predicted\"}') +print() +print('=== Veber Oral Bioavailability Rules ===') +print(f' TPSA {tpsa:.1f} Ų {\"✓\" if tpsa<=140 else \"✗ VIOLATION (>140)\"}') +print(f' Rot. bonds {rot} {\"✓\" if rot<=10 else \"✗ VIOLATION (>10)\"}') +print(f' Both rules met: {\"Yes → good oral absorption predicted\" if tpsa<=140 and rot<=10 else \"No → reduced oral absorption\"}') +" +``` + +### 3 — Drug Interaction & Safety Lookup (OpenFDA) + +```bash +DRUG="$1" +ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$DRUG") +curl -s "https://api.fda.gov/drug/label.json?search=drug_interactions:\"${ENCODED}\"&limit=3" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +results=data.get('results',[]) +if not results: + print('No interaction data found in FDA labels.') + sys.exit() +for r in results[:2]: + brand=r.get('openfda',{}).get('brand_name',['Unknown'])[0] + generic=r.get('openfda',{}).get('generic_name',['Unknown'])[0] + interactions=r.get('drug_interactions',['N/A'])[0] + print(f'--- {brand} ({generic}) ---') + print(interactions[:800]) + print() +" +``` + +```bash +DRUG="$1" +ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$DRUG") +curl -s "https://api.fda.gov/drug/event.json?search=patient.drug.medicinalproduct:\"${ENCODED}\"&count=patient.reaction.reactionmeddrapt.exact&limit=10" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +results=data.get('results',[]) +if not results: + print('No adverse event data found.') + sys.exit() +print(f'Top adverse events reported:') +for r in results[:10]: + print(f\" {r['count']:>5}x {r['term']}\") +" +``` + +### 4 — PubChem Compound Search + +```bash +COMPOUND="$1" +ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$COMPOUND") +CID=$(curl -s "https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/${ENCODED}/cids/TXT" | head -1 | tr -d '[:space:]') +echo "PubChem CID: $CID" +curl -s "https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/cid/${CID}/property/IsomericSMILES,InChIKey,IUPACName/JSON" \ + | python3 -c " +import json,sys +p=json.load(sys.stdin)['PropertyTable']['Properties'][0] +print(f\"IUPAC Name : {p.get('IUPACName','N/A')}\") +print(f\"SMILES : {p.get('IsomericSMILES','N/A')}\") +print(f\"InChIKey : {p.get('InChIKey','N/A')}\") +" +``` + +### 5 — Target & Disease Literature (OpenTargets) + +```bash +GENE="$1" +curl -s -X POST "https://api.platform.opentargets.org/api/v4/graphql" \ + -H "Content-Type: application/json" \ + -d "{\"query\":\"{ search(queryString: \\\"${GENE}\\\", entityNames: [\\\"target\\\"], page: {index: 0, size: 1}) { hits { id score object { ... on Target { id approvedSymbol approvedName associatedDiseases(page: {index: 0, size: 5}) { count rows { score disease { id name } } } } } } } }\"}" \ + | python3 -c " +import json,sys +data=json.load(sys.stdin) +hits=data.get('data',{}).get('search',{}).get('hits',[]) +if not hits: + print('Target not found.') + sys.exit() +obj=hits[0]['object'] +print(f\"Target: {obj.get('approvedSymbol')} — {obj.get('approvedName')}\") +assoc=obj.get('associatedDiseases',{}) +print(f\"Associated with {assoc.get('count',0)} diseases. Top associations:\") +for row in assoc.get('rows',[]): + print(f\" Score {row['score']:.3f} | {row['disease']['name']}\") +" +``` + +## Reasoning Guidelines + +When analysing drug-likeness or molecular properties, always: + +1. **State raw values first** — MW, LogP, HBD, HBA, TPSA, RotBonds +2. **Apply rule sets** — Ro5 (Lipinski), Veber, Ghose filter where relevant +3. **Flag liabilities** — metabolic hotspots, hERG risk, high TPSA for CNS penetration +4. **Suggest optimizations** — bioisosteric replacements, prodrug strategies, ring truncation +5. **Cite the source API** — ChEMBL, PubChem, OpenFDA, or OpenTargets + +For ADMET questions, reason through Absorption, Distribution, Metabolism, Excretion, Toxicity systematically. See references/ADMET_REFERENCE.md for detailed guidance. + +## Important Notes + +- All APIs are free, public, require no authentication +- ChEMBL rate limits: add sleep 1 between batch requests +- FDA data reflects reported adverse events, not necessarily causation +- Always recommend consulting a licensed pharmacist or physician for clinical decisions + +## Quick Reference + +| Task | API | Endpoint | +|------|-----|----------| +| Find target | ChEMBL | `/api/data/target/search?q=` | +| Get bioactivity | ChEMBL | `/api/data/activity?target_chembl_id=` | +| Molecule properties | PubChem | `/rest/pug/compound/name/{name}/property/` | +| Drug interactions | OpenFDA | `/drug/label.json?search=drug_interactions:` | +| Adverse events | OpenFDA | `/drug/event.json?search=...&count=reaction` | +| Gene-disease | OpenTargets | GraphQL POST `/api/v4/graphql` | diff --git a/optional-skills/research/drug-discovery/references/ADMET_REFERENCE.md b/optional-skills/research/drug-discovery/references/ADMET_REFERENCE.md new file mode 100644 index 000000000000..92a5e9503882 --- /dev/null +++ b/optional-skills/research/drug-discovery/references/ADMET_REFERENCE.md @@ -0,0 +1,66 @@ +# ADMET Reference Guide + +Comprehensive reference for Absorption, Distribution, Metabolism, Excretion, and Toxicity (ADMET) analysis in drug discovery. + +## Drug-Likeness Rule Sets + +### Lipinski's Rule of Five (Ro5) + +| Property | Threshold | +|----------|-----------| +| Molecular Weight (MW) | ≤ 500 Da | +| Lipophilicity (LogP) | ≤ 5 | +| H-Bond Donors (HBD) | ≤ 5 | +| H-Bond Acceptors (HBA) | ≤ 10 | + +Reference: Lipinski et al., Adv. Drug Deliv. Rev. 23, 3–25 (1997). + +### Veber's Oral Bioavailability Rules + +| Property | Threshold | +|----------|-----------| +| TPSA | ≤ 140 Ų | +| Rotatable Bonds | ≤ 10 | + +Reference: Veber et al., J. Med. Chem. 45, 2615–2623 (2002). + +### CNS Penetration (BBB) + +| Property | CNS-Optimal | +|----------|-------------| +| MW | ≤ 400 Da | +| LogP | 1–3 | +| TPSA | < 90 Ų | +| HBD | ≤ 3 | + +## CYP450 Metabolism + +| Isoform | % Drugs | Notable inhibitors | +|---------|---------|-------------------| +| CYP3A4 | ~50% | Grapefruit, ketoconazole | +| CYP2D6 | ~25% | Fluoxetine, paroxetine | +| CYP2C9 | ~15% | Fluconazole, amiodarone | +| CYP2C19 | ~10% | Omeprazole, fluoxetine | +| CYP1A2 | ~5% | Fluvoxamine, ciprofloxacin | + +## hERG Cardiac Toxicity Risk + +Structural alerts: basic nitrogen (pKa 7–9) + aromatic ring + hydrophobic moiety, LogP > 3.5 + basic amine. + +Mitigation: reduce basicity, introduce polar groups, break planarity. + +## Common Bioisosteric Replacements + +| Original | Bioisostere | Purpose | +|----------|-------------|---------| +| -COOH | -tetrazole, -SO₂NH₂ | Improve permeability | +| -OH (phenol) | -F, -CN | Reduce glucuronidation | +| Phenyl | Pyridine, thiophene | Reduce LogP | +| Ester | -CONHR | Reduce hydrolysis | + +## Key APIs + +- ChEMBL: https://www.ebi.ac.uk/chembl/api/data/ +- PubChem: https://pubchem.ncbi.nlm.nih.gov/rest/pug/ +- OpenFDA: https://api.fda.gov/drug/ +- OpenTargets GraphQL: https://api.platform.opentargets.org/api/v4/graphql diff --git a/optional-skills/research/drug-discovery/scripts/chembl_target.py b/optional-skills/research/drug-discovery/scripts/chembl_target.py new file mode 100644 index 000000000000..1346b999ab34 --- /dev/null +++ b/optional-skills/research/drug-discovery/scripts/chembl_target.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +""" +chembl_target.py — Search ChEMBL for a target and retrieve top active compounds. +Usage: python3 chembl_target.py "EGFR" --min-pchembl 7 --limit 20 +No external dependencies. +""" +import sys, json, time, argparse +import urllib.request, urllib.parse, urllib.error + +BASE = "https://www.ebi.ac.uk/chembl/api/data" + +def get(endpoint): + try: + req = urllib.request.Request(f"{BASE}{endpoint}", headers={"Accept":"application/json"}) + with urllib.request.urlopen(req, timeout=15) as r: + return json.loads(r.read()) + except Exception as e: + print(f"API error: {e}", file=sys.stderr); return None + +def main(): + parser = argparse.ArgumentParser(description="ChEMBL target → active compounds") + parser.add_argument("target") + parser.add_argument("--min-pchembl", type=float, default=6.0) + parser.add_argument("--limit", type=int, default=10) + args = parser.parse_args() + + enc = urllib.parse.quote(args.target) + data = get(f"/target/search?q={enc}&limit=5&format=json") + if not data or not data.get("targets"): + print("No targets found."); sys.exit(1) + + t = data["targets"][0] + tid = t.get("target_chembl_id","") + print(f"\nTarget: {t.get('pref_name')} ({tid})") + print(f"Type: {t.get('target_type')} | Organism: {t.get('organism','N/A')}") + print(f"\nFetching compounds with pChEMBL ≥ {args.min_pchembl}...\n") + + acts = get(f"/activity?target_chembl_id={tid}&pchembl_value__gte={args.min_pchembl}&assay_type=B&limit={args.limit}&order_by=-pchembl_value&format=json") + if not acts or not acts.get("activities"): + print("No activities found."); sys.exit(0) + + print(f"{'Molecule':<18} {'pChEMBL':>8} {'Type':<12} {'Value':<10} {'Units'}") + print("-"*65) + seen = set() + for a in acts["activities"]: + mid = a.get("molecule_chembl_id","N/A") + if mid in seen: continue + seen.add(mid) + print(f"{mid:<18} {str(a.get('pchembl_value','N/A')):>8} {str(a.get('standard_type','N/A')):<12} {str(a.get('standard_value','N/A')):<10} {a.get('standard_units','N/A')}") + time.sleep(0.1) + print(f"\nTotal: {len(seen)} unique molecules") + +if __name__ == "__main__": main() diff --git a/optional-skills/research/drug-discovery/scripts/ro5_screen.py b/optional-skills/research/drug-discovery/scripts/ro5_screen.py new file mode 100644 index 000000000000..84e438fa14b9 --- /dev/null +++ b/optional-skills/research/drug-discovery/scripts/ro5_screen.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +""" +ro5_screen.py — Batch Lipinski Ro5 + Veber screening via PubChem API. +Usage: python3 ro5_screen.py aspirin ibuprofen paracetamol +No external dependencies beyond stdlib. +""" +import sys, json, time, argparse +import urllib.request, urllib.parse, urllib.error + +BASE = "https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name" +PROPS = "MolecularWeight,XLogP,HBondDonorCount,HBondAcceptorCount,RotatableBondCount,TPSA" + +def fetch(name): + url = f"{BASE}/{urllib.parse.quote(name)}/property/{PROPS}/JSON" + try: + with urllib.request.urlopen(url, timeout=10) as r: + return json.loads(r.read())["PropertyTable"]["Properties"][0] + except Exception: + return None + +def check(p): + mw,logp,hbd,hba,rot,tpsa = float(p.get("MolecularWeight",0)),float(p.get("XLogP",0)),int(p.get("HBondDonorCount",0)),int(p.get("HBondAcceptorCount",0)),int(p.get("RotatableBondCount",0)),float(p.get("TPSA",0)) + v = sum([mw>500,logp>5,hbd>5,hba>10]) + return dict(mw=mw,logp=logp,hbd=hbd,hba=hba,rot=rot,tpsa=tpsa,violations=v,ro5=v<=1,veber=tpsa<=140 and rot<=10,ok=v<=1 and tpsa<=140 and rot<=10) + +def report(name, r): + if not r: print(f"✗ {name:30s} — not found"); return + s = "✓ PASS" if r["ok"] else "✗ FAIL" + flags = (f" [Ro5 violations:{r['violations']}]" if not r["ro5"] else "") + (" [Veber fail]" if not r["veber"] else "") + print(f"{s} {name:28s} MW={r['mw']:.0f} LogP={r['logp']:.2f} HBD={r['hbd']} HBA={r['hba']} TPSA={r['tpsa']:.0f} RotB={r['rot']}{flags}") + +def main(): + compounds = sys.stdin.read().splitlines() if len(sys.argv)<2 or sys.argv[1]=="-" else sys.argv[1:] + print(f"\n{'Status':<8} {'Compound':<30} Properties\n" + "-"*85) + passed = 0 + for name in compounds: + props = fetch(name.strip()) + result = check(props) if props else None + report(name.strip(), result) + if result and result["ok"]: passed += 1 + time.sleep(0.3) + print(f"\nSummary: {passed}/{len(compounds)} passed Ro5 + Veber.\n") + +if __name__ == "__main__": main() diff --git a/optional-skills/research/duckduckgo-search/SKILL.md b/optional-skills/research/duckduckgo-search/SKILL.md index ea14e6b30f2d..c24fc1b9564e 100644 --- a/optional-skills/research/duckduckgo-search/SKILL.md +++ b/optional-skills/research/duckduckgo-search/SKILL.md @@ -57,32 +57,32 @@ Use the `ddgs` command via `terminal` when it exists. This is the preferred path ```bash # Text search -ddgs text -k "python async programming" -m 5 +ddgs text -q "python async programming" -m 5 # News search -ddgs news -k "artificial intelligence" -m 5 +ddgs news -q "artificial intelligence" -m 5 # Image search -ddgs images -k "landscape photography" -m 10 +ddgs images -q "landscape photography" -m 10 # Video search -ddgs videos -k "python tutorial" -m 5 +ddgs videos -q "python tutorial" -m 5 # With region filter -ddgs text -k "best restaurants" -m 5 -r us-en +ddgs text -q "best restaurants" -m 5 -r us-en # Recent results only (d=day, w=week, m=month, y=year) -ddgs text -k "latest AI news" -m 5 -t w +ddgs text -q "latest AI news" -m 5 -t w # JSON output for parsing -ddgs text -k "fastapi tutorial" -m 5 -o json +ddgs text -q "fastapi tutorial" -m 5 -o json ``` ### CLI Flags | Flag | Description | Example | |------|-------------|---------| -| `-k` | Keywords (query) — **required** | `-k "search terms"` | +| `-q` | Query — **required** | `-q "search terms"` | | `-m` | Max results | `-m 5` | | `-r` | Region | `-r us-en` | | `-t` | Time limit | `-t w` (week) | @@ -189,7 +189,7 @@ DuckDuckGo returns titles, URLs, and snippets — not full page content. To get CLI example: ```bash -ddgs text -k "fastapi deployment guide" -m 3 -o json +ddgs text -q "fastapi deployment guide" -m 3 -o json ``` Python example, only after verifying `ddgs` is installed in that runtime: @@ -229,7 +229,7 @@ Then extract the best URL with `web_extract` or another content-retrieval tool. - **Do not assume the CLI exists**: Check `command -v ddgs` before using it. - **Do not assume `execute_code` can import `ddgs`**: `from ddgs import DDGS` may fail with `ModuleNotFoundError` unless that runtime was prepared separately. - **Package name**: The package is `ddgs` (previously `duckduckgo-search`). Install with `pip install ddgs`. -- **Don't confuse `-k` and `-m`** (CLI): `-k` is for keywords, `-m` is for max results count. +- **Don't confuse `-q` and `-m`** (CLI): `-q` is for the query, `-m` is for max results count. - **Empty results**: If `ddgs` returns nothing, it may be rate-limited. Wait a few seconds and retry. ## Validated With diff --git a/optional-skills/research/duckduckgo-search/scripts/duckduckgo.sh b/optional-skills/research/duckduckgo-search/scripts/duckduckgo.sh index b33ac8a60d15..1553d45968cb 100755 --- a/optional-skills/research/duckduckgo-search/scripts/duckduckgo.sh +++ b/optional-skills/research/duckduckgo-search/scripts/duckduckgo.sh @@ -25,4 +25,4 @@ if ! command -v ddgs &> /dev/null; then exit 1 fi -ddgs text -k "$QUERY" -m "$MAX_RESULTS" +ddgs text -q "$QUERY" -m "$MAX_RESULTS" diff --git a/optional-skills/web-development/DESCRIPTION.md b/optional-skills/web-development/DESCRIPTION.md new file mode 100644 index 000000000000..588817bbcacd --- /dev/null +++ b/optional-skills/web-development/DESCRIPTION.md @@ -0,0 +1,5 @@ +# Web Development + +Optional skills for client-side web development workflows — embedding agents, copilots, and AI-native UX patterns into user-facing web apps. + +These are distinct from Hermes' own browser automation (Browserbase, Camofox), which operate *on* websites from outside. Web-development skills here help users build *into* their own websites. diff --git a/optional-skills/web-development/page-agent/SKILL.md b/optional-skills/web-development/page-agent/SKILL.md new file mode 100644 index 000000000000..caab19901fe8 --- /dev/null +++ b/optional-skills/web-development/page-agent/SKILL.md @@ -0,0 +1,189 @@ +--- +name: page-agent +description: Embed alibaba/page-agent into your own web application — a pure-JavaScript in-page GUI agent that ships as a single +``` + +A panel appears. Type an instruction. Done. + +Bookmarklet form (drop into bookmarks bar, click on any page): + +```javascript +javascript:(function(){var s=document.createElement('script');s.src='https://cdn.jsdelivr.net/npm/page-agent@1.8.0/dist/iife/page-agent.demo.js';document.head.appendChild(s);})(); +``` + +## Path 2 — npm install into your own web app (production use) + +Inside an existing web project (React / Vue / Svelte / plain): + +```bash +npm install page-agent +``` + +Wire it up with your own LLM endpoint — **never ship the demo CDN to real users**: + +```javascript +import { PageAgent } from 'page-agent' + +const agent = new PageAgent({ + model: 'qwen3.5-plus', + baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1', + apiKey: process.env.LLM_API_KEY, // never hardcode + language: 'en-US', +}) + +// Show the panel for end users: +agent.panel.show() + +// Or drive it programmatically: +await agent.execute('Click submit button, then fill username as John') +``` + +Provider examples (any OpenAI-compatible endpoint works): + +| Provider | `baseURL` | `model` | +|----------|-----------|---------| +| Qwen / DashScope | `https://dashscope.aliyuncs.com/compatible-mode/v1` | `qwen3.5-plus` | +| OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` | +| Ollama (local) | `http://localhost:11434/v1` | `qwen3:14b` | +| OpenRouter | `https://openrouter.ai/api/v1` | `anthropic/claude-sonnet-4.6` | + +**Key config fields** (passed to `new PageAgent({...})`): + +- `model`, `baseURL`, `apiKey` — LLM connection +- `language` — UI language (`en-US`, `zh-CN`, etc.) +- Allowlist and data-masking hooks exist for locking down what the agent can touch — see https://alibaba.github.io/page-agent/ for the full option list + +**Security.** Don't put your `apiKey` in client-side code for a real deployment — proxy LLM calls through your backend and point `baseURL` at your proxy. The demo CDN exists because alibaba runs that proxy for evaluation. + +## Path 3 — clone the source repo (contributing, or hacking on it) + +Use this when the user wants to modify page-agent itself, test it against arbitrary sites via a local IIFE bundle, or develop the browser extension. + +```bash +git clone https://github.com/alibaba/page-agent.git +cd page-agent +npm ci # exact lockfile install (or `npm i` to allow updates) +``` + +Create `.env` in the repo root with an LLM endpoint. Example: + +``` +LLM_MODEL_NAME=gpt-4o-mini +LLM_API_KEY=sk-... +LLM_BASE_URL=https://api.openai.com/v1 +``` + +Ollama flavor: + +``` +LLM_BASE_URL=http://localhost:11434/v1 +LLM_API_KEY=NA +LLM_MODEL_NAME=qwen3:14b +``` + +Common commands: + +```bash +npm start # docs/website dev server +npm run build # build every package +npm run dev:demo # serve IIFE bundle at http://localhost:5174/page-agent.demo.js +npm run dev:ext # develop the browser extension (WXT + React) +npm run build:ext # build the extension +``` + +**Test on any website** using the local IIFE bundle. Add this bookmarklet: + +```javascript +javascript:(function(){var s=document.createElement('script');s.src=`http://localhost:5174/page-agent.demo.js?t=${Math.random()}`;s.onload=()=>console.log('PageAgent ready!');document.head.appendChild(s);})(); +``` + +Then: `npm run dev:demo`, click the bookmarklet on any page, and the local build injects. Auto-rebuilds on save. + +**Warning:** your `.env` `LLM_API_KEY` is inlined into the IIFE bundle during dev builds. Don't share the bundle. Don't commit it. Don't paste the URL into Slack. (Verified: grepping the public dev bundle returns the literal values from `.env`.) + +## Repo layout (Path 3) + +Monorepo with npm workspaces. Key packages: + +| Package | Path | Purpose | +|---------|------|---------| +| `page-agent` | `packages/page-agent/` | Main entry with UI panel | +| `@page-agent/core` | `packages/core/` | Core agent logic, no UI | +| `@page-agent/mcp` | `packages/mcp/` | MCP server (beta) | +| — | `packages/llms/` | LLM client | +| — | `packages/page-controller/` | DOM ops + visual feedback | +| — | `packages/ui/` | Panel + i18n | +| — | `packages/extension/` | Chrome/Firefox extension | +| — | `packages/website/` | Docs + landing site | + +## Verifying it works + +After Path 1 or Path 2: +1. Open the page in a browser with devtools open +2. You should see a floating panel. If not, check the console for errors (most common: CORS on the LLM endpoint, wrong `baseURL`, or a bad API key) +3. Type a simple instruction matching something visible on the page ("click the Login link") +4. Watch the Network tab — you should see a request to your `baseURL` + +After Path 3: +1. `npm run dev:demo` prints `Accepting connections at http://localhost:5174` +2. `curl -I http://localhost:5174/page-agent.demo.js` returns `HTTP/1.1 200 OK` with `Content-Type: application/javascript` +3. Click the bookmarklet on any site; panel appears + +## Pitfalls + +- **Demo CDN in production** — don't. It's rate-limited, uses alibaba's free proxy, and their terms forbid production use. +- **API key exposure** — any key passed to `new PageAgent({apiKey: ...})` ships in your JS bundle. Always proxy through your own backend for real deployments. +- **Non-OpenAI-compatible endpoints** fail silently or with cryptic errors. If your provider needs native Anthropic/Gemini formatting, use an OpenAI-compatibility proxy (LiteLLM, OpenRouter) in front. +- **CSP blocks** — sites with strict Content-Security-Policy may refuse to load the CDN script or disallow inline eval. In that case, self-host from your origin. +- **Restart dev server** after editing `.env` in Path 3 — Vite only reads env at startup. +- **Node version** — the repo declares `^22.13.0 || >=24`. Node 20 will fail `npm ci` with engine errors. +- **npm 10 vs 11** — docs say npm 11+; npm 10.9 actually works fine. + +## Reference + +- Repo: https://github.com/alibaba/page-agent +- Docs: https://alibaba.github.io/page-agent/ +- License: MIT (built on browser-use's DOM processing internals, Copyright 2024 Gregor Zunic) diff --git a/package-lock.json b/package-lock.json index de94d14675bd..8309e3b7a96d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -10,215 +10,39 @@ "hasInstallScript": true, "license": "MIT", "dependencies": { - "@askjo/camoufox-browser": "^1.0.0", - "agent-browser": "^0.13.0" + "@askjo/camofox-browser": "^1.5.2", + "agent-browser": "^0.26.0" }, "engines": { - "node": ">=18.0.0" - } - }, - "node_modules/@appium/logger": { - "version": "1.7.1", - "resolved": "https://registry.npmjs.org/@appium/logger/-/logger-1.7.1.tgz", - "integrity": "sha512-9C2o9X/lBEDBUnKfAi3mRo9oG7Z03nmISLwsGkWxIWjMAvBdJD0RRSJMekWVKzfXN3byrI1WlCXTITzN4LAoLw==", - "license": "ISC", - "dependencies": { - "console-control-strings": "1.1.0", - "lodash": "4.17.21", - "lru-cache": "10.4.3", - "set-blocking": "2.0.0" - }, - "engines": { - "node": "^14.17.0 || ^16.13.0 || >=18.0.0", - "npm": ">=8" + "node": ">=20.0.0" } }, - "node_modules/@askjo/camoufox-browser": { - "version": "1.0.12", - "resolved": "https://registry.npmjs.org/@askjo/camoufox-browser/-/camoufox-browser-1.0.12.tgz", - "integrity": "sha512-MxRvjK6SkX6zJSNleoO32g9iwhJAcXpaAgj4pik7y2SrYXqcHllpG7FfLkKE7d5bnBt7pO82rdarVYu6xtW2RA==", - "deprecated": "Renamed to @askjo/camofox-browser", + "node_modules/@askjo/camofox-browser": { + "version": "1.5.2", + "resolved": "https://registry.npmjs.org/@askjo/camofox-browser/-/camofox-browser-1.5.2.tgz", + "integrity": "sha512-SvRCzhWnJaplxHkRVF9l1OWako6pp2eUw2mZKHOERUfLWDO2Xe/IKI+5bB+UT1TNvO45P6XdhgfAtihcTEARCg==", "hasInstallScript": true, "license": "MIT", "dependencies": { "camoufox-js": "^0.8.5", - "dotenv": "^17.2.3", "express": "^4.18.2", "playwright": "^1.50.0", "playwright-core": "^1.58.0", "playwright-extra": "^4.3.6", + "prom-client": "^15.1.3", "puppeteer-extra-plugin-stealth": "^2.11.2" }, "engines": { "node": ">=18" } }, - "node_modules/@isaacs/cliui": { - "version": "8.0.2", - "resolved": "https://registry.npmjs.org/@isaacs/cliui/-/cliui-8.0.2.tgz", - "integrity": "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==", - "license": "ISC", - "dependencies": { - "string-width": "^5.1.2", - "string-width-cjs": "npm:string-width@^4.2.0", - "strip-ansi": "^7.0.1", - "strip-ansi-cjs": "npm:strip-ansi@^6.0.1", - "wrap-ansi": "^8.1.0", - "wrap-ansi-cjs": "npm:wrap-ansi@^7.0.0" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/@isaacs/cliui/node_modules/ansi-styles": { - "version": "6.2.3", - "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-6.2.3.tgz", - "integrity": "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg==", - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/chalk/ansi-styles?sponsor=1" - } - }, - "node_modules/@isaacs/cliui/node_modules/emoji-regex": { - "version": "9.2.2", - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-9.2.2.tgz", - "integrity": "sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg==", - "license": "MIT" - }, - "node_modules/@isaacs/cliui/node_modules/string-width": { - "version": "5.1.2", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-5.1.2.tgz", - "integrity": "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA==", - "license": "MIT", - "dependencies": { - "eastasianwidth": "^0.2.0", - "emoji-regex": "^9.2.2", - "strip-ansi": "^7.0.1" - }, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/@isaacs/cliui/node_modules/wrap-ansi": { - "version": "8.1.0", - "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-8.1.0.tgz", - "integrity": "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==", - "license": "MIT", - "dependencies": { - "ansi-styles": "^6.1.0", - "string-width": "^5.0.1", - "strip-ansi": "^7.0.1" - }, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/chalk/wrap-ansi?sponsor=1" - } - }, - "node_modules/@pkgjs/parseargs": { - "version": "0.11.0", - "resolved": "https://registry.npmjs.org/@pkgjs/parseargs/-/parseargs-0.11.0.tgz", - "integrity": "sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=14" - } - }, - "node_modules/@promptbook/utils": { - "version": "0.69.5", - "resolved": "https://registry.npmjs.org/@promptbook/utils/-/utils-0.69.5.tgz", - "integrity": "sha512-xm5Ti/Hp3o4xHrsK9Yy3MS6KbDxYbq485hDsFvxqaNA7equHLPdo8H8faTitTeb14QCDfLW4iwCxdVYu5sn6YQ==", - "funding": [ - { - "type": "individual", - "url": "https://buymeacoffee.com/hejny" - }, - { - "type": "github", - "url": "https://github.com/webgptorg/promptbook/blob/main/README.md#%EF%B8%8F-contributing" - } - ], - "license": "CC-BY-4.0", - "dependencies": { - "spacetrim": "0.11.59" - } - }, - "node_modules/@puppeteer/browsers": { - "version": "2.13.0", - "resolved": "https://registry.npmjs.org/@puppeteer/browsers/-/browsers-2.13.0.tgz", - "integrity": "sha512-46BZJYJjc/WwmKjsvDFykHtXrtomsCIrwYQPOP7VfMJoZY2bsDF9oROBABR3paDjDcmkUye1Pb1BqdcdiipaWA==", + "node_modules/@opentelemetry/api": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/@opentelemetry/api/-/api-1.9.1.tgz", + "integrity": "sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==", "license": "Apache-2.0", - "dependencies": { - "debug": "^4.4.3", - "extract-zip": "^2.0.1", - "progress": "^2.0.3", - "proxy-agent": "^6.5.0", - "semver": "^7.7.4", - "tar-fs": "^3.1.1", - "yargs": "^17.7.2" - }, - "bin": { - "browsers": "lib/cjs/main-cli.js" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/@puppeteer/browsers/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/@puppeteer/browsers/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/@puppeteer/browsers/node_modules/tar-fs": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.1.2.tgz", - "integrity": "sha512-QGxxTxxyleAdyM3kpFs14ymbYmNFrfY+pHj7Z8FgtbZ7w2//VAgLMac7sT6nRpIHjppXO2AwwEOg0bPFVRcmXw==", - "license": "MIT", - "dependencies": { - "pump": "^3.0.0", - "tar-stream": "^3.1.5" - }, - "optionalDependencies": { - "bare-fs": "^4.0.1", - "bare-path": "^3.0.0" - } - }, - "node_modules/@puppeteer/browsers/node_modules/tar-stream": { - "version": "3.1.8", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.1.8.tgz", - "integrity": "sha512-U6QpVRyCGHva435KoNWy9PRoi2IFYCgtEhq9nmrPPpbRacPs9IH4aJ3gbrFC8dPcXvdSZ4XXfXT5Fshbp2MtlQ==", - "license": "MIT", - "dependencies": { - "b4a": "^1.6.4", - "bare-fs": "^4.5.5", - "fast-fifo": "^1.2.0", - "streamx": "^2.15.0" + "node": ">=8.0.0" } }, "node_modules/@sindresorhus/is": { @@ -233,12 +57,6 @@ "url": "https://github.com/sindresorhus/is?sponsor=1" } }, - "node_modules/@tootallnate/quickjs-emscripten": { - "version": "0.23.0", - "resolved": "https://registry.npmjs.org/@tootallnate/quickjs-emscripten/-/quickjs-emscripten-0.23.0.tgz", - "integrity": "sha512-C5Mc6rdnsaJDjO3UpGW/CQTHtCKaYlScZTly4JIu97Jxo/odCiH0ITnDXSJPTOrEKk/ycSZ0AOgTmkDtkOsvIA==", - "license": "MIT" - }, "node_modules/@types/debug": { "version": "4.1.13", "resolved": "https://registry.npmjs.org/@types/debug/-/debug-4.1.13.tgz", @@ -254,225 +72,6 @@ "integrity": "sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA==", "license": "MIT" }, - "node_modules/@types/node": { - "version": "20.19.39", - "resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.39.tgz", - "integrity": "sha512-orrrD74MBUyK8jOAD/r0+lfa1I2MO6I+vAkmAWzMYbCcgrN4lCrmK52gRFQq/JRxfYPfonkr4b0jcY7Olqdqbw==", - "license": "MIT", - "dependencies": { - "undici-types": "~6.21.0" - } - }, - "node_modules/@types/sinonjs__fake-timers": { - "version": "8.1.5", - "resolved": "https://registry.npmjs.org/@types/sinonjs__fake-timers/-/sinonjs__fake-timers-8.1.5.tgz", - "integrity": "sha512-mQkU2jY8jJEF7YHjHvsQO8+3ughTL1mcnn96igfhONmR+fUPSKIkefQYpSe8bsly2Ep7oQbn/6VG5/9/0qcArQ==", - "license": "MIT" - }, - "node_modules/@types/which": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/@types/which/-/which-2.0.2.tgz", - "integrity": "sha512-113D3mDkZDjo+EeUEHCFy0qniNc1ZpecGiAU7WSo7YDoSzolZIQKpYFHrPpjkB2nuyahcKfrmLXeQlh7gqJYdw==", - "license": "MIT" - }, - "node_modules/@types/ws": { - "version": "8.18.1", - "resolved": "https://registry.npmjs.org/@types/ws/-/ws-8.18.1.tgz", - "integrity": "sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg==", - "license": "MIT", - "dependencies": { - "@types/node": "*" - } - }, - "node_modules/@types/yauzl": { - "version": "2.10.3", - "resolved": "https://registry.npmjs.org/@types/yauzl/-/yauzl-2.10.3.tgz", - "integrity": "sha512-oJoftv0LSuaDZE3Le4DbKX+KS9G36NzOeSap90UIK0yMA/NhKJhqlSGtNDORNRaIbQfzjXDrQa0ytJ6mNRGz/Q==", - "license": "MIT", - "optional": true, - "dependencies": { - "@types/node": "*" - } - }, - "node_modules/@wdio/config": { - "version": "9.27.0", - "resolved": "https://registry.npmjs.org/@wdio/config/-/config-9.27.0.tgz", - "integrity": "sha512-9y8z7ugIbU6ycKrA2SqCpKh1/hobut2rDq9CLt/BNVzSlebBBVOTMiAt1XroZzcPnA7/ZqpbkpOsbpPUaAQuNQ==", - "license": "MIT", - "dependencies": { - "@wdio/logger": "9.18.0", - "@wdio/types": "9.27.0", - "@wdio/utils": "9.27.0", - "deepmerge-ts": "^7.0.3", - "glob": "^10.2.2", - "import-meta-resolve": "^4.0.0", - "jiti": "^2.6.1" - }, - "engines": { - "node": ">=18.20.0" - } - }, - "node_modules/@wdio/config/node_modules/balanced-match": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", - "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", - "license": "MIT" - }, - "node_modules/@wdio/config/node_modules/brace-expansion": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.0.tgz", - "integrity": "sha512-TN1kCZAgdgweJhWWpgKYrQaMNHcDULHkWwQIspdtjV4Y5aurRdZpjAqn6yX3FPqTA9ngHCc4hJxMAMgGfve85w==", - "license": "MIT", - "dependencies": { - "balanced-match": "^1.0.0" - } - }, - "node_modules/@wdio/config/node_modules/glob": { - "version": "10.5.0", - "resolved": "https://registry.npmjs.org/glob/-/glob-10.5.0.tgz", - "integrity": "sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==", - "deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me", - "license": "ISC", - "dependencies": { - "foreground-child": "^3.1.0", - "jackspeak": "^3.1.2", - "minimatch": "^9.0.4", - "minipass": "^7.1.2", - "package-json-from-dist": "^1.0.0", - "path-scurry": "^1.11.1" - }, - "bin": { - "glob": "dist/esm/bin.mjs" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/@wdio/config/node_modules/minimatch": { - "version": "9.0.9", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.9.tgz", - "integrity": "sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg==", - "license": "ISC", - "dependencies": { - "brace-expansion": "^2.0.2" - }, - "engines": { - "node": ">=16 || 14 >=14.17" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/@wdio/config/node_modules/path-scurry": { - "version": "1.11.1", - "resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-1.11.1.tgz", - "integrity": "sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA==", - "license": "BlueOak-1.0.0", - "dependencies": { - "lru-cache": "^10.2.0", - "minipass": "^5.0.0 || ^6.0.2 || ^7.0.0" - }, - "engines": { - "node": ">=16 || 14 >=14.18" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/@wdio/logger": { - "version": "9.18.0", - "resolved": "https://registry.npmjs.org/@wdio/logger/-/logger-9.18.0.tgz", - "integrity": "sha512-HdzDrRs+ywAqbXGKqe1i/bLtCv47plz4TvsHFH3j729OooT5VH38ctFn5aLXgECmiAKDkmH/A6kOq2Zh5DIxww==", - "license": "MIT", - "dependencies": { - "chalk": "^5.1.2", - "loglevel": "^1.6.0", - "loglevel-plugin-prefix": "^0.8.4", - "safe-regex2": "^5.0.0", - "strip-ansi": "^7.1.0" - }, - "engines": { - "node": ">=18.20.0" - } - }, - "node_modules/@wdio/protocols": { - "version": "9.27.0", - "resolved": "https://registry.npmjs.org/@wdio/protocols/-/protocols-9.27.0.tgz", - "integrity": "sha512-rIk69BsY1+6uU2PEN5FiRpI6K7HJ86YHzZRFBe4iRzKXQgGNk1zWzbdVJIuNFoOWsnmYUkK42KSSOT4Le6EmiQ==", - "license": "MIT" - }, - "node_modules/@wdio/repl": { - "version": "9.16.2", - "resolved": "https://registry.npmjs.org/@wdio/repl/-/repl-9.16.2.tgz", - "integrity": "sha512-FLTF0VL6+o5BSTCO7yLSXocm3kUnu31zYwzdsz4n9s5YWt83sCtzGZlZpt7TaTzb3jVUfxuHNQDTb8UMkCu0lQ==", - "license": "MIT", - "dependencies": { - "@types/node": "^20.1.0" - }, - "engines": { - "node": ">=18.20.0" - } - }, - "node_modules/@wdio/types": { - "version": "9.27.0", - "resolved": "https://registry.npmjs.org/@wdio/types/-/types-9.27.0.tgz", - "integrity": "sha512-DQJ+OdRBqUBcQ30DN2Z651hEVh3OoxnlDUSRqlWy9An2AY6v9rYWTj825B6zsj5pLLEToYO1tfwWq0ab183pXg==", - "license": "MIT", - "dependencies": { - "@types/node": "^20.1.0" - }, - "engines": { - "node": ">=18.20.0" - } - }, - "node_modules/@wdio/utils": { - "version": "9.27.0", - "resolved": "https://registry.npmjs.org/@wdio/utils/-/utils-9.27.0.tgz", - "integrity": "sha512-fUasd5OKJTy2seJfWnYZ9xlxTtY0p/Kyeuh7Tbb8kcofBqmBi2fTvM3sfZlo1tGQX9yCh+IS2N7hlfyFMmuZ+w==", - "license": "MIT", - "dependencies": { - "@puppeteer/browsers": "^2.2.0", - "@wdio/logger": "9.18.0", - "@wdio/types": "9.27.0", - "decamelize": "^6.0.0", - "deepmerge-ts": "^7.0.3", - "edgedriver": "^6.1.2", - "geckodriver": "^6.1.0", - "get-port": "^7.0.0", - "import-meta-resolve": "^4.0.0", - "locate-app": "^2.2.24", - "mitt": "^3.0.1", - "safaridriver": "^1.0.0", - "split2": "^4.2.0", - "wait-port": "^1.1.0" - }, - "engines": { - "node": ">=18.20.0" - } - }, - "node_modules/@zip.js/zip.js": { - "version": "2.8.26", - "resolved": "https://registry.npmjs.org/@zip.js/zip.js/-/zip.js-2.8.26.tgz", - "integrity": "sha512-RQ4h9F6DOiHxpdocUDrOl6xBM+yOtz+LkUol47AVWcfebGBDpZ7w7Xvz9PS24JgXvLGiXXzSAfdCdVy1tPlaFA==", - "license": "BSD-3-Clause", - "engines": { - "bun": ">=0.7.0", - "deno": ">=1.0.0", - "node": ">=18.0.0" - } - }, - "node_modules/abort-controller": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz", - "integrity": "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg==", - "license": "MIT", - "dependencies": { - "event-target-shim": "^5.0.0" - }, - "engines": { - "node": ">=6.5" - } - }, "node_modules/accepts": { "version": "1.3.8", "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz", @@ -495,263 +94,16 @@ "node": ">=12.0" } }, - "node_modules/agent-base": { - "version": "7.1.4", - "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz", - "integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==", - "license": "MIT", - "engines": { - "node": ">= 14" - } - }, "node_modules/agent-browser": { - "version": "0.13.0", - "resolved": "https://registry.npmjs.org/agent-browser/-/agent-browser-0.13.0.tgz", - "integrity": "sha512-KGtiqzu8EA8nPAZIp+1lq+PBG86brLEvB28aE/Aeh1ErOVBHICsh/ShwCPUKMjMIS65qiVV/FKG/3xN0jn8J3A==", + "version": "0.26.0", + "resolved": "https://registry.npmjs.org/agent-browser/-/agent-browser-0.26.0.tgz", + "integrity": "sha512-pdqSfjwbFSp+qnwlb2g23e9wXveIOfMi19xpPA9xZUbzEAUp6W4YBZj6Ybj8z4M7WkcbGDDYc+oDIHDt9R3EDQ==", "hasInstallScript": true, "license": "Apache-2.0", - "dependencies": { - "node-simctl": "^7.4.0", - "playwright-core": "^1.57.0", - "webdriverio": "^9.15.0", - "ws": "^8.19.0", - "zod": "^3.22.4" - }, "bin": { "agent-browser": "bin/agent-browser.js" } }, - "node_modules/ansi-regex": { - "version": "6.2.2", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-6.2.2.tgz", - "integrity": "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg==", - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/chalk/ansi-regex?sponsor=1" - } - }, - "node_modules/ansi-styles": { - "version": "4.3.0", - "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz", - "integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==", - "license": "MIT", - "dependencies": { - "color-convert": "^2.0.1" - }, - "engines": { - "node": ">=8" - }, - "funding": { - "url": "https://github.com/chalk/ansi-styles?sponsor=1" - } - }, - "node_modules/archiver": { - "version": "7.0.1", - "resolved": "https://registry.npmjs.org/archiver/-/archiver-7.0.1.tgz", - "integrity": "sha512-ZcbTaIqJOfCc03QwD468Unz/5Ir8ATtvAHsK+FdXbDIbGfihqh9mrvdcYunQzqn4HrvWWaFyaxJhGZagaJJpPQ==", - "license": "MIT", - "dependencies": { - "archiver-utils": "^5.0.2", - "async": "^3.2.4", - "buffer-crc32": "^1.0.0", - "readable-stream": "^4.0.0", - "readdir-glob": "^1.1.2", - "tar-stream": "^3.0.0", - "zip-stream": "^6.0.1" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/archiver-utils": { - "version": "5.0.2", - "resolved": "https://registry.npmjs.org/archiver-utils/-/archiver-utils-5.0.2.tgz", - "integrity": "sha512-wuLJMmIBQYCsGZgYLTy5FIB2pF6Lfb6cXMSF8Qywwk3t20zWnAi7zLcQFdKQmIB8wyZpY5ER38x08GbwtR2cLA==", - "license": "MIT", - "dependencies": { - "glob": "^10.0.0", - "graceful-fs": "^4.2.0", - "is-stream": "^2.0.1", - "lazystream": "^1.0.0", - "lodash": "^4.17.15", - "normalize-path": "^3.0.0", - "readable-stream": "^4.0.0" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/archiver-utils/node_modules/balanced-match": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", - "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", - "license": "MIT" - }, - "node_modules/archiver-utils/node_modules/brace-expansion": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.0.tgz", - "integrity": "sha512-TN1kCZAgdgweJhWWpgKYrQaMNHcDULHkWwQIspdtjV4Y5aurRdZpjAqn6yX3FPqTA9ngHCc4hJxMAMgGfve85w==", - "license": "MIT", - "dependencies": { - "balanced-match": "^1.0.0" - } - }, - "node_modules/archiver-utils/node_modules/buffer": { - "version": "6.0.3", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz", - "integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.2.1" - } - }, - "node_modules/archiver-utils/node_modules/glob": { - "version": "10.5.0", - "resolved": "https://registry.npmjs.org/glob/-/glob-10.5.0.tgz", - "integrity": "sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==", - "deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me", - "license": "ISC", - "dependencies": { - "foreground-child": "^3.1.0", - "jackspeak": "^3.1.2", - "minimatch": "^9.0.4", - "minipass": "^7.1.2", - "package-json-from-dist": "^1.0.0", - "path-scurry": "^1.11.1" - }, - "bin": { - "glob": "dist/esm/bin.mjs" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/archiver-utils/node_modules/minimatch": { - "version": "9.0.9", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.9.tgz", - "integrity": "sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg==", - "license": "ISC", - "dependencies": { - "brace-expansion": "^2.0.2" - }, - "engines": { - "node": ">=16 || 14 >=14.17" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/archiver-utils/node_modules/path-scurry": { - "version": "1.11.1", - "resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-1.11.1.tgz", - "integrity": "sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA==", - "license": "BlueOak-1.0.0", - "dependencies": { - "lru-cache": "^10.2.0", - "minipass": "^5.0.0 || ^6.0.2 || ^7.0.0" - }, - "engines": { - "node": ">=16 || 14 >=14.18" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/archiver-utils/node_modules/readable-stream": { - "version": "4.7.0", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz", - "integrity": "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg==", - "license": "MIT", - "dependencies": { - "abort-controller": "^3.0.0", - "buffer": "^6.0.3", - "events": "^3.3.0", - "process": "^0.11.10", - "string_decoder": "^1.3.0" - }, - "engines": { - "node": "^12.22.0 || ^14.17.0 || >=16.0.0" - } - }, - "node_modules/archiver/node_modules/buffer": { - "version": "6.0.3", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz", - "integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.2.1" - } - }, - "node_modules/archiver/node_modules/readable-stream": { - "version": "4.7.0", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz", - "integrity": "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg==", - "license": "MIT", - "dependencies": { - "abort-controller": "^3.0.0", - "buffer": "^6.0.3", - "events": "^3.3.0", - "process": "^0.11.10", - "string_decoder": "^1.3.0" - }, - "engines": { - "node": "^12.22.0 || ^14.17.0 || >=16.0.0" - } - }, - "node_modules/archiver/node_modules/tar-stream": { - "version": "3.1.8", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.1.8.tgz", - "integrity": "sha512-U6QpVRyCGHva435KoNWy9PRoi2IFYCgtEhq9nmrPPpbRacPs9IH4aJ3gbrFC8dPcXvdSZ4XXfXT5Fshbp2MtlQ==", - "license": "MIT", - "dependencies": { - "b4a": "^1.6.4", - "bare-fs": "^4.5.5", - "fast-fifo": "^1.2.0", - "streamx": "^2.15.0" - } - }, - "node_modules/aria-query": { - "version": "5.3.2", - "resolved": "https://registry.npmjs.org/aria-query/-/aria-query-5.3.2.tgz", - "integrity": "sha512-COROpnaoap1E2F000S62r6A60uHZnmlvomhfyT2DlTcrY1OrBKn2UhH7qn5wTC9zMvD0AY7csdPSNwKP+7WiQw==", - "license": "Apache-2.0", - "engines": { - "node": ">= 0.4" - } - }, "node_modules/arr-union": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/arr-union/-/arr-union-3.1.0.tgz", @@ -767,52 +119,6 @@ "integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==", "license": "MIT" }, - "node_modules/ast-types": { - "version": "0.13.4", - "resolved": "https://registry.npmjs.org/ast-types/-/ast-types-0.13.4.tgz", - "integrity": "sha512-x1FCFnFifvYDDzTaLII71vG5uvDwgtmDTEVWAxrgeiR8VjMONcCXJx7E+USjDtHlwFmt9MysbqgF9b9Vjr6w+w==", - "license": "MIT", - "dependencies": { - "tslib": "^2.0.1" - }, - "engines": { - "node": ">=4" - } - }, - "node_modules/async": { - "version": "3.2.6", - "resolved": "https://registry.npmjs.org/async/-/async-3.2.6.tgz", - "integrity": "sha512-htCUDlxyyCLMgaM3xXg0C0LW2xqfuQ6p05pCEIsXuyQ+a1koYKTuBMzRNwmybfLgvJDMd0r1LTn4+E0Ti6C2AA==", - "license": "MIT" - }, - "node_modules/asyncbox": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/asyncbox/-/asyncbox-3.0.0.tgz", - "integrity": "sha512-X7U0nedUMKV3nn9c4R0Zgvdvv6cw97tbDlHSZicq1snGPi/oX9DgGmFSURWtxDdnBWd3V0YviKhqAYAVvoWQ/A==", - "license": "Apache-2.0", - "dependencies": { - "bluebird": "^3.5.1", - "lodash": "^4.17.4", - "source-map-support": "^0.x" - }, - "engines": { - "node": ">=16" - } - }, - "node_modules/b4a": { - "version": "1.8.0", - "resolved": "https://registry.npmjs.org/b4a/-/b4a-1.8.0.tgz", - "integrity": "sha512-qRuSmNSkGQaHwNbM7J78Wwy+ghLEYF1zNrSeMxj4Kgw6y33O3mXcQ6Ie9fRvfU/YnxWkOchPXbaLb73TkIsfdg==", - "license": "Apache-2.0", - "peerDependencies": { - "react-native-b4a": "*" - }, - "peerDependenciesMeta": { - "react-native-b4a": { - "optional": true - } - } - }, "node_modules/balanced-match": { "version": "4.0.4", "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", @@ -822,97 +128,6 @@ "node": "18 || 20 || >=22" } }, - "node_modules/bare-events": { - "version": "2.8.2", - "resolved": "https://registry.npmjs.org/bare-events/-/bare-events-2.8.2.tgz", - "integrity": "sha512-riJjyv1/mHLIPX4RwiK+oW9/4c3TEUeORHKefKAKnZ5kyslbN+HXowtbaVEqt4IMUB7OXlfixcs6gsFeo/jhiQ==", - "license": "Apache-2.0", - "peerDependencies": { - "bare-abort-controller": "*" - }, - "peerDependenciesMeta": { - "bare-abort-controller": { - "optional": true - } - } - }, - "node_modules/bare-fs": { - "version": "4.7.0", - "resolved": "https://registry.npmjs.org/bare-fs/-/bare-fs-4.7.0.tgz", - "integrity": "sha512-xzqKsCFxAek9aezYhjJuJRXBIaYlg/0OGDTZp+T8eYmYMlm66cs6cYko02drIyjN2CBbi+I6L7YfXyqpqtKRXA==", - "license": "Apache-2.0", - "dependencies": { - "bare-events": "^2.5.4", - "bare-path": "^3.0.0", - "bare-stream": "^2.6.4", - "bare-url": "^2.2.2", - "fast-fifo": "^1.3.2" - }, - "engines": { - "bare": ">=1.16.0" - }, - "peerDependencies": { - "bare-buffer": "*" - }, - "peerDependenciesMeta": { - "bare-buffer": { - "optional": true - } - } - }, - "node_modules/bare-os": { - "version": "3.8.7", - "resolved": "https://registry.npmjs.org/bare-os/-/bare-os-3.8.7.tgz", - "integrity": "sha512-G4Gr1UsGeEy2qtDTZwL7JFLo2wapUarz7iTMcYcMFdS89AIQuBoyjgXZz0Utv7uHs3xA9LckhVbeBi8lEQrC+w==", - "license": "Apache-2.0", - "engines": { - "bare": ">=1.14.0" - } - }, - "node_modules/bare-path": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/bare-path/-/bare-path-3.0.0.tgz", - "integrity": "sha512-tyfW2cQcB5NN8Saijrhqn0Zh7AnFNsnczRcuWODH0eYAXBsJ5gVxAUuNr7tsHSC6IZ77cA0SitzT+s47kot8Mw==", - "license": "Apache-2.0", - "dependencies": { - "bare-os": "^3.0.1" - } - }, - "node_modules/bare-stream": { - "version": "2.13.0", - "resolved": "https://registry.npmjs.org/bare-stream/-/bare-stream-2.13.0.tgz", - "integrity": "sha512-3zAJRZMDFGjdn+RVnNpF9kuELw+0Fl3lpndM4NcEOhb9zwtSo/deETfuIwMSE5BXanA0FrN1qVjffGwAg2Y7EA==", - "license": "Apache-2.0", - "dependencies": { - "streamx": "^2.25.0", - "teex": "^1.0.1" - }, - "peerDependencies": { - "bare-abort-controller": "*", - "bare-buffer": "*", - "bare-events": "*" - }, - "peerDependenciesMeta": { - "bare-abort-controller": { - "optional": true - }, - "bare-buffer": { - "optional": true - }, - "bare-events": { - "optional": true - } - } - }, - "node_modules/bare-url": { - "version": "2.4.0", - "resolved": "https://registry.npmjs.org/bare-url/-/bare-url-2.4.0.tgz", - "integrity": "sha512-NSTU5WN+fy/L0DDenfE8SXQna4voXuW0FHM7wH8i3/q9khUSchfPbPezO4zSFMnDGIf9YE+mt/RWhZgNRKRIXA==", - "license": "Apache-2.0", - "dependencies": { - "bare-path": "^3.0.0" - } - }, "node_modules/base64-js": { "version": "1.5.1", "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", @@ -945,15 +160,6 @@ "node": ">=6.0.0" } }, - "node_modules/basic-ftp": { - "version": "5.2.2", - "resolved": "https://registry.npmjs.org/basic-ftp/-/basic-ftp-5.2.2.tgz", - "integrity": "sha512-1tDrzKsdCg70WGvbFss/ulVAxupNauGnOlgpyjKzeQxzyllBLS0CGLV7tjIXTK3ZQA9/FBEm9qyFFN1bciA6pw==", - "license": "MIT", - "engines": { - "node": ">=10.0.0" - } - }, "node_modules/better-sqlite3": { "version": "12.9.0", "resolved": "https://registry.npmjs.org/better-sqlite3/-/better-sqlite3-12.9.0.tgz", @@ -977,6 +183,12 @@ "file-uri-to-path": "1.0.0" } }, + "node_modules/bintrees": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/bintrees/-/bintrees-1.0.2.tgz", + "integrity": "sha512-VOMgTMwjAaUG580SXn3LacVgjurrbMme7ZZNYGSSV7mmtY6QQRh0Eg3pwIcntQ77DErK1L0NxkbetjcoXzVwKw==", + "license": "MIT" + }, "node_modules/bl": { "version": "4.1.0", "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz", @@ -988,12 +200,6 @@ "readable-stream": "^3.4.0" } }, - "node_modules/bluebird": { - "version": "3.7.2", - "resolved": "https://registry.npmjs.org/bluebird/-/bluebird-3.7.2.tgz", - "integrity": "sha512-XpNj6GDQzdfW+r2Wnn7xiSAd7TM3jzkxGXBGTtWKuSXv1xUV+azxAm8jdWZN06QTQk+2N2XB9jRDkvbmQmcRtg==", - "license": "MIT" - }, "node_modules/body-parser": { "version": "1.20.4", "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.4.tgz", @@ -1018,12 +224,6 @@ "npm": "1.2.8000 || >= 1.4.16" } }, - "node_modules/boolbase": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/boolbase/-/boolbase-1.0.0.tgz", - "integrity": "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==", - "license": "ISC" - }, "node_modules/brace-expansion": { "version": "5.0.5", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", @@ -1093,21 +293,6 @@ "ieee754": "^1.1.13" } }, - "node_modules/buffer-crc32": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-1.0.0.tgz", - "integrity": "sha512-Db1SbgBS/fg/392AblrMJk97KggmvYhr4pB5ZIMTWtaivCPMWLkmb7m21cJvpvgK+J3nsU2CmmixNBZx4vFj/w==", - "license": "MIT", - "engines": { - "node": ">=8.0.0" - } - }, - "node_modules/buffer-from": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz", - "integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==", - "license": "MIT" - }, "node_modules/bytes": { "version": "3.1.2", "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", @@ -1195,108 +380,19 @@ { "type": "tidelift", "url": "https://tidelift.com/funding/github/npm/caniuse-lite" - }, - { - "type": "github", - "url": "https://github.com/sponsors/ai" - } - ], - "license": "CC-BY-4.0" - }, - "node_modules/chalk": { - "version": "5.6.2", - "resolved": "https://registry.npmjs.org/chalk/-/chalk-5.6.2.tgz", - "integrity": "sha512-7NzBL0rN6fMUW+f7A6Io4h40qQlG+xGmtMxfbnH/K7TAtt8JQWVQK+6g0UXKMeVJoyV5EkkNsErQ8pVD3bLHbA==", - "license": "MIT", - "engines": { - "node": "^12.17.0 || ^14.13 || >=16.0.0" - }, - "funding": { - "url": "https://github.com/chalk/chalk?sponsor=1" - } - }, - "node_modules/cheerio": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/cheerio/-/cheerio-1.2.0.tgz", - "integrity": "sha512-WDrybc/gKFpTYQutKIK6UvfcuxijIZfMfXaYm8NMsPQxSYvf+13fXUJ4rztGGbJcBQ/GF55gvrZ0Bc0bj/mqvg==", - "license": "MIT", - "dependencies": { - "cheerio-select": "^2.1.0", - "dom-serializer": "^2.0.0", - "domhandler": "^5.0.3", - "domutils": "^3.2.2", - "encoding-sniffer": "^0.2.1", - "htmlparser2": "^10.1.0", - "parse5": "^7.3.0", - "parse5-htmlparser2-tree-adapter": "^7.1.0", - "parse5-parser-stream": "^7.1.2", - "undici": "^7.19.0", - "whatwg-mimetype": "^4.0.0" - }, - "engines": { - "node": ">=20.18.1" - }, - "funding": { - "url": "https://github.com/cheeriojs/cheerio?sponsor=1" - } - }, - "node_modules/cheerio-select": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/cheerio-select/-/cheerio-select-2.1.0.tgz", - "integrity": "sha512-9v9kG0LvzrlcungtnJtpGNxY+fzECQKhK4EGJX2vByejiMX84MFNQw4UxPJl3bFbTMw+Dfs37XaIkCwTZfLh4g==", - "license": "BSD-2-Clause", - "dependencies": { - "boolbase": "^1.0.0", - "css-select": "^5.1.0", - "css-what": "^6.1.0", - "domelementtype": "^2.3.0", - "domhandler": "^5.0.3", - "domutils": "^3.0.1" - }, - "funding": { - "url": "https://github.com/sponsors/fb55" - } - }, - "node_modules/chownr": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", - "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", - "license": "ISC" - }, - "node_modules/cliui": { - "version": "8.0.1", - "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", - "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==", - "license": "ISC", - "dependencies": { - "string-width": "^4.2.0", - "strip-ansi": "^6.0.1", - "wrap-ansi": "^7.0.0" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/cliui/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "CC-BY-4.0" }, - "node_modules/cliui/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } + "node_modules/chownr": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", + "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", + "license": "ISC" }, "node_modules/clone-deep": { "version": "0.2.4", @@ -1314,24 +410,6 @@ "node": ">=0.10.0" } }, - "node_modules/color-convert": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", - "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", - "license": "MIT", - "dependencies": { - "color-name": "~1.1.4" - }, - "engines": { - "node": ">=7.0.0" - } - }, - "node_modules/color-name": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", - "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", - "license": "MIT" - }, "node_modules/commander": { "version": "14.0.3", "resolved": "https://registry.npmjs.org/commander/-/commander-14.0.3.tgz", @@ -1341,74 +419,12 @@ "node": ">=20" } }, - "node_modules/compress-commons": { - "version": "6.0.2", - "resolved": "https://registry.npmjs.org/compress-commons/-/compress-commons-6.0.2.tgz", - "integrity": "sha512-6FqVXeETqWPoGcfzrXb37E50NP0LXT8kAMu5ooZayhWWdgEY4lBEEcbQNXtkuKQsGduxiIcI4gOTsxTmuq/bSg==", - "license": "MIT", - "dependencies": { - "crc-32": "^1.2.0", - "crc32-stream": "^6.0.0", - "is-stream": "^2.0.1", - "normalize-path": "^3.0.0", - "readable-stream": "^4.0.0" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/compress-commons/node_modules/buffer": { - "version": "6.0.3", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz", - "integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.2.1" - } - }, - "node_modules/compress-commons/node_modules/readable-stream": { - "version": "4.7.0", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz", - "integrity": "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg==", - "license": "MIT", - "dependencies": { - "abort-controller": "^3.0.0", - "buffer": "^6.0.3", - "events": "^3.3.0", - "process": "^0.11.10", - "string_decoder": "^1.3.0" - }, - "engines": { - "node": "^12.22.0 || ^14.17.0 || >=16.0.0" - } - }, "node_modules/concat-map": { "version": "0.0.1", "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", "integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==", "license": "MIT" }, - "node_modules/console-control-strings": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/console-control-strings/-/console-control-strings-1.1.0.tgz", - "integrity": "sha512-ty/fTekppD2fIwRvnZAVdeOiGd1c7YXEixbgJTNzqcxJWKQnjJ/V1bNEEE6hygpM3WjwHFUVK6HTjWSzV4a8sQ==", - "license": "ISC" - }, "node_modules/content-disposition": { "version": "0.5.4", "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz", @@ -1445,160 +461,6 @@ "integrity": "sha512-NXdYc3dLr47pBkpUCHtKSwIOQXLVn8dZEuywboCOJY/osA0wFSLlSawr3KN8qXJEyX66FcONTH8EIlVuK0yyFA==", "license": "MIT" }, - "node_modules/core-util-is": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.3.tgz", - "integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==", - "license": "MIT" - }, - "node_modules/crc-32": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/crc-32/-/crc-32-1.2.2.tgz", - "integrity": "sha512-ROmzCKrTnOwybPcJApAA6WBWij23HVfGVNKqqrZpuyZOHqK2CwHSvpGuyt/UNNvaIjEd8X5IFGp4Mh+Ie1IHJQ==", - "license": "Apache-2.0", - "bin": { - "crc32": "bin/crc32.njs" - }, - "engines": { - "node": ">=0.8" - } - }, - "node_modules/crc32-stream": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/crc32-stream/-/crc32-stream-6.0.0.tgz", - "integrity": "sha512-piICUB6ei4IlTv1+653yq5+KoqfBYmj9bw6LqXoOneTMDXk5nM1qt12mFW1caG3LlJXEKW1Bp0WggEmIfQB34g==", - "license": "MIT", - "dependencies": { - "crc-32": "^1.2.0", - "readable-stream": "^4.0.0" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/crc32-stream/node_modules/buffer": { - "version": "6.0.3", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz", - "integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.2.1" - } - }, - "node_modules/crc32-stream/node_modules/readable-stream": { - "version": "4.7.0", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz", - "integrity": "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg==", - "license": "MIT", - "dependencies": { - "abort-controller": "^3.0.0", - "buffer": "^6.0.3", - "events": "^3.3.0", - "process": "^0.11.10", - "string_decoder": "^1.3.0" - }, - "engines": { - "node": "^12.22.0 || ^14.17.0 || >=16.0.0" - } - }, - "node_modules/cross-spawn": { - "version": "7.0.6", - "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", - "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", - "license": "MIT", - "dependencies": { - "path-key": "^3.1.0", - "shebang-command": "^2.0.0", - "which": "^2.0.1" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/cross-spawn/node_modules/isexe": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", - "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", - "license": "ISC" - }, - "node_modules/cross-spawn/node_modules/which": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", - "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", - "license": "ISC", - "dependencies": { - "isexe": "^2.0.0" - }, - "bin": { - "node-which": "bin/node-which" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/css-select": { - "version": "5.2.2", - "resolved": "https://registry.npmjs.org/css-select/-/css-select-5.2.2.tgz", - "integrity": "sha512-TizTzUddG/xYLA3NXodFM0fSbNizXjOKhqiQQwvhlspadZokn1KDy0NZFS0wuEubIYAV5/c1/lAr0TaaFXEXzw==", - "license": "BSD-2-Clause", - "dependencies": { - "boolbase": "^1.0.0", - "css-what": "^6.1.0", - "domhandler": "^5.0.2", - "domutils": "^3.0.1", - "nth-check": "^2.0.1" - }, - "funding": { - "url": "https://github.com/sponsors/fb55" - } - }, - "node_modules/css-shorthand-properties": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/css-shorthand-properties/-/css-shorthand-properties-1.1.2.tgz", - "integrity": "sha512-C2AugXIpRGQTxaCW0N7n5jD/p5irUmCrwl03TrnMFBHDbdq44CFWR2zO7rK9xPN4Eo3pUxC4vQzQgbIpzrD1PQ==", - "license": "MIT" - }, - "node_modules/css-value": { - "version": "0.0.1", - "resolved": "https://registry.npmjs.org/css-value/-/css-value-0.0.1.tgz", - "integrity": "sha512-FUV3xaJ63buRLgHrLQVlVgQnQdR4yqdLGaDu7g8CQcWjInDfM9plBTPI9FRfpahju1UBSaMckeb2/46ApS/V1Q==" - }, - "node_modules/css-what": { - "version": "6.2.2", - "resolved": "https://registry.npmjs.org/css-what/-/css-what-6.2.2.tgz", - "integrity": "sha512-u/O3vwbptzhMs3L1fQE82ZSLHQQfto5gyZzwteVIEyeaY5Fc7R4dapF/BvRoSYFeqfBk4m0V1Vafq5Pjv25wvA==", - "license": "BSD-2-Clause", - "engines": { - "node": ">= 6" - }, - "funding": { - "url": "https://github.com/sponsors/fb55" - } - }, - "node_modules/data-uri-to-buffer": { - "version": "6.0.2", - "resolved": "https://registry.npmjs.org/data-uri-to-buffer/-/data-uri-to-buffer-6.0.2.tgz", - "integrity": "sha512-7hvf7/GW8e86rW0ptuwS3OcBGDjIi6SZva7hCyWC0yYry2cOPmLIjXAUHI6DK2HsnwJd9ifmt57i8eV2n4YNpw==", - "license": "MIT", - "engines": { - "node": ">= 14" - } - }, "node_modules/debug": { "version": "2.6.9", "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz", @@ -1608,18 +470,6 @@ "ms": "2.0.0" } }, - "node_modules/decamelize": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/decamelize/-/decamelize-6.0.1.tgz", - "integrity": "sha512-G7Cqgaelq68XHJNGlZ7lrNQyhZGsFqpwtGFexqUv4IQdjKoSYF7ipZ9UuTJZUSQXFj/XaoBLuEVIVqr8EJngEQ==", - "license": "MIT", - "engines": { - "node": "^12.20.0 || ^14.13.1 || >=16.0.0" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/decompress-response": { "version": "6.0.0", "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", @@ -1653,29 +503,6 @@ "node": ">=0.10.0" } }, - "node_modules/deepmerge-ts": { - "version": "7.1.5", - "resolved": "https://registry.npmjs.org/deepmerge-ts/-/deepmerge-ts-7.1.5.tgz", - "integrity": "sha512-HOJkrhaYsweh+W+e74Yn7YStZOilkoPb6fycpwNLKzSPtruFs48nYis0zy5yJz1+ktUhHxoRDJ27RQAWLIJVJw==", - "license": "BSD-3-Clause", - "engines": { - "node": ">=16.0.0" - } - }, - "node_modules/degenerator": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/degenerator/-/degenerator-5.0.1.tgz", - "integrity": "sha512-TllpMR/t0M5sqCXfj85i4XaAzxmS5tVA16dqvdkMwGmzI+dXLXnw3J+3Vdv7VKw+ThlTMboK6i9rnZ6Nntj5CQ==", - "license": "MIT", - "dependencies": { - "ast-types": "^0.13.4", - "escodegen": "^2.1.0", - "esprima": "^4.0.1" - }, - "engines": { - "node": ">= 14" - } - }, "node_modules/depd": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", @@ -1724,61 +551,6 @@ "node": ">=8" } }, - "node_modules/dom-serializer": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/dom-serializer/-/dom-serializer-2.0.0.tgz", - "integrity": "sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg==", - "license": "MIT", - "dependencies": { - "domelementtype": "^2.3.0", - "domhandler": "^5.0.2", - "entities": "^4.2.0" - }, - "funding": { - "url": "https://github.com/cheeriojs/dom-serializer?sponsor=1" - } - }, - "node_modules/domelementtype": { - "version": "2.3.0", - "resolved": "https://registry.npmjs.org/domelementtype/-/domelementtype-2.3.0.tgz", - "integrity": "sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/fb55" - } - ], - "license": "BSD-2-Clause" - }, - "node_modules/domhandler": { - "version": "5.0.3", - "resolved": "https://registry.npmjs.org/domhandler/-/domhandler-5.0.3.tgz", - "integrity": "sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w==", - "license": "BSD-2-Clause", - "dependencies": { - "domelementtype": "^2.3.0" - }, - "engines": { - "node": ">= 4" - }, - "funding": { - "url": "https://github.com/fb55/domhandler?sponsor=1" - } - }, - "node_modules/domutils": { - "version": "3.2.2", - "resolved": "https://registry.npmjs.org/domutils/-/domutils-3.2.2.tgz", - "integrity": "sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw==", - "license": "BSD-2-Clause", - "dependencies": { - "dom-serializer": "^2.0.0", - "domelementtype": "^2.3.0", - "domhandler": "^5.0.3" - }, - "funding": { - "url": "https://github.com/fb55/domutils?sponsor=1" - } - }, "node_modules/dot-prop": { "version": "6.0.1", "resolved": "https://registry.npmjs.org/dot-prop/-/dot-prop-6.0.1.tgz", @@ -1794,18 +566,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/dotenv": { - "version": "17.4.2", - "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz", - "integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://dotenvx.com" - } - }, "node_modules/dunder-proto": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", @@ -1820,96 +580,6 @@ "node": ">= 0.4" } }, - "node_modules/eastasianwidth": { - "version": "0.2.0", - "resolved": "https://registry.npmjs.org/eastasianwidth/-/eastasianwidth-0.2.0.tgz", - "integrity": "sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==", - "license": "MIT" - }, - "node_modules/edge-paths": { - "version": "3.0.5", - "resolved": "https://registry.npmjs.org/edge-paths/-/edge-paths-3.0.5.tgz", - "integrity": "sha512-sB7vSrDnFa4ezWQk9nZ/n0FdpdUuC6R1EOrlU3DL+bovcNFK28rqu2emmAUjujYEJTWIgQGqgVVWUZXMnc8iWg==", - "license": "MIT", - "dependencies": { - "@types/which": "^2.0.1", - "which": "^2.0.2" - }, - "engines": { - "node": ">=14.0.0" - }, - "funding": { - "url": "https://github.com/sponsors/shirshak55" - } - }, - "node_modules/edge-paths/node_modules/isexe": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", - "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", - "license": "ISC" - }, - "node_modules/edge-paths/node_modules/which": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", - "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", - "license": "ISC", - "dependencies": { - "isexe": "^2.0.0" - }, - "bin": { - "node-which": "bin/node-which" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/edgedriver": { - "version": "6.3.0", - "resolved": "https://registry.npmjs.org/edgedriver/-/edgedriver-6.3.0.tgz", - "integrity": "sha512-ggEQL+oEyIcM4nP2QC3AtCQ04o4kDNefRM3hja0odvlPSnsaxiruMxEZ93v3gDCKWYW6BXUr51PPradb+3nffw==", - "hasInstallScript": true, - "license": "MIT", - "dependencies": { - "@wdio/logger": "^9.18.0", - "@zip.js/zip.js": "^2.8.11", - "decamelize": "^6.0.1", - "edge-paths": "^3.0.5", - "fast-xml-parser": "^5.3.3", - "http-proxy-agent": "^7.0.2", - "https-proxy-agent": "^7.0.6", - "which": "^6.0.0" - }, - "bin": { - "edgedriver": "bin/edgedriver.js" - }, - "engines": { - "node": ">=20.0.0" - } - }, - "node_modules/edgedriver/node_modules/isexe": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/isexe/-/isexe-4.0.0.tgz", - "integrity": "sha512-FFUtZMpoZ8RqHS3XeXEmHWLA4thH+ZxCv2lOiPIn1Xc7CxrqhWzNSDzD+/chS/zbYezmiwWLdQC09JdQKmthOw==", - "license": "BlueOak-1.0.0", - "engines": { - "node": ">=20" - } - }, - "node_modules/edgedriver/node_modules/which": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/which/-/which-6.0.1.tgz", - "integrity": "sha512-oGLe46MIrCRqX7ytPUf66EAYvdeMIZYn3WaocqqKZAxrBpkqHfL/qvTyJ/bTk5+AqHCjXmrv3CEWgy368zhRUg==", - "license": "ISC", - "dependencies": { - "isexe": "^4.0.0" - }, - "bin": { - "node-which": "bin/which.js" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, "node_modules/ee-first": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", @@ -1922,44 +592,13 @@ "integrity": "sha512-q9n5T4BR4Xwa2cwbrwcsDJtHD/enpQ5S1xF1IAtdqf5AAgqDFmR/aakqH3ChFdqd/QXJhS3rnnXFtexU7rax6Q==", "license": "ISC" }, - "node_modules/emoji-regex": { - "version": "8.0.0", - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", - "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", - "license": "MIT" - }, "node_modules/encodeurl": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", "license": "MIT", "engines": { - "node": ">= 0.8" - } - }, - "node_modules/encoding-sniffer": { - "version": "0.2.1", - "resolved": "https://registry.npmjs.org/encoding-sniffer/-/encoding-sniffer-0.2.1.tgz", - "integrity": "sha512-5gvq20T6vfpekVtqrYQsSCFZ1wEg5+wW0/QaZMWkFr6BqD3NfKs0rLCx4rrVlSWJeZb5NBJgVLswK/w2MWU+Gw==", - "license": "MIT", - "dependencies": { - "iconv-lite": "^0.6.3", - "whatwg-encoding": "^3.1.1" - }, - "funding": { - "url": "https://github.com/fb55/encoding-sniffer?sponsor=1" - } - }, - "node_modules/encoding-sniffer/node_modules/iconv-lite": { - "version": "0.6.3", - "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz", - "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==", - "license": "MIT", - "dependencies": { - "safer-buffer": ">= 2.1.2 < 3.0.0" - }, - "engines": { - "node": ">=0.10.0" + "node": ">= 0.8" } }, "node_modules/end-of-stream": { @@ -1971,18 +610,6 @@ "once": "^1.4.0" } }, - "node_modules/entities": { - "version": "4.5.0", - "resolved": "https://registry.npmjs.org/entities/-/entities-4.5.0.tgz", - "integrity": "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=0.12" - }, - "funding": { - "url": "https://github.com/fb55/entities?sponsor=1" - } - }, "node_modules/es-define-property": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", @@ -2028,58 +655,6 @@ "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", "license": "MIT" }, - "node_modules/escodegen": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/escodegen/-/escodegen-2.1.0.tgz", - "integrity": "sha512-2NlIDTwUWJN0mRPQOdtQBzbUHvdGY2P1VXSyU83Q3xKxM7WHX2Ql8dKq782Q9TgQUNOLEzEYu9bzLNj1q88I5w==", - "license": "BSD-2-Clause", - "dependencies": { - "esprima": "^4.0.1", - "estraverse": "^5.2.0", - "esutils": "^2.0.2" - }, - "bin": { - "escodegen": "bin/escodegen.js", - "esgenerate": "bin/esgenerate.js" - }, - "engines": { - "node": ">=6.0" - }, - "optionalDependencies": { - "source-map": "~0.6.1" - } - }, - "node_modules/esprima": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/esprima/-/esprima-4.0.1.tgz", - "integrity": "sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A==", - "license": "BSD-2-Clause", - "bin": { - "esparse": "bin/esparse.js", - "esvalidate": "bin/esvalidate.js" - }, - "engines": { - "node": ">=4" - } - }, - "node_modules/estraverse": { - "version": "5.3.0", - "resolved": "https://registry.npmjs.org/estraverse/-/estraverse-5.3.0.tgz", - "integrity": "sha512-MMdARuVEQziNTeJD8DgMqmhwR11BRQ/cBP+pLtYdSTnf3MIO8fFeiINEbX36ZdNlfU/7A9f3gUw49B3oQsvwBA==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=4.0" - } - }, - "node_modules/esutils": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz", - "integrity": "sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=0.10.0" - } - }, "node_modules/etag": { "version": "1.8.1", "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", @@ -2089,33 +664,6 @@ "node": ">= 0.6" } }, - "node_modules/event-target-shim": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz", - "integrity": "sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ==", - "license": "MIT", - "engines": { - "node": ">=6" - } - }, - "node_modules/events": { - "version": "3.3.0", - "resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz", - "integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==", - "license": "MIT", - "engines": { - "node": ">=0.8.x" - } - }, - "node_modules/events-universal": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/events-universal/-/events-universal-1.0.1.tgz", - "integrity": "sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw==", - "license": "Apache-2.0", - "dependencies": { - "bare-events": "^2.7.0" - } - }, "node_modules/expand-template": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz", @@ -2171,105 +719,6 @@ "url": "https://opencollective.com/express" } }, - "node_modules/extract-zip": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/extract-zip/-/extract-zip-2.0.1.tgz", - "integrity": "sha512-GDhU9ntwuKyGXdZBUgTIe+vXnWj0fppUEtMDL0+idd5Sta8TGpHssn/eusA9mrPr9qNDym6SxAYZjNvCn/9RBg==", - "license": "BSD-2-Clause", - "dependencies": { - "debug": "^4.1.1", - "get-stream": "^5.1.0", - "yauzl": "^2.10.0" - }, - "bin": { - "extract-zip": "cli.js" - }, - "engines": { - "node": ">= 10.17.0" - }, - "optionalDependencies": { - "@types/yauzl": "^2.9.1" - } - }, - "node_modules/extract-zip/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/extract-zip/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/fast-deep-equal": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-2.0.1.tgz", - "integrity": "sha512-bCK/2Z4zLidyB4ReuIsvALH6w31YfAQDmXMqMx6FyfHqvBxtjC0eRumeSu4Bs3XtXwpyIywtSTrVT99BxY1f9w==", - "license": "MIT" - }, - "node_modules/fast-fifo": { - "version": "1.3.2", - "resolved": "https://registry.npmjs.org/fast-fifo/-/fast-fifo-1.3.2.tgz", - "integrity": "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==", - "license": "MIT" - }, - "node_modules/fast-xml-builder": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/fast-xml-builder/-/fast-xml-builder-1.1.4.tgz", - "integrity": "sha512-f2jhpN4Eccy0/Uz9csxh3Nu6q4ErKxf0XIsasomfOihuSUa3/xw6w8dnOtCDgEItQFJG8KyXPzQXzcODDrrbOg==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/NaturalIntelligence" - } - ], - "license": "MIT", - "dependencies": { - "path-expression-matcher": "^1.1.3" - } - }, - "node_modules/fast-xml-parser": { - "version": "5.5.11", - "resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-5.5.11.tgz", - "integrity": "sha512-QL0eb0YbSTVWF6tTf1+LEMSgtCEjBYPpnAjoLC8SscESlAjXEIRJ7cHtLG0pLeDFaZLa4VKZLArtA/60ZS7vyA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/NaturalIntelligence" - } - ], - "license": "MIT", - "dependencies": { - "fast-xml-builder": "^1.1.4", - "path-expression-matcher": "^1.4.0", - "strnum": "^2.2.3" - }, - "bin": { - "fxparser": "src/cli/cli.js" - } - }, - "node_modules/fd-slicer": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/fd-slicer/-/fd-slicer-1.1.0.tgz", - "integrity": "sha512-cE1qsB/VwyQozZ+q1dGxR8LBYNZeofhEdUNGSMbQD3Gw2lAzX9Zb3uIU6Ebc/Fmyjo9AWWfnn0AUCHqtevs/8g==", - "license": "MIT", - "dependencies": { - "pend": "~1.2.0" - } - }, "node_modules/file-uri-to-path": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz", @@ -2329,22 +778,6 @@ "node": ">=0.10.0" } }, - "node_modules/foreground-child": { - "version": "3.3.1", - "resolved": "https://registry.npmjs.org/foreground-child/-/foreground-child-3.3.1.tgz", - "integrity": "sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw==", - "license": "ISC", - "dependencies": { - "cross-spawn": "^7.0.6", - "signal-exit": "^4.0.1" - }, - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/forwarded": { "version": "0.2.0", "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", @@ -2412,27 +845,6 @@ "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/geckodriver": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/geckodriver/-/geckodriver-6.1.0.tgz", - "integrity": "sha512-ZRXLa4ZaYTTgUO4Eefw+RsQCleugU2QLb1ME7qTYxxuRj51yAhfnXaItXNs5/vUzfIaDHuZ+YnSF005hfp07nQ==", - "hasInstallScript": true, - "license": "MIT", - "dependencies": { - "@wdio/logger": "^9.18.0", - "@zip.js/zip.js": "^2.8.11", - "decamelize": "^6.0.1", - "http-proxy-agent": "^7.0.2", - "https-proxy-agent": "^7.0.6", - "modern-tar": "^0.7.2" - }, - "bin": { - "geckodriver": "bin/geckodriver.js" - }, - "engines": { - "node": ">=20.0.0" - } - }, "node_modules/generative-bayesian-network": { "version": "2.1.82", "resolved": "https://registry.npmjs.org/generative-bayesian-network/-/generative-bayesian-network-2.1.82.tgz", @@ -2443,15 +855,6 @@ "tslib": "^2.4.0" } }, - "node_modules/get-caller-file": { - "version": "2.0.5", - "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz", - "integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==", - "license": "ISC", - "engines": { - "node": "6.* || 8.* || >= 10.*" - } - }, "node_modules/get-intrinsic": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", @@ -2476,18 +879,6 @@ "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/get-port": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/get-port/-/get-port-7.2.0.tgz", - "integrity": "sha512-afP4W205ONCuMoPBqcR6PSXnzX35KTcJygfJfcp+QY+uwm3p20p1YczWXhlICIzGMCxYBQcySEcOgsJcrkyobg==", - "license": "MIT", - "engines": { - "node": ">=16" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/get-proto": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", @@ -2501,58 +892,6 @@ "node": ">= 0.4" } }, - "node_modules/get-stream": { - "version": "5.2.0", - "resolved": "https://registry.npmjs.org/get-stream/-/get-stream-5.2.0.tgz", - "integrity": "sha512-nBF+F1rAZVCu/p7rjzgA+Yb4lfYXrpl7a6VmJrU8wF9I1CKvP/QwPNZHnOlwbTkY6dvtFIzFMSyQXbLoTQPRpA==", - "license": "MIT", - "dependencies": { - "pump": "^3.0.0" - }, - "engines": { - "node": ">=8" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/get-uri": { - "version": "6.0.5", - "resolved": "https://registry.npmjs.org/get-uri/-/get-uri-6.0.5.tgz", - "integrity": "sha512-b1O07XYq8eRuVzBNgJLstU6FYc1tS6wnMtF1I1D9lE8LxZSOGZ7LhxN54yPP6mGw5f2CkXY2BQUL9Fx41qvcIg==", - "license": "MIT", - "dependencies": { - "basic-ftp": "^5.0.2", - "data-uri-to-buffer": "^6.0.2", - "debug": "^4.3.4" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/get-uri/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/get-uri/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, "node_modules/github-from-package": { "version": "0.0.0", "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", @@ -2594,21 +933,6 @@ "integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==", "license": "ISC" }, - "node_modules/grapheme-splitter": { - "version": "1.0.4", - "resolved": "https://registry.npmjs.org/grapheme-splitter/-/grapheme-splitter-1.0.4.tgz", - "integrity": "sha512-bzh50DW9kTPM00T8y4o8vQg89Di9oLJVLW/KaOGIXJWP/iqCN6WKYkbNOF04vFLJhwcpYUh9ydh/+5vpOqV4YQ==", - "license": "MIT" - }, - "node_modules/has-flag": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz", - "integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, "node_modules/has-symbols": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", @@ -2648,43 +972,6 @@ "node": ">=16.0.0" } }, - "node_modules/htmlfy": { - "version": "0.8.1", - "resolved": "https://registry.npmjs.org/htmlfy/-/htmlfy-0.8.1.tgz", - "integrity": "sha512-xWROBw9+MEGwxpotll0h672KCaLrKKiCYzsyN8ZgL9cQbVumFnyvsk2JqiB9ELAV1GLj1GG/jxZUjV9OZZi/yQ==", - "license": "MIT" - }, - "node_modules/htmlparser2": { - "version": "10.1.0", - "resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-10.1.0.tgz", - "integrity": "sha512-VTZkM9GWRAtEpveh7MSF6SjjrpNVNNVJfFup7xTY3UpFtm67foy9HDVXneLtFVt4pMz5kZtgNcvCniNFb1hlEQ==", - "funding": [ - "https://github.com/fb55/htmlparser2?sponsor=1", - { - "type": "github", - "url": "https://github.com/sponsors/fb55" - } - ], - "license": "MIT", - "dependencies": { - "domelementtype": "^2.3.0", - "domhandler": "^5.0.3", - "domutils": "^3.2.2", - "entities": "^7.0.1" - } - }, - "node_modules/htmlparser2/node_modules/entities": { - "version": "7.0.1", - "resolved": "https://registry.npmjs.org/entities/-/entities-7.0.1.tgz", - "integrity": "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=0.12" - }, - "funding": { - "url": "https://github.com/fb55/entities?sponsor=1" - } - }, "node_modules/http-errors": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", @@ -2700,83 +987,11 @@ "engines": { "node": ">= 0.8" }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/http-proxy-agent": { - "version": "7.0.2", - "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz", - "integrity": "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==", - "license": "MIT", - "dependencies": { - "agent-base": "^7.1.0", - "debug": "^4.3.4" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/http-proxy-agent/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/http-proxy-agent/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/https-proxy-agent": { - "version": "7.0.6", - "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz", - "integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==", - "license": "MIT", - "dependencies": { - "agent-base": "^7.1.2", - "debug": "4" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/https-proxy-agent/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" } }, - "node_modules/https-proxy-agent/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, "node_modules/iconv-lite": { "version": "0.4.24", "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz", @@ -2809,12 +1024,6 @@ ], "license": "BSD-3-Clause" }, - "node_modules/immediate": { - "version": "3.0.6", - "resolved": "https://registry.npmjs.org/immediate/-/immediate-3.0.6.tgz", - "integrity": "sha512-XXOFtyqDjNDAQxVfYxuF7g9Il/IbWmmlQg2MYKOH8ExIT1qg6xc4zyS3HaEEATgs1btfzxq15ciUiY7gjSXRGQ==", - "license": "MIT" - }, "node_modules/impit": { "version": "0.7.6", "resolved": "https://registry.npmjs.org/impit/-/impit-0.7.6.tgz", @@ -2962,16 +1171,6 @@ "node": ">= 10" } }, - "node_modules/import-meta-resolve": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/import-meta-resolve/-/import-meta-resolve-4.2.0.tgz", - "integrity": "sha512-Iqv2fzaTQN28s/FwZAoFq0ZSs/7hMAHJVX+w8PZl3cY19Pxk6jFFalxQoIfW2826i/fDLXv8IiEZRIT0lDuWcg==", - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, "node_modules/inflight": { "version": "1.0.6", "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz", @@ -2995,15 +1194,6 @@ "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", "license": "ISC" }, - "node_modules/ip-address": { - "version": "10.1.0", - "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.1.0.tgz", - "integrity": "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q==", - "license": "MIT", - "engines": { - "node": ">= 12" - } - }, "node_modules/ipaddr.js": { "version": "1.9.1", "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", @@ -3028,15 +1218,6 @@ "node": ">=0.10.0" } }, - "node_modules/is-fullwidth-code-point": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", - "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, "node_modules/is-obj": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/is-obj/-/is-obj-2.0.0.tgz", @@ -3046,18 +1227,6 @@ "node": ">=8" } }, - "node_modules/is-plain-obj": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-4.1.0.tgz", - "integrity": "sha512-+Pgi+vMuUNkJyExiMBt5IlFoMyKnr5zhJ4Uspz58WOhBF5QoIZkFyNHIbBAtHwzVAgk5RtndVNsDRN61/mmDqg==", - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/is-plain-object": { "version": "2.0.4", "resolved": "https://registry.npmjs.org/is-plain-object/-/is-plain-object-2.0.4.tgz", @@ -3090,33 +1259,6 @@ ], "license": "MIT" }, - "node_modules/is-stream": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/is-stream/-/is-stream-2.0.1.tgz", - "integrity": "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg==", - "license": "MIT", - "engines": { - "node": ">=8" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/isarray": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz", - "integrity": "sha512-VLghIWNM6ELQzo7zwmcg0NmTVyWKYjvIeM83yjp0wRDTmUnrM678fQbcKBo6n2CJEF0szoG//ytg+TKla89ALQ==", - "license": "MIT" - }, - "node_modules/isexe": { - "version": "3.1.5", - "resolved": "https://registry.npmjs.org/isexe/-/isexe-3.1.5.tgz", - "integrity": "sha512-6B3tLtFqtQS4ekarvLVMZ+X+VlvQekbe4taUkf/rhVO3d/h0M2rfARm/pXLcPEsjjMsFgrFgSrhQIxcSVrBz8w==", - "license": "BlueOak-1.0.0", - "engines": { - "node": ">=18" - } - }, "node_modules/isobject": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/isobject/-/isobject-3.0.1.tgz", @@ -3126,30 +1268,6 @@ "node": ">=0.10.0" } }, - "node_modules/jackspeak": { - "version": "3.4.3", - "resolved": "https://registry.npmjs.org/jackspeak/-/jackspeak-3.4.3.tgz", - "integrity": "sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw==", - "license": "BlueOak-1.0.0", - "dependencies": { - "@isaacs/cliui": "^8.0.2" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - }, - "optionalDependencies": { - "@pkgjs/parseargs": "^0.11.0" - } - }, - "node_modules/jiti": { - "version": "2.6.1", - "resolved": "https://registry.npmjs.org/jiti/-/jiti-2.6.1.tgz", - "integrity": "sha512-ekilCSN1jwRvIbgeg/57YFh8qQDNbwDb9xT/qu2DAHbFFZUicIl4ygVaAvzveMhMVr3LnpSKTNnwt8PoOfmKhQ==", - "license": "MIT", - "bin": { - "jiti": "lib/jiti-cli.mjs" - } - }, "node_modules/jsonfile": { "version": "6.2.0", "resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.2.0.tgz", @@ -3162,48 +1280,6 @@ "graceful-fs": "^4.1.6" } }, - "node_modules/jszip": { - "version": "3.10.1", - "resolved": "https://registry.npmjs.org/jszip/-/jszip-3.10.1.tgz", - "integrity": "sha512-xXDvecyTpGLrqFrvkrUSoxxfJI5AH7U8zxxtVclpsUtMCq4JQ290LY8AW5c7Ggnr/Y/oK+bQMbqK2qmtk3pN4g==", - "license": "(MIT OR GPL-3.0-or-later)", - "dependencies": { - "lie": "~3.3.0", - "pako": "~1.0.2", - "readable-stream": "~2.3.6", - "setimmediate": "^1.0.5" - } - }, - "node_modules/jszip/node_modules/readable-stream": { - "version": "2.3.8", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz", - "integrity": "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA==", - "license": "MIT", - "dependencies": { - "core-util-is": "~1.0.0", - "inherits": "~2.0.3", - "isarray": "~1.0.0", - "process-nextick-args": "~2.0.0", - "safe-buffer": "~5.1.1", - "string_decoder": "~1.1.1", - "util-deprecate": "~1.0.1" - } - }, - "node_modules/jszip/node_modules/safe-buffer": { - "version": "5.1.2", - "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz", - "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==", - "license": "MIT" - }, - "node_modules/jszip/node_modules/string_decoder": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz", - "integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==", - "license": "MIT", - "dependencies": { - "safe-buffer": "~5.1.0" - } - }, "node_modules/kind-of": { "version": "3.2.2", "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-3.2.2.tgz", @@ -3243,90 +1319,6 @@ "node": ">=0.10.0" } }, - "node_modules/lazystream": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/lazystream/-/lazystream-1.0.1.tgz", - "integrity": "sha512-b94GiNHQNy6JNTrt5w6zNyffMrNkXZb3KTkCZJb2V1xaEGCk093vkZ2jk3tpaeP33/OiXC+WvK9AxUebnf5nbw==", - "license": "MIT", - "dependencies": { - "readable-stream": "^2.0.5" - }, - "engines": { - "node": ">= 0.6.3" - } - }, - "node_modules/lazystream/node_modules/readable-stream": { - "version": "2.3.8", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz", - "integrity": "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA==", - "license": "MIT", - "dependencies": { - "core-util-is": "~1.0.0", - "inherits": "~2.0.3", - "isarray": "~1.0.0", - "process-nextick-args": "~2.0.0", - "safe-buffer": "~5.1.1", - "string_decoder": "~1.1.1", - "util-deprecate": "~1.0.1" - } - }, - "node_modules/lazystream/node_modules/safe-buffer": { - "version": "5.1.2", - "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz", - "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==", - "license": "MIT" - }, - "node_modules/lazystream/node_modules/string_decoder": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz", - "integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==", - "license": "MIT", - "dependencies": { - "safe-buffer": "~5.1.0" - } - }, - "node_modules/lie": { - "version": "3.3.0", - "resolved": "https://registry.npmjs.org/lie/-/lie-3.3.0.tgz", - "integrity": "sha512-UaiMJzeWRlEujzAuw5LokY1L5ecNQYZKfmyZ9L7wDHb/p5etKaxXhohBcrw0EYby+G/NA52vRSN4N39dxHAIwQ==", - "license": "MIT", - "dependencies": { - "immediate": "~3.0.5" - } - }, - "node_modules/locate-app": { - "version": "2.5.0", - "resolved": "https://registry.npmjs.org/locate-app/-/locate-app-2.5.0.tgz", - "integrity": "sha512-xIqbzPMBYArJRmPGUZD9CzV9wOqmVtQnaAn3wrj3s6WYW0bQvPI7x+sPYUGmDTYMHefVK//zc6HEYZ1qnxIK+Q==", - "funding": [ - { - "type": "individual", - "url": "https://buymeacoffee.com/hejny" - }, - { - "type": "github", - "url": "https://github.com/hejny/locate-app/blob/main/README.md#%EF%B8%8F-contributing" - } - ], - "license": "Apache-2.0", - "dependencies": { - "@promptbook/utils": "0.69.5", - "type-fest": "4.26.0", - "userhome": "1.0.1" - } - }, - "node_modules/lodash": { - "version": "4.18.1", - "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.18.1.tgz", - "integrity": "sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==", - "license": "MIT" - }, - "node_modules/lodash.clonedeep": { - "version": "4.5.0", - "resolved": "https://registry.npmjs.org/lodash.clonedeep/-/lodash.clonedeep-4.5.0.tgz", - "integrity": "sha512-H5ZhCF25riFd9uB5UCkVKo61m3S/xZk1x4wA6yp/L3RFP6Z/eHH1ymQcGLo7J3GMPfm0V/7m1tryHuGVxpqEBQ==", - "license": "MIT" - }, "node_modules/lodash.isequal": { "version": "4.5.0", "resolved": "https://registry.npmjs.org/lodash.isequal/-/lodash.isequal-4.5.0.tgz", @@ -3334,37 +1326,6 @@ "deprecated": "This package is deprecated. Use require('node:util').isDeepStrictEqual instead.", "license": "MIT" }, - "node_modules/lodash.zip": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/lodash.zip/-/lodash.zip-4.2.0.tgz", - "integrity": "sha512-C7IOaBBK/0gMORRBd8OETNx3kmOkgIWIPvyDpZSCTwUrpYmgZwJkjZeOD8ww4xbOUOs4/attY+pciKvadNfFbg==", - "license": "MIT" - }, - "node_modules/loglevel": { - "version": "1.9.2", - "resolved": "https://registry.npmjs.org/loglevel/-/loglevel-1.9.2.tgz", - "integrity": "sha512-HgMmCqIJSAKqo68l0rS2AanEWfkxaZ5wNiEFb5ggm08lDs9Xl2KxBlX3PTcaD2chBM1gXAYf491/M2Rv8Jwayg==", - "license": "MIT", - "engines": { - "node": ">= 0.6.0" - }, - "funding": { - "type": "tidelift", - "url": "https://tidelift.com/funding/github/npm/loglevel" - } - }, - "node_modules/loglevel-plugin-prefix": { - "version": "0.8.4", - "resolved": "https://registry.npmjs.org/loglevel-plugin-prefix/-/loglevel-plugin-prefix-0.8.4.tgz", - "integrity": "sha512-WpG9CcFAOjz/FtNht+QJeGpvVl/cdR6P0z6OcXSkr8wFJOsV2GRj2j10JLfjuA4aYkcKCNIEqRGCyTife9R8/g==", - "license": "MIT" - }, - "node_modules/lru-cache": { - "version": "10.4.3", - "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-10.4.3.tgz", - "integrity": "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ==", - "license": "ISC" - }, "node_modules/math-intrinsics": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", @@ -3507,12 +1468,6 @@ "node": ">=16 || 14 >=14.17" } }, - "node_modules/mitt": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/mitt/-/mitt-3.0.1.tgz", - "integrity": "sha512-vKivATfr97l2/QBCYAkXYDbrIWPM2IIKEl7YPhjCvKlG3kE2gm+uBo6nEXK3M5/Ffh/FLpKExzOQ3JJoJGFKBw==", - "license": "MIT" - }, "node_modules/mixin-object": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/mixin-object/-/mixin-object-2.0.1.tgz", @@ -3551,15 +1506,6 @@ "npm": ">=6" } }, - "node_modules/modern-tar": { - "version": "0.7.6", - "resolved": "https://registry.npmjs.org/modern-tar/-/modern-tar-0.7.6.tgz", - "integrity": "sha512-sweCIVXzx1aIGTCdzcMlSZt1h8k5Tmk08VNAuRk3IU28XamGiOH5ypi11g6De2CH7PhYqSSnGy2A/EFhbWnVKg==", - "license": "MIT", - "engines": { - "node": ">=18.0.0" - } - }, "node_modules/ms": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz", @@ -3578,245 +1524,77 @@ "integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==", "license": "MIT", "engines": { - "node": ">= 0.6" - } - }, - "node_modules/netmask": { - "version": "2.1.1", - "resolved": "https://registry.npmjs.org/netmask/-/netmask-2.1.1.tgz", - "integrity": "sha512-eonl3sLUha+S1GzTPxychyhnUzKyeQkZ7jLjKrBagJgPla13F+uQ71HgpFefyHgqrjEbCPkDArxYsjY8/+gLKA==", - "license": "MIT", - "engines": { - "node": ">= 0.4.0" - } - }, - "node_modules/node-abi": { - "version": "3.89.0", - "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.89.0.tgz", - "integrity": "sha512-6u9UwL0HlAl21+agMN3YAMXcKByMqwGx+pq+P76vii5f7hTPtKDp08/H9py6DY+cfDw7kQNTGEj/rly3IgbNQA==", - "license": "MIT", - "dependencies": { - "semver": "^7.3.5" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/node-releases": { - "version": "2.0.37", - "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.37.tgz", - "integrity": "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg==", - "license": "MIT" - }, - "node_modules/node-simctl": { - "version": "7.7.5", - "resolved": "https://registry.npmjs.org/node-simctl/-/node-simctl-7.7.5.tgz", - "integrity": "sha512-lWflzDW9xLuOOvR6mTJ9efbDtO/iSCH6rEGjxFxTV0vGgz5XjoZlW2BkNCCZib0B6Y23tCOiYhYJaMQYB8FKIQ==", - "license": "Apache-2.0", - "dependencies": { - "@appium/logger": "^1.3.0", - "asyncbox": "^3.0.0", - "bluebird": "^3.5.1", - "lodash": "^4.2.1", - "rimraf": "^5.0.0", - "semver": "^7.0.0", - "source-map-support": "^0.x", - "teen_process": "^2.2.0", - "uuid": "^11.0.1", - "which": "^5.0.0" - }, - "engines": { - "node": ">=14", - "npm": ">=8" - } - }, - "node_modules/normalize-path": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/normalize-path/-/normalize-path-3.0.0.tgz", - "integrity": "sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/nth-check": { - "version": "2.1.1", - "resolved": "https://registry.npmjs.org/nth-check/-/nth-check-2.1.1.tgz", - "integrity": "sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w==", - "license": "BSD-2-Clause", - "dependencies": { - "boolbase": "^1.0.0" - }, - "funding": { - "url": "https://github.com/fb55/nth-check?sponsor=1" - } - }, - "node_modules/object-inspect": { - "version": "1.13.4", - "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", - "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/on-finished": { - "version": "2.4.1", - "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", - "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", - "license": "MIT", - "dependencies": { - "ee-first": "1.1.1" - }, - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/once": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", - "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", - "license": "ISC", - "dependencies": { - "wrappy": "1" - } - }, - "node_modules/ow": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/ow/-/ow-0.28.2.tgz", - "integrity": "sha512-dD4UpyBh/9m4X2NVjA+73/ZPBRF+uF4zIMFvvQsabMiEK8x41L3rQ8EENOi35kyyoaJwNxEeJcP6Fj1H4U409Q==", - "license": "MIT", - "dependencies": { - "@sindresorhus/is": "^4.2.0", - "callsites": "^3.1.0", - "dot-prop": "^6.0.1", - "lodash.isequal": "^4.5.0", - "vali-date": "^1.0.0" - }, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/pac-proxy-agent": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/pac-proxy-agent/-/pac-proxy-agent-7.2.0.tgz", - "integrity": "sha512-TEB8ESquiLMc0lV8vcd5Ql/JAKAoyzHFXaStwjkzpOpC5Yv+pIzLfHvjTSdf3vpa2bMiUQrg9i6276yn8666aA==", - "license": "MIT", - "dependencies": { - "@tootallnate/quickjs-emscripten": "^0.23.0", - "agent-base": "^7.1.2", - "debug": "^4.3.4", - "get-uri": "^6.0.1", - "http-proxy-agent": "^7.0.0", - "https-proxy-agent": "^7.0.6", - "pac-resolver": "^7.0.1", - "socks-proxy-agent": "^8.0.5" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/pac-proxy-agent/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/pac-proxy-agent/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" + "node": ">= 0.6" + } }, - "node_modules/pac-resolver": { - "version": "7.0.1", - "resolved": "https://registry.npmjs.org/pac-resolver/-/pac-resolver-7.0.1.tgz", - "integrity": "sha512-5NPgf87AT2STgwa2ntRMr45jTKrYBGkVU36yT0ig/n/GMAa3oPqhZfIQ2kMEimReg0+t9kZViDVZ83qfVUlckg==", + "node_modules/node-abi": { + "version": "3.89.0", + "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.89.0.tgz", + "integrity": "sha512-6u9UwL0HlAl21+agMN3YAMXcKByMqwGx+pq+P76vii5f7hTPtKDp08/H9py6DY+cfDw7kQNTGEj/rly3IgbNQA==", "license": "MIT", "dependencies": { - "degenerator": "^5.0.0", - "netmask": "^2.0.2" + "semver": "^7.3.5" }, "engines": { - "node": ">= 14" + "node": ">=10" } }, - "node_modules/package-json-from-dist": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/package-json-from-dist/-/package-json-from-dist-1.0.1.tgz", - "integrity": "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==", - "license": "BlueOak-1.0.0" - }, - "node_modules/pako": { - "version": "1.0.11", - "resolved": "https://registry.npmjs.org/pako/-/pako-1.0.11.tgz", - "integrity": "sha512-4hLB8Py4zZce5s4yd9XzopqwVv/yGNhV1Bl8NTmCq1763HeK2+EwVTv+leGeL13Dnh2wfbqowVPXCIO0z4taYw==", - "license": "(MIT AND Zlib)" + "node_modules/node-releases": { + "version": "2.0.37", + "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.37.tgz", + "integrity": "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg==", + "license": "MIT" }, - "node_modules/parse5": { - "version": "7.3.0", - "resolved": "https://registry.npmjs.org/parse5/-/parse5-7.3.0.tgz", - "integrity": "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==", + "node_modules/object-inspect": { + "version": "1.13.4", + "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", + "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", "license": "MIT", - "dependencies": { - "entities": "^6.0.0" + "engines": { + "node": ">= 0.4" }, "funding": { - "url": "https://github.com/inikulin/parse5?sponsor=1" + "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/parse5-htmlparser2-tree-adapter": { - "version": "7.1.0", - "resolved": "https://registry.npmjs.org/parse5-htmlparser2-tree-adapter/-/parse5-htmlparser2-tree-adapter-7.1.0.tgz", - "integrity": "sha512-ruw5xyKs6lrpo9x9rCZqZZnIUntICjQAd0Wsmp396Ul9lN/h+ifgVV1x1gZHi8euej6wTfpqX8j+BFQxF0NS/g==", + "node_modules/on-finished": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", + "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", "license": "MIT", "dependencies": { - "domhandler": "^5.0.3", - "parse5": "^7.0.0" + "ee-first": "1.1.1" }, - "funding": { - "url": "https://github.com/inikulin/parse5?sponsor=1" + "engines": { + "node": ">= 0.8" } }, - "node_modules/parse5-parser-stream": { - "version": "7.1.2", - "resolved": "https://registry.npmjs.org/parse5-parser-stream/-/parse5-parser-stream-7.1.2.tgz", - "integrity": "sha512-JyeQc9iwFLn5TbvvqACIF/VXG6abODeB3Fwmv/TGdLk2LfbWkaySGY72at4+Ty7EkPZj854u4CrICqNk2qIbow==", - "license": "MIT", + "node_modules/once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", + "license": "ISC", "dependencies": { - "parse5": "^7.0.0" - }, - "funding": { - "url": "https://github.com/inikulin/parse5?sponsor=1" + "wrappy": "1" } }, - "node_modules/parse5/node_modules/entities": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz", - "integrity": "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==", - "license": "BSD-2-Clause", + "node_modules/ow": { + "version": "0.28.2", + "resolved": "https://registry.npmjs.org/ow/-/ow-0.28.2.tgz", + "integrity": "sha512-dD4UpyBh/9m4X2NVjA+73/ZPBRF+uF4zIMFvvQsabMiEK8x41L3rQ8EENOi35kyyoaJwNxEeJcP6Fj1H4U409Q==", + "license": "MIT", + "dependencies": { + "@sindresorhus/is": "^4.2.0", + "callsites": "^3.1.0", + "dot-prop": "^6.0.1", + "lodash.isequal": "^4.5.0", + "vali-date": "^1.0.0" + }, "engines": { - "node": ">=0.12" + "node": ">=12" }, "funding": { - "url": "https://github.com/fb55/entities?sponsor=1" + "url": "https://github.com/sponsors/sindresorhus" } }, "node_modules/parseurl": { @@ -3828,21 +1606,6 @@ "node": ">= 0.8" } }, - "node_modules/path-expression-matcher": { - "version": "1.5.0", - "resolved": "https://registry.npmjs.org/path-expression-matcher/-/path-expression-matcher-1.5.0.tgz", - "integrity": "sha512-cbrerZV+6rvdQrrD+iGMcZFEiiSrbv9Tfdkvnusy6y0x0GKBXREFg/Y65GhIfm0tnLntThhzCnfKwp1WRjeCyQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/NaturalIntelligence" - } - ], - "license": "MIT", - "engines": { - "node": ">=14.0.0" - } - }, "node_modules/path-is-absolute": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz", @@ -3852,15 +1615,6 @@ "node": ">=0.10.0" } }, - "node_modules/path-key": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", - "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, "node_modules/path-scurry": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-2.0.2.tgz", @@ -3892,12 +1646,6 @@ "integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==", "license": "MIT" }, - "node_modules/pend": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz", - "integrity": "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==", - "license": "MIT" - }, "node_modules/picocolors": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", @@ -4008,21 +1756,6 @@ "node": ">=10" } }, - "node_modules/process": { - "version": "0.11.10", - "resolved": "https://registry.npmjs.org/process/-/process-0.11.10.tgz", - "integrity": "sha512-cdGef/drWFoydD1JsMzuFf8100nZl+GT+yacc2bEced5f9Rjk4z+WtFUTBu9PhOi9j/jfmBPu0mMEY4wIdAF8A==", - "license": "MIT", - "engines": { - "node": ">= 0.6.0" - } - }, - "node_modules/process-nextick-args": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.1.tgz", - "integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==", - "license": "MIT" - }, "node_modules/progress": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/progress/-/progress-2.0.3.tgz", @@ -4032,6 +1765,19 @@ "node": ">=0.4.0" } }, + "node_modules/prom-client": { + "version": "15.1.3", + "resolved": "https://registry.npmjs.org/prom-client/-/prom-client-15.1.3.tgz", + "integrity": "sha512-6ZiOBfCywsD4k1BN9IX0uZhF+tJkV8q8llP64G5Hajs4JOeVLPCwpPVcpXy3BwYiUGgyJzsJJQeOIv7+hDSq8g==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/api": "^1.4.0", + "tdigest": "^0.1.1" + }, + "engines": { + "node": "^16 || ^18 || >=20" + } + }, "node_modules/proxy-addr": { "version": "2.0.7", "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", @@ -4045,63 +1791,6 @@ "node": ">= 0.10" } }, - "node_modules/proxy-agent": { - "version": "6.5.0", - "resolved": "https://registry.npmjs.org/proxy-agent/-/proxy-agent-6.5.0.tgz", - "integrity": "sha512-TmatMXdr2KlRiA2CyDu8GqR8EjahTG3aY3nXjdzFyoZbmB8hrBsTyMezhULIXKnC0jpfjlmiZ3+EaCzoInSu/A==", - "license": "MIT", - "dependencies": { - "agent-base": "^7.1.2", - "debug": "^4.3.4", - "http-proxy-agent": "^7.0.1", - "https-proxy-agent": "^7.0.6", - "lru-cache": "^7.14.1", - "pac-proxy-agent": "^7.1.0", - "proxy-from-env": "^1.1.0", - "socks-proxy-agent": "^8.0.5" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/proxy-agent/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/proxy-agent/node_modules/lru-cache": { - "version": "7.18.3", - "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-7.18.3.tgz", - "integrity": "sha512-jumlc0BIUrS3qJGgIkWZsyfAM7NCWiBcCDhnd+3NNM5KbBmLTgHVfWBcg6W+rLUsIpzpERPsvwUP7CckAQSOoA==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/proxy-agent/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/proxy-from-env": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz", - "integrity": "sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg==", - "license": "MIT" - }, "node_modules/pump": { "version": "3.0.4", "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz", @@ -4378,235 +2067,69 @@ "node_modules/qs": { "version": "6.14.2", "resolved": "https://registry.npmjs.org/qs/-/qs-6.14.2.tgz", - "integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==", - "license": "BSD-3-Clause", - "dependencies": { - "side-channel": "^1.1.0" - }, - "engines": { - "node": ">=0.6" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/query-selector-shadow-dom": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/query-selector-shadow-dom/-/query-selector-shadow-dom-1.0.1.tgz", - "integrity": "sha512-lT5yCqEBgfoMYpf3F2xQRK7zEr1rhIIZuceDK6+xRkJQ4NMbHTwXqk4NkwDwQMNqXgG9r9fyHnzwNVs6zV5KRw==", - "license": "MIT" - }, - "node_modules/range-parser": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", - "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/raw-body": { - "version": "2.5.3", - "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.3.tgz", - "integrity": "sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==", - "license": "MIT", - "dependencies": { - "bytes": "~3.1.2", - "http-errors": "~2.0.1", - "iconv-lite": "~0.4.24", - "unpipe": "~1.0.0" - }, - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/rc": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", - "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", - "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", - "dependencies": { - "deep-extend": "^0.6.0", - "ini": "~1.3.0", - "minimist": "^1.2.0", - "strip-json-comments": "~2.0.1" - }, - "bin": { - "rc": "cli.js" - } - }, - "node_modules/readable-stream": { - "version": "3.6.2", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", - "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", - "license": "MIT", - "dependencies": { - "inherits": "^2.0.3", - "string_decoder": "^1.1.1", - "util-deprecate": "^1.0.1" - }, - "engines": { - "node": ">= 6" - } - }, - "node_modules/readdir-glob": { - "version": "1.1.3", - "resolved": "https://registry.npmjs.org/readdir-glob/-/readdir-glob-1.1.3.tgz", - "integrity": "sha512-v05I2k7xN8zXvPD9N+z/uhXPaj0sUFCe2rcWZIpBsqxfP7xXFQ0tipAd/wjj1YxWyWtUS5IDJpOG82JKt2EAVA==", - "license": "Apache-2.0", - "dependencies": { - "minimatch": "^5.1.0" - } - }, - "node_modules/readdir-glob/node_modules/balanced-match": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", - "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", - "license": "MIT" - }, - "node_modules/readdir-glob/node_modules/brace-expansion": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.0.tgz", - "integrity": "sha512-TN1kCZAgdgweJhWWpgKYrQaMNHcDULHkWwQIspdtjV4Y5aurRdZpjAqn6yX3FPqTA9ngHCc4hJxMAMgGfve85w==", - "license": "MIT", - "dependencies": { - "balanced-match": "^1.0.0" - } - }, - "node_modules/readdir-glob/node_modules/minimatch": { - "version": "5.1.9", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-5.1.9.tgz", - "integrity": "sha512-7o1wEA2RyMP7Iu7GNba9vc0RWWGACJOCZBJX2GJWip0ikV+wcOsgVuY9uE8CPiyQhkGFSlhuSkZPavN7u1c2Fw==", - "license": "ISC", - "dependencies": { - "brace-expansion": "^2.0.1" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/require-directory": { - "version": "2.1.1", - "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz", - "integrity": "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/resq": { - "version": "1.11.0", - "resolved": "https://registry.npmjs.org/resq/-/resq-1.11.0.tgz", - "integrity": "sha512-G10EBz+zAAy3zUd/CDoBbXRL6ia9kOo3xRHrMDsHljI0GDkhYlyjwoCx5+3eCC4swi1uCoZQhskuJkj7Gp57Bw==", - "license": "MIT", - "dependencies": { - "fast-deep-equal": "^2.0.1" - } - }, - "node_modules/ret": { - "version": "0.5.0", - "resolved": "https://registry.npmjs.org/ret/-/ret-0.5.0.tgz", - "integrity": "sha512-I1XxrZSQ+oErkRR4jYbAyEEu2I0avBvvMM5JN+6EBprOGRCs63ENqZ3vjavq8fBw2+62G5LF5XelKwuJpcvcxw==", - "license": "MIT", - "engines": { - "node": ">=10" - } - }, - "node_modules/rgb2hex": { - "version": "0.2.5", - "resolved": "https://registry.npmjs.org/rgb2hex/-/rgb2hex-0.2.5.tgz", - "integrity": "sha512-22MOP1Rh7sAo1BZpDG6R5RFYzR2lYEgwq7HEmyW2qcsOqR2lQKmn+O//xV3YG/0rrhMC6KVX2hU+ZXuaw9a5bw==", - "license": "MIT" - }, - "node_modules/rimraf": { - "version": "5.0.10", - "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-5.0.10.tgz", - "integrity": "sha512-l0OE8wL34P4nJH/H2ffoaniAokM2qSmrtXHmlpvYr5AVVX8msAyW0l8NVJFDxlSK4u3Uh/f41cQheDVdnYijwQ==", - "license": "ISC", - "dependencies": { - "glob": "^10.3.7" - }, - "bin": { - "rimraf": "dist/esm/bin.mjs" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/rimraf/node_modules/balanced-match": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", - "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", - "license": "MIT" - }, - "node_modules/rimraf/node_modules/brace-expansion": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.0.tgz", - "integrity": "sha512-TN1kCZAgdgweJhWWpgKYrQaMNHcDULHkWwQIspdtjV4Y5aurRdZpjAqn6yX3FPqTA9ngHCc4hJxMAMgGfve85w==", - "license": "MIT", - "dependencies": { - "balanced-match": "^1.0.0" - } - }, - "node_modules/rimraf/node_modules/glob": { - "version": "10.5.0", - "resolved": "https://registry.npmjs.org/glob/-/glob-10.5.0.tgz", - "integrity": "sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==", - "deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me", - "license": "ISC", - "dependencies": { - "foreground-child": "^3.1.0", - "jackspeak": "^3.1.2", - "minimatch": "^9.0.4", - "minipass": "^7.1.2", - "package-json-from-dist": "^1.0.0", - "path-scurry": "^1.11.1" - }, - "bin": { - "glob": "dist/esm/bin.mjs" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/rimraf/node_modules/minimatch": { - "version": "9.0.9", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.9.tgz", - "integrity": "sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg==", - "license": "ISC", + "integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==", + "license": "BSD-3-Clause", "dependencies": { - "brace-expansion": "^2.0.2" + "side-channel": "^1.1.0" }, "engines": { - "node": ">=16 || 14 >=14.17" + "node": ">=0.6" }, "funding": { - "url": "https://github.com/sponsors/isaacs" + "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/rimraf/node_modules/path-scurry": { - "version": "1.11.1", - "resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-1.11.1.tgz", - "integrity": "sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA==", - "license": "BlueOak-1.0.0", + "node_modules/range-parser": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", + "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/raw-body": { + "version": "2.5.3", + "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.3.tgz", + "integrity": "sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==", + "license": "MIT", "dependencies": { - "lru-cache": "^10.2.0", - "minipass": "^5.0.0 || ^6.0.2 || ^7.0.0" + "bytes": "~3.1.2", + "http-errors": "~2.0.1", + "iconv-lite": "~0.4.24", + "unpipe": "~1.0.0" }, "engines": { - "node": ">=16 || 14 >=14.18" + "node": ">= 0.8" + } + }, + "node_modules/rc": { + "version": "1.2.8", + "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", + "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", + "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", + "dependencies": { + "deep-extend": "^0.6.0", + "ini": "~1.3.0", + "minimist": "^1.2.0", + "strip-json-comments": "~2.0.1" }, - "funding": { - "url": "https://github.com/sponsors/isaacs" + "bin": { + "rc": "cli.js" } }, - "node_modules/safaridriver": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/safaridriver/-/safaridriver-1.0.1.tgz", - "integrity": "sha512-jkg4434cYgtrIF2AeY/X0Wmd2W73cK5qIEFE3hDrrQenJH/2SDJIXGvPAigfvQTcE9+H31zkiNHbUqcihEiMRA==", + "node_modules/readable-stream": { + "version": "3.6.2", + "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", + "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", "license": "MIT", + "dependencies": { + "inherits": "^2.0.3", + "string_decoder": "^1.1.1", + "util-deprecate": "^1.0.1" + }, "engines": { - "node": ">=18.0.0" + "node": ">= 6" } }, "node_modules/safe-buffer": { @@ -4629,28 +2152,6 @@ ], "license": "MIT" }, - "node_modules/safe-regex2": { - "version": "5.1.0", - "resolved": "https://registry.npmjs.org/safe-regex2/-/safe-regex2-5.1.0.tgz", - "integrity": "sha512-pNHAuBW7TrcleFHsxBr5QMi/Iyp0ENjUKz7GCcX1UO7cMh+NmVK6HxQckNL1tJp1XAJVjG6B8OKIPqodqj9rtw==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/fastify" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/fastify" - } - ], - "license": "MIT", - "dependencies": { - "ret": "~0.5.0" - }, - "bin": { - "safe-regex2": "bin/safe-regex2.js" - } - }, "node_modules/safer-buffer": { "version": "2.1.2", "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", @@ -4708,33 +2209,6 @@ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "license": "MIT" }, - "node_modules/serialize-error": { - "version": "12.0.0", - "resolved": "https://registry.npmjs.org/serialize-error/-/serialize-error-12.0.0.tgz", - "integrity": "sha512-ZYkZLAvKTKQXWuh5XpBw7CdbSzagarX39WyZ2H07CDLC5/KfsRGlIXV8d4+tfqX1M7916mRqR1QfNHSij+c9Pw==", - "license": "MIT", - "dependencies": { - "type-fest": "^4.31.0" - }, - "engines": { - "node": ">=18" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/serialize-error/node_modules/type-fest": { - "version": "4.41.0", - "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-4.41.0.tgz", - "integrity": "sha512-TeTSQ6H5YHvpqVwBRcnLDCBnDOHWYu7IvGbHT6N8AOymcr9PJGjc1GTtiWZTYg0NCgYwvnYWEkVChQAr9bjfwA==", - "license": "(MIT OR CC0-1.0)", - "engines": { - "node": ">=16" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/serve-static": { "version": "1.16.3", "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.3.tgz", @@ -4750,18 +2224,6 @@ "node": ">= 0.8.0" } }, - "node_modules/set-blocking": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/set-blocking/-/set-blocking-2.0.0.tgz", - "integrity": "sha512-KiKBS8AnWGEyLzofFfmvKwpdPzqiy16LvQfK3yv/fVH7Bj13/wl3JSR1J+rfgRE9q7xUJK4qvgS8raSOeLUehw==", - "license": "ISC" - }, - "node_modules/setimmediate": { - "version": "1.0.5", - "resolved": "https://registry.npmjs.org/setimmediate/-/setimmediate-1.0.5.tgz", - "integrity": "sha512-MATJdZp8sLqDl/68LfQmbP8zKPLQNV6BIZoIgrscFDQ+RsvK/BxeDQOgyxKKoh0y/8h3BqVFnCqQ/gd+reiIXA==", - "license": "MIT" - }, "node_modules/setprototypeof": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", @@ -4804,39 +2266,6 @@ "node": ">=0.10.0" } }, - "node_modules/shebang-command": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", - "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", - "license": "MIT", - "dependencies": { - "shebang-regex": "^3.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/shebang-regex": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", - "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/shell-quote": { - "version": "1.8.3", - "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.3.tgz", - "integrity": "sha512-ObmnIF4hXNg1BqhnHmgbDETF8dLPCggZWBjkQfhZpbszZnYur5DUljTcCHii5LC3J5E0yeO/1LIMyH+UvHQgyw==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, "node_modules/side-channel": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz", @@ -4909,18 +2338,6 @@ "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/signal-exit": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz", - "integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==", - "license": "ISC", - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/simple-concat": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz", @@ -4966,111 +2383,6 @@ "simple-concat": "^1.0.0" } }, - "node_modules/smart-buffer": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/smart-buffer/-/smart-buffer-4.2.0.tgz", - "integrity": "sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg==", - "license": "MIT", - "engines": { - "node": ">= 6.0.0", - "npm": ">= 3.0.0" - } - }, - "node_modules/socks": { - "version": "2.8.7", - "resolved": "https://registry.npmjs.org/socks/-/socks-2.8.7.tgz", - "integrity": "sha512-HLpt+uLy/pxB+bum/9DzAgiKS8CX1EvbWxI4zlmgGCExImLdiad2iCwXT5Z4c9c3Eq8rP2318mPW2c+QbtjK8A==", - "license": "MIT", - "dependencies": { - "ip-address": "^10.0.1", - "smart-buffer": "^4.2.0" - }, - "engines": { - "node": ">= 10.0.0", - "npm": ">= 3.0.0" - } - }, - "node_modules/socks-proxy-agent": { - "version": "8.0.5", - "resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-8.0.5.tgz", - "integrity": "sha512-HehCEsotFqbPW9sJ8WVYB6UbmIMv7kUUORIF2Nncq4VQvBfNBLibW9YZR5dlYCSUhwcD628pRllm7n+E+YTzJw==", - "license": "MIT", - "dependencies": { - "agent-base": "^7.1.2", - "debug": "^4.3.4", - "socks": "^2.8.3" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/socks-proxy-agent/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/socks-proxy-agent/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/source-map": { - "version": "0.6.1", - "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz", - "integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==", - "license": "BSD-3-Clause", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/source-map-support": { - "version": "0.5.21", - "resolved": "https://registry.npmjs.org/source-map-support/-/source-map-support-0.5.21.tgz", - "integrity": "sha512-uBHU3L3czsIyYXKX88fdrGovxdSCoTGDRZ6SYXtSRxLZUzHg5P/66Ht6uoUlHu9EZod+inXhKo3qQgwXUT/y1w==", - "license": "MIT", - "dependencies": { - "buffer-from": "^1.0.0", - "source-map": "^0.6.0" - } - }, - "node_modules/spacetrim": { - "version": "0.11.59", - "resolved": "https://registry.npmjs.org/spacetrim/-/spacetrim-0.11.59.tgz", - "integrity": "sha512-lLYsktklSRKprreOm7NXReW8YiX2VBjbgmXYEziOoGf/qsJqAEACaDvoTtUOycwjpaSh+bT8eu0KrJn7UNxiCg==", - "funding": [ - { - "type": "individual", - "url": "https://buymeacoffee.com/hejny" - }, - { - "type": "github", - "url": "https://github.com/hejny/spacetrim/blob/main/README.md#%EF%B8%8F-contributing" - } - ], - "license": "Apache-2.0" - }, - "node_modules/split2": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz", - "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==", - "license": "ISC", - "engines": { - "node": ">= 10.x" - } - }, "node_modules/statuses": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", @@ -5080,17 +2392,6 @@ "node": ">= 0.8" } }, - "node_modules/streamx": { - "version": "2.25.0", - "resolved": "https://registry.npmjs.org/streamx/-/streamx-2.25.0.tgz", - "integrity": "sha512-0nQuG6jf1w+wddNEEXCF4nTg3LtufWINB5eFEN+5TNZW7KWJp6x87+JFL43vaAUPyCfH1wID+mNVyW6OHtFamg==", - "license": "MIT", - "dependencies": { - "events-universal": "^1.0.0", - "fast-fifo": "^1.3.2", - "text-decoder": "^1.1.0" - } - }, "node_modules/string_decoder": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", @@ -5100,145 +2401,13 @@ "safe-buffer": "~5.2.0" } }, - "node_modules/string-width": { - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", - "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", - "license": "MIT", - "dependencies": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/string-width-cjs": { - "name": "string-width", - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", - "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", - "license": "MIT", - "dependencies": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/string-width-cjs/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/string-width-cjs/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/string-width/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/string-width/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/strip-ansi": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-7.2.0.tgz", - "integrity": "sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^6.2.2" - }, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/chalk/strip-ansi?sponsor=1" - } - }, - "node_modules/strip-ansi-cjs": { - "name": "strip-ansi", - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/strip-ansi-cjs/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, "node_modules/strip-json-comments": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==", "license": "MIT", "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/strnum": { - "version": "2.2.3", - "resolved": "https://registry.npmjs.org/strnum/-/strnum-2.2.3.tgz", - "integrity": "sha512-oKx6RUCuHfT3oyVjtnrmn19H1SiCqgJSg+54XqURKp5aCMbrXrhLjRN9TjuwMjiYstZ0MzDrHqkGZ5dFTKd+zg==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/NaturalIntelligence" - } - ], - "license": "MIT" - }, - "node_modules/supports-color": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz", - "integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==", - "license": "MIT", - "dependencies": { - "has-flag": "^4.0.0" - }, - "engines": { - "node": ">=8" + "node": ">=0.10.0" } }, "node_modules/tar-fs": { @@ -5269,38 +2438,13 @@ "node": ">=6" } }, - "node_modules/teen_process": { - "version": "2.3.3", - "resolved": "https://registry.npmjs.org/teen_process/-/teen_process-2.3.3.tgz", - "integrity": "sha512-NIdeetf/6gyEqLjnzvfgQe7PfipSceq2xDQM2Py2BkBnIIeWh3HRD3vNhulyO5WppfCv9z4mtsEHyq8kdiULTA==", - "license": "Apache-2.0", - "dependencies": { - "bluebird": "^3.7.2", - "lodash": "^4.17.21", - "shell-quote": "^1.8.1", - "source-map-support": "^0.x" - }, - "engines": { - "node": "^16.13.0 || >=18.0.0", - "npm": ">=8" - } - }, - "node_modules/teex": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/teex/-/teex-1.0.1.tgz", - "integrity": "sha512-eYE6iEI62Ni1H8oIa7KlDU6uQBtqr4Eajni3wX7rpfXD8ysFx8z0+dri+KWEPWpBsxXfxu58x/0jvTVT1ekOSg==", + "node_modules/tdigest": { + "version": "0.1.2", + "resolved": "https://registry.npmjs.org/tdigest/-/tdigest-0.1.2.tgz", + "integrity": "sha512-+G0LLgjjo9BZX2MfdvPfH+MKLCrxlXSYec5DaPYP1fe6Iyhf0/fSmJ0bFiZ1F8BT6cGXl2LpltQptzjXKWEkKA==", "license": "MIT", "dependencies": { - "streamx": "^2.12.5" - } - }, - "node_modules/text-decoder": { - "version": "1.2.7", - "resolved": "https://registry.npmjs.org/text-decoder/-/text-decoder-1.2.7.tgz", - "integrity": "sha512-vlLytXkeP4xvEq2otHeJfSQIRyWxo/oZGEbXrtEEF9Hnmrdly59sUbzZ/QgyWuLYHctCHxFF4tRQZNQ9k60ExQ==", - "license": "Apache-2.0", - "dependencies": { - "b4a": "^1.6.4" + "bintrees": "1.0.2" } }, "node_modules/tiny-lru": { @@ -5339,18 +2483,6 @@ "node": "*" } }, - "node_modules/type-fest": { - "version": "4.26.0", - "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-4.26.0.tgz", - "integrity": "sha512-OduNjVJsFbifKb57UqZ2EMP1i4u64Xwow3NYXUtBbD4vIwJdQd4+xl8YDou1dlm4DVrtwT/7Ky8z8WyCULVfxw==", - "license": "(MIT OR CC0-1.0)", - "engines": { - "node": ">=16" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/type-is": { "version": "1.6.18", "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz", @@ -5415,21 +2547,6 @@ "node": "*" } }, - "node_modules/undici": { - "version": "7.24.8", - "resolved": "https://registry.npmjs.org/undici/-/undici-7.24.8.tgz", - "integrity": "sha512-6KQ/+QxK49Z/p3HO6E5ZCZWNnCasyZLa5ExaVYyvPxUwKtbCPMKELJOqh7EqOle0t9cH/7d2TaaTRRa6Nhs4YQ==", - "license": "MIT", - "engines": { - "node": ">=20.18.1" - } - }, - "node_modules/undici-types": { - "version": "6.21.0", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", - "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", - "license": "MIT" - }, "node_modules/universalify": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/universalify/-/universalify-2.0.1.tgz", @@ -5478,21 +2595,6 @@ "browserslist": ">= 4.21.0" } }, - "node_modules/urlpattern-polyfill": { - "version": "10.1.0", - "resolved": "https://registry.npmjs.org/urlpattern-polyfill/-/urlpattern-polyfill-10.1.0.tgz", - "integrity": "sha512-IGjKp/o0NL3Bso1PymYURCJxMPNAf/ILOpendP9f5B6e1rTJgdgiOvgfoT8VxCAdY+Wisb9uhGaJJf3yZ2V9nw==", - "license": "MIT" - }, - "node_modules/userhome": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/userhome/-/userhome-1.0.1.tgz", - "integrity": "sha512-5cnLm4gseXjAclKowC4IjByaGsjtAoV6PrOQOljplNB54ReUYJP8HdAFq2muHinSDAh09PPX/uXDPfdxRHvuSA==", - "license": "MIT", - "engines": { - "node": ">= 0.8.0" - } - }, "node_modules/util-deprecate": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", @@ -5508,19 +2610,6 @@ "node": ">= 0.4.0" } }, - "node_modules/uuid": { - "version": "11.1.0", - "resolved": "https://registry.npmjs.org/uuid/-/uuid-11.1.0.tgz", - "integrity": "sha512-0/A9rDy9P7cJ+8w1c9WD9V//9Wj15Ce2MPz8Ri6032usz+NfePxx5AcN3bN+r6ZL6jEo066/yNYB3tn4pQEx+A==", - "funding": [ - "https://github.com/sponsors/broofa", - "https://github.com/sponsors/ctavan" - ], - "license": "MIT", - "bin": { - "uuid": "dist/esm/bin/uuid" - } - }, "node_modules/vali-date": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/vali-date/-/vali-date-1.0.0.tgz", @@ -5539,299 +2628,12 @@ "node": ">= 0.8" } }, - "node_modules/wait-port": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/wait-port/-/wait-port-1.1.0.tgz", - "integrity": "sha512-3e04qkoN3LxTMLakdqeWth8nih8usyg+sf1Bgdf9wwUkp05iuK1eSY/QpLvscT/+F/gA89+LpUmmgBtesbqI2Q==", - "license": "MIT", - "dependencies": { - "chalk": "^4.1.2", - "commander": "^9.3.0", - "debug": "^4.3.4" - }, - "bin": { - "wait-port": "bin/wait-port.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/wait-port/node_modules/chalk": { - "version": "4.1.2", - "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz", - "integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==", - "license": "MIT", - "dependencies": { - "ansi-styles": "^4.1.0", - "supports-color": "^7.1.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/chalk/chalk?sponsor=1" - } - }, - "node_modules/wait-port/node_modules/commander": { - "version": "9.5.0", - "resolved": "https://registry.npmjs.org/commander/-/commander-9.5.0.tgz", - "integrity": "sha512-KRs7WVDKg86PWiuAqhDrAQnTXZKraVcCc6vFdL14qrZ/DcWwuRo7VoiYXalXO7S5GKpqYiVEwCbgFDfxNHKJBQ==", - "license": "MIT", - "engines": { - "node": "^12.20.0 || >=14" - } - }, - "node_modules/wait-port/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/wait-port/node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/webdriver": { - "version": "9.27.0", - "resolved": "https://registry.npmjs.org/webdriver/-/webdriver-9.27.0.tgz", - "integrity": "sha512-w07ThZND48SIr0b4S7eFougYUyclmoUwdmju8yXvEJiXYjDjeYUpl8wZrYPEYRBylxpSx+sBHfEUBrPQkcTTRQ==", - "license": "MIT", - "dependencies": { - "@types/node": "^20.1.0", - "@types/ws": "^8.5.3", - "@wdio/config": "9.27.0", - "@wdio/logger": "9.18.0", - "@wdio/protocols": "9.27.0", - "@wdio/types": "9.27.0", - "@wdio/utils": "9.27.0", - "deepmerge-ts": "^7.0.3", - "https-proxy-agent": "^7.0.6", - "undici": "^6.21.3", - "ws": "^8.8.0" - }, - "engines": { - "node": ">=18.20.0" - } - }, - "node_modules/webdriver/node_modules/undici": { - "version": "6.24.1", - "resolved": "https://registry.npmjs.org/undici/-/undici-6.24.1.tgz", - "integrity": "sha512-sC+b0tB1whOCzbtlx20fx3WgCXwkW627p4EA9uM+/tNNPkSS+eSEld6pAs9nDv7WbY1UUljBMYPtu9BCOrCWKA==", - "license": "MIT", - "engines": { - "node": ">=18.17" - } - }, - "node_modules/webdriverio": { - "version": "9.27.0", - "resolved": "https://registry.npmjs.org/webdriverio/-/webdriverio-9.27.0.tgz", - "integrity": "sha512-Y4FbMf4bKBXpPB0lYpglzQ2GfDDe6uojmMZl85uPyrDx18NW7mqN84ZawGoIg/FRvcLaVhcOzc98WOPo725Rag==", - "license": "MIT", - "dependencies": { - "@types/node": "^20.11.30", - "@types/sinonjs__fake-timers": "^8.1.5", - "@wdio/config": "9.27.0", - "@wdio/logger": "9.18.0", - "@wdio/protocols": "9.27.0", - "@wdio/repl": "9.16.2", - "@wdio/types": "9.27.0", - "@wdio/utils": "9.27.0", - "archiver": "^7.0.1", - "aria-query": "^5.3.0", - "cheerio": "^1.0.0-rc.12", - "css-shorthand-properties": "^1.1.1", - "css-value": "^0.0.1", - "grapheme-splitter": "^1.0.4", - "htmlfy": "^0.8.1", - "is-plain-obj": "^4.1.0", - "jszip": "^3.10.1", - "lodash.clonedeep": "^4.5.0", - "lodash.zip": "^4.2.0", - "query-selector-shadow-dom": "^1.0.1", - "resq": "^1.11.0", - "rgb2hex": "0.2.5", - "serialize-error": "^12.0.0", - "urlpattern-polyfill": "^10.0.0", - "webdriver": "9.27.0" - }, - "engines": { - "node": ">=18.20.0" - }, - "peerDependencies": { - "puppeteer-core": ">=22.x || <=24.x" - }, - "peerDependenciesMeta": { - "puppeteer-core": { - "optional": true - } - } - }, - "node_modules/whatwg-encoding": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/whatwg-encoding/-/whatwg-encoding-3.1.1.tgz", - "integrity": "sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ==", - "deprecated": "Use @exodus/bytes instead for a more spec-conformant and faster implementation", - "license": "MIT", - "dependencies": { - "iconv-lite": "0.6.3" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/whatwg-encoding/node_modules/iconv-lite": { - "version": "0.6.3", - "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz", - "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==", - "license": "MIT", - "dependencies": { - "safer-buffer": ">= 2.1.2 < 3.0.0" - }, - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/whatwg-mimetype": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-4.0.0.tgz", - "integrity": "sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg==", - "license": "MIT", - "engines": { - "node": ">=18" - } - }, - "node_modules/which": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/which/-/which-5.0.0.tgz", - "integrity": "sha512-JEdGzHwwkrbWoGOlIHqQ5gtprKGOenpDHpxE9zVR1bWbOtYRyPPHMe9FaP6x61CmNaTThSkb0DAJte5jD+DmzQ==", - "license": "ISC", - "dependencies": { - "isexe": "^3.1.1" - }, - "bin": { - "node-which": "bin/which.js" - }, - "engines": { - "node": "^18.17.0 || >=20.5.0" - } - }, - "node_modules/wrap-ansi": { - "version": "7.0.0", - "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz", - "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", - "license": "MIT", - "dependencies": { - "ansi-styles": "^4.0.0", - "string-width": "^4.1.0", - "strip-ansi": "^6.0.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/chalk/wrap-ansi?sponsor=1" - } - }, - "node_modules/wrap-ansi-cjs": { - "name": "wrap-ansi", - "version": "7.0.0", - "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz", - "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", - "license": "MIT", - "dependencies": { - "ansi-styles": "^4.0.0", - "string-width": "^4.1.0", - "strip-ansi": "^6.0.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/chalk/wrap-ansi?sponsor=1" - } - }, - "node_modules/wrap-ansi-cjs/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/wrap-ansi-cjs/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/wrap-ansi/node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/wrap-ansi/node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, "node_modules/wrappy": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", "license": "ISC" }, - "node_modules/ws": { - "version": "8.20.0", - "resolved": "https://registry.npmjs.org/ws/-/ws-8.20.0.tgz", - "integrity": "sha512-sAt8BhgNbzCtgGbt2OxmpuryO63ZoDk/sqaB/znQm94T4fCEsy/yV+7CdC1kJhOU9lboAEU7R3kquuycDoibVA==", - "license": "MIT", - "engines": { - "node": ">=10.0.0" - }, - "peerDependencies": { - "bufferutil": "^4.0.1", - "utf-8-validate": ">=5.0.2" - }, - "peerDependenciesMeta": { - "bufferutil": { - "optional": true - }, - "utf-8-validate": { - "optional": true - } - } - }, "node_modules/xml2js": { "version": "0.6.2", "resolved": "https://registry.npmjs.org/xml2js/-/xml2js-0.6.2.tgz", @@ -5853,124 +2655,6 @@ "engines": { "node": ">=4.0" } - }, - "node_modules/y18n": { - "version": "5.0.8", - "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz", - "integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==", - "license": "ISC", - "engines": { - "node": ">=10" - } - }, - "node_modules/yargs": { - "version": "17.7.2", - "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.2.tgz", - "integrity": "sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w==", - "license": "MIT", - "dependencies": { - "cliui": "^8.0.1", - "escalade": "^3.1.1", - "get-caller-file": "^2.0.5", - "require-directory": "^2.1.1", - "string-width": "^4.2.3", - "y18n": "^5.0.5", - "yargs-parser": "^21.1.1" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/yargs-parser": { - "version": "21.1.1", - "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz", - "integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/yauzl": { - "version": "2.10.0", - "resolved": "https://registry.npmjs.org/yauzl/-/yauzl-2.10.0.tgz", - "integrity": "sha512-p4a9I6X6nu6IhoGmBqAcbJy1mlC4j27vEPZX9F4L4/vZT3Lyq1VkFHw/V/PUcB9Buo+DG3iHkT0x3Qya58zc3g==", - "license": "MIT", - "dependencies": { - "buffer-crc32": "~0.2.3", - "fd-slicer": "~1.1.0" - } - }, - "node_modules/yauzl/node_modules/buffer-crc32": { - "version": "0.2.13", - "resolved": "https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-0.2.13.tgz", - "integrity": "sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ==", - "license": "MIT", - "engines": { - "node": "*" - } - }, - "node_modules/zip-stream": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/zip-stream/-/zip-stream-6.0.1.tgz", - "integrity": "sha512-zK7YHHz4ZXpW89AHXUPbQVGKI7uvkd3hzusTdotCg1UxyaVtg0zFJSTfW/Dq5f7OBBVnq6cZIaC8Ti4hb6dtCA==", - "license": "MIT", - "dependencies": { - "archiver-utils": "^5.0.0", - "compress-commons": "^6.0.2", - "readable-stream": "^4.0.0" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/zip-stream/node_modules/buffer": { - "version": "6.0.3", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz", - "integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.2.1" - } - }, - "node_modules/zip-stream/node_modules/readable-stream": { - "version": "4.7.0", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz", - "integrity": "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg==", - "license": "MIT", - "dependencies": { - "abort-controller": "^3.0.0", - "buffer": "^6.0.3", - "events": "^3.3.0", - "process": "^0.11.10", - "string_decoder": "^1.3.0" - }, - "engines": { - "node": "^12.22.0 || ^14.17.0 || >=16.0.0" - } - }, - "node_modules/zod": { - "version": "3.25.76", - "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", - "integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/colinhacks" - } } } } diff --git a/package.json b/package.json index 8d738c36e305..8fcf5cea6969 100644 --- a/package.json +++ b/package.json @@ -16,13 +16,13 @@ }, "homepage": "https://github.com/NousResearch/Hermes-Agent#readme", "dependencies": { - "agent-browser": "^0.13.0", - "@askjo/camoufox-browser": "^1.0.0" + "@askjo/camofox-browser": "^1.5.2", + "agent-browser": "^0.26.0" }, "overrides": { "lodash": "4.18.1" }, "engines": { - "node": ">=18.0.0" + "node": ">=20.0.0" } } diff --git a/plans/gemini-oauth-provider.md b/plans/gemini-oauth-provider.md index 9953d0eca5ec..a466183e8056 100644 --- a/plans/gemini-oauth-provider.md +++ b/plans/gemini-oauth-provider.md @@ -4,7 +4,7 @@ Add a first-class `gemini` provider that authenticates via Google OAuth, using the standard Gemini API (not Cloud Code Assist). Users who have a Google AI subscription or Gemini API access can authenticate through the browser without needing to manually copy API keys. ## Architecture Decision -- **Path A (chosen):** Standard Gemini API at `generativelanguage.googleapis.com/v1beta/openai/` +- **Path A (chosen):** Standard Gemini API at `generativelanguage.googleapis.com/v1beta` - **NOT Path B:** Cloud Code Assist (`cloudcode-pa.googleapis.com`) — rate-limited free tier, internal API, account ban risk - Standard `chat_completions` api_mode via OpenAI SDK — no new api_mode needed - Our own OAuth credentials — NOT sharing tokens with Gemini CLI @@ -32,9 +32,9 @@ Add a first-class `gemini` provider that authenticates via Google OAuth, using t - File locking for concurrent access (multiple agent sessions) ## API Integration -- Base URL: `https://generativelanguage.googleapis.com/v1beta/openai/` -- Auth: `Authorization: Bearer ` (passed as `api_key` to OpenAI SDK) -- api_mode: `chat_completions` (standard) +- Base URL: `https://generativelanguage.googleapis.com/v1beta` +- Auth: native Gemini API authentication handled by the provider adapter +- api_mode: `chat_completions` (standard facade over native transport) - Models: gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash, etc. ## Files to Create/Modify diff --git a/plugins/disk-cleanup/README.md b/plugins/disk-cleanup/README.md new file mode 100644 index 000000000000..bc46047325ad --- /dev/null +++ b/plugins/disk-cleanup/README.md @@ -0,0 +1,51 @@ +# disk-cleanup + +Auto-tracks and cleans up ephemeral files created during Hermes Agent +sessions — test scripts, temp outputs, cron logs, stale chrome profiles. +Scoped strictly to `$HERMES_HOME` and `/tmp/hermes-*`. + +Originally contributed by [@LVT382009](https://github.com/LVT382009) as a +skill in PR #12212. Ported to the plugin system so the behaviour runs +automatically via `post_tool_call` and `on_session_end` hooks — the agent +never needs to remember to call a tool. + +## How it works + +| Hook | Behaviour | +|---|---| +| `post_tool_call` | When `write_file` / `terminal` / `patch` creates a file matching `test_*`, `tmp_*`, or `*.test.*` inside `HERMES_HOME`, track it silently as `test` / `temp` / `cron-output`. | +| `on_session_end` | If any test files were auto-tracked during this turn, run `quick` cleanup (no prompts). | + +Deletion rules (same as the original PR): + +| Category | Threshold | Confirmation | +|---|---|---| +| `test` | every session end | Never | +| `temp` | >7 days since tracked | Never | +| `cron-output` | >14 days since tracked | Never | +| empty dirs under HERMES_HOME | always | Never | +| `research` | >30 days, beyond 10 newest | Always (deep only) | +| `chrome-profile` | >14 days since tracked | Always (deep only) | +| files >500 MB | never auto | Always (deep only) | + +## Slash command + +``` +/disk-cleanup status # breakdown + top-10 largest +/disk-cleanup dry-run # preview without deleting +/disk-cleanup quick # run safe cleanup now +/disk-cleanup deep # quick + list items needing prompt +/disk-cleanup track # manual tracking +/disk-cleanup forget # stop tracking +``` + +## Safety + +- `is_safe_path()` rejects anything outside `HERMES_HOME` or `/tmp/hermes-*` +- Windows mounts (`/mnt/c` etc.) are rejected +- The state directory `$HERMES_HOME/disk-cleanup/` is itself excluded +- `$HERMES_HOME/logs/`, `memories/`, `sessions/`, `skills/`, `plugins/`, + and config files are never tracked +- Backup/restore is scoped to `tracked.json` — the plugin never touches + agent logs +- Atomic writes: `.tmp` → backup → rename diff --git a/plugins/disk-cleanup/__init__.py b/plugins/disk-cleanup/__init__.py new file mode 100644 index 000000000000..0a4b6c7ae164 --- /dev/null +++ b/plugins/disk-cleanup/__init__.py @@ -0,0 +1,316 @@ +"""disk-cleanup plugin — auto-cleanup of ephemeral Hermes session files. + +Wires three behaviours: + +1. ``post_tool_call`` hook — inspects ``write_file`` and ``terminal`` + tool results for newly-created paths matching test/temp patterns + under ``HERMES_HOME`` and tracks them silently. Zero agent + compliance required. + +2. ``on_session_end`` hook — when any test files were auto-tracked + during the just-finished turn, runs :func:`disk_cleanup.quick` and + logs a single line to ``$HERMES_HOME/disk-cleanup/cleanup.log``. + +3. ``/disk-cleanup`` slash command — manual ``status``, ``dry-run``, + ``quick``, ``deep``, ``track``, ``forget``. + +Replaces PR #12212's skill-plus-script design: the agent no longer +needs to remember to run commands. +""" + +from __future__ import annotations + +import logging +import re +import shlex +import threading +from pathlib import Path +from typing import Any, Dict, Optional, Set + +from . import disk_cleanup as dg + +logger = logging.getLogger(__name__) + + +# Per-task set of "test files newly tracked this turn". Keyed by task_id +# (or session_id as fallback) so on_session_end can decide whether to run +# cleanup. Guarded by a lock — post_tool_call can fire concurrently on +# parallel tool calls. +_recent_test_tracks: Dict[str, Set[str]] = {} +_lock = threading.Lock() + + +# Tool-call result shapes we can parse +_WRITE_FILE_PATH_KEY = "path" +_TERMINAL_PATH_REGEX = re.compile(r"(?:^|\s)(/[^\s'\"`]+|\~/[^\s'\"`]+)") + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _tracker_key(task_id: str, session_id: str) -> str: + return task_id or session_id or "default" + + +def _record_track(task_id: str, session_id: str, path: Path, category: str) -> None: + """Record that we tracked *path* as *category* during this turn.""" + if category != "test": + return + key = _tracker_key(task_id, session_id) + with _lock: + _recent_test_tracks.setdefault(key, set()).add(str(path)) + + +def _drain(task_id: str, session_id: str) -> Set[str]: + """Pop the set of test paths tracked during this turn.""" + key = _tracker_key(task_id, session_id) + with _lock: + return _recent_test_tracks.pop(key, set()) + + +def _attempt_track(path_str: str, task_id: str, session_id: str) -> None: + """Best-effort auto-track. Never raises.""" + try: + p = Path(path_str).expanduser() + except Exception: + return + if not p.exists(): + return + category = dg.guess_category(p) + if category is None: + return + newly = dg.track(str(p), category, silent=True) + if newly: + _record_track(task_id, session_id, p, category) + + +def _extract_paths_from_write_file(args: Dict[str, Any]) -> Set[str]: + path = args.get(_WRITE_FILE_PATH_KEY) + return {path} if isinstance(path, str) and path else set() + + +def _extract_paths_from_patch(args: Dict[str, Any]) -> Set[str]: + # The patch tool creates new files via the `mode="patch"` path too, but + # most of its use is editing existing files — we only care about new + # ephemeral creations, so treat patch conservatively and only pick up + # the single-file `path` arg. Track-then-cleanup is idempotent, so + # re-tracking an already-tracked file is a no-op (dedup in track()). + path = args.get("path") + return {path} if isinstance(path, str) and path else set() + + +def _extract_paths_from_terminal(args: Dict[str, Any], result: str) -> Set[str]: + """Best-effort: pull candidate filesystem paths from a terminal command + and its output, then let ``guess_category`` / ``is_safe_path`` filter. + """ + paths: Set[str] = set() + cmd = args.get("command") or "" + if isinstance(cmd, str) and cmd: + # Tokenise the command — catches `touch /tmp/hermes-x/test_foo.py` + try: + for tok in shlex.split(cmd, posix=True): + if tok.startswith(("/", "~")): + paths.add(tok) + except ValueError: + pass + # Only scan the result text if it's a reasonable size (avoid 50KB dumps). + if isinstance(result, str) and len(result) < 4096: + for match in _TERMINAL_PATH_REGEX.findall(result): + paths.add(match) + return paths + + +# --------------------------------------------------------------------------- +# Hooks +# --------------------------------------------------------------------------- + +def _on_post_tool_call( + tool_name: str = "", + args: Optional[Dict[str, Any]] = None, + result: Any = None, + task_id: str = "", + session_id: str = "", + tool_call_id: str = "", + **_: Any, +) -> None: + """Auto-track ephemeral files created by recent tool calls.""" + if not isinstance(args, dict): + return + + candidates: Set[str] = set() + if tool_name == "write_file": + candidates = _extract_paths_from_write_file(args) + elif tool_name == "patch": + candidates = _extract_paths_from_patch(args) + elif tool_name == "terminal": + candidates = _extract_paths_from_terminal(args, result if isinstance(result, str) else "") + else: + return + + for path_str in candidates: + _attempt_track(path_str, task_id, session_id) + + +def _on_session_end( + session_id: str = "", + completed: bool = True, + interrupted: bool = False, + **_: Any, +) -> None: + """Run quick cleanup if any test files were tracked during this turn.""" + # Drain both task-level and session-level buckets. In practice only one + # is populated per turn; the other is empty. + drained_session = _drain("", session_id) + # Also drain any task-scoped buckets that happen to exist. This is a + # cheap sweep: if an agent spawned subagents (each with their own + # task_id) they'll have recorded into separate buckets; we want to + # cleanup them all at session end. + with _lock: + task_buckets = list(_recent_test_tracks.keys()) + for key in task_buckets: + if key and key != session_id: + _recent_test_tracks.pop(key, None) + + if not drained_session and not task_buckets: + return + + try: + summary = dg.quick() + except Exception as exc: + logger.debug("disk-cleanup quick cleanup failed: %s", exc) + return + + if summary["deleted"] or summary["empty_dirs"]: + dg._log( + f"AUTO_QUICK (session_end): deleted={summary['deleted']} " + f"dirs={summary['empty_dirs']} freed={dg.fmt_size(summary['freed'])}" + ) + + +# --------------------------------------------------------------------------- +# Slash command +# --------------------------------------------------------------------------- + +_HELP_TEXT = """\ +/disk-cleanup — ephemeral-file cleanup + +Subcommands: + status Per-category breakdown + top-10 largest + dry-run Preview what quick/deep would delete + quick Run safe cleanup now (no prompts) + deep Run quick, then list items that need prompts + track Manually add a path to tracking + forget Stop tracking a path (does not delete) + +Categories: temp | test | research | download | chrome-profile | cron-output | other + +All operations are scoped to HERMES_HOME and /tmp/hermes-*. +Test files are auto-tracked on write_file / terminal and auto-cleaned at session end. +""" + + +def _fmt_summary(summary: Dict[str, Any]) -> str: + base = ( + f"[disk-cleanup] Cleaned {summary['deleted']} files + " + f"{summary['empty_dirs']} empty dirs, freed {dg.fmt_size(summary['freed'])}." + ) + if summary.get("errors"): + base += f"\n {len(summary['errors'])} error(s); see cleanup.log." + return base + + +def _handle_slash(raw_args: str) -> Optional[str]: + argv = raw_args.strip().split() + if not argv or argv[0] in ("help", "-h", "--help"): + return _HELP_TEXT + + sub = argv[0] + + if sub == "status": + return dg.format_status(dg.status()) + + if sub == "dry-run": + auto, prompt = dg.dry_run() + auto_size = sum(i["size"] for i in auto) + prompt_size = sum(i["size"] for i in prompt) + lines = [ + "Dry-run preview (nothing deleted):", + f" Auto-delete : {len(auto)} files ({dg.fmt_size(auto_size)})", + ] + for item in auto: + lines.append(f" [{item['category']}] {item['path']}") + lines.append( + f" Needs prompt: {len(prompt)} files ({dg.fmt_size(prompt_size)})" + ) + for item in prompt: + lines.append(f" [{item['category']}] {item['path']}") + lines.append( + f"\n Total potential: {dg.fmt_size(auto_size + prompt_size)}" + ) + return "\n".join(lines) + + if sub == "quick": + return _fmt_summary(dg.quick()) + + if sub == "deep": + # In-session deep can't prompt the user interactively — show what + # quick cleaned plus the items that WOULD need confirmation. + quick_summary = dg.quick() + _auto, prompt_items = dg.dry_run() + lines = [_fmt_summary(quick_summary)] + if prompt_items: + size = sum(i["size"] for i in prompt_items) + lines.append( + f"\n{len(prompt_items)} item(s) need confirmation " + f"({dg.fmt_size(size)}):" + ) + for item in prompt_items: + lines.append(f" [{item['category']}] {item['path']}") + lines.append( + "\nRun `/disk-cleanup forget ` to skip, or delete " + "manually via terminal." + ) + return "\n".join(lines) + + if sub == "track": + if len(argv) < 3: + return "Usage: /disk-cleanup track " + path_arg = argv[1] + category = argv[2] + if category not in dg.ALLOWED_CATEGORIES: + return ( + f"Unknown category '{category}'. " + f"Allowed: {sorted(dg.ALLOWED_CATEGORIES)}" + ) + if dg.track(path_arg, category, silent=True): + return f"Tracked {path_arg} as '{category}'." + return ( + f"Not tracked (already present, missing, or outside HERMES_HOME): " + f"{path_arg}" + ) + + if sub == "forget": + if len(argv) < 2: + return "Usage: /disk-cleanup forget " + n = dg.forget(argv[1]) + return ( + f"Removed {n} tracking entr{'y' if n == 1 else 'ies'} for {argv[1]}." + if n else f"Not found in tracking: {argv[1]}" + ) + + return f"Unknown subcommand: {sub}\n\n{_HELP_TEXT}" + + +# --------------------------------------------------------------------------- +# Plugin registration +# --------------------------------------------------------------------------- + +def register(ctx) -> None: + ctx.register_hook("post_tool_call", _on_post_tool_call) + ctx.register_hook("on_session_end", _on_session_end) + ctx.register_command( + "disk-cleanup", + handler=_handle_slash, + description="Track and clean up ephemeral Hermes session files.", + ) diff --git a/plugins/disk-cleanup/disk_cleanup.py b/plugins/disk-cleanup/disk_cleanup.py new file mode 100755 index 000000000000..cef2698316f6 --- /dev/null +++ b/plugins/disk-cleanup/disk_cleanup.py @@ -0,0 +1,496 @@ +"""disk_cleanup — ephemeral file cleanup for Hermes Agent. + +Library module wrapping the deterministic cleanup rules written by +@LVT382009 in PR #12212. The plugin ``__init__.py`` wires these +functions into ``post_tool_call`` and ``on_session_end`` hooks so +tracking and cleanup happen automatically — the agent never needs to +call a tool or remember a skill. + +Rules: + - test files → delete immediately at task end (age >= 0) + - temp files → delete after 7 days + - cron-output → delete after 14 days + - empty dirs → always delete (under HERMES_HOME) + - research → keep 10 newest, prompt for older (deep only) + - chrome-profile→ prompt after 14 days (deep only) + - >500 MB files → prompt always (deep only) + +Scope: strictly HERMES_HOME and /tmp/hermes-* +Never touches: ~/.hermes/logs/ or any system directory. +""" + +from __future__ import annotations + +import json +import logging +import shutil +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +try: + from hermes_constants import get_hermes_home +except Exception: # pragma: no cover — plugin may load before constants resolves + import os + + def get_hermes_home() -> Path: # type: ignore[no-redef] + val = (os.environ.get("HERMES_HOME") or "").strip() + return Path(val).resolve() if val else (Path.home() / ".hermes").resolve() + + +logger = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# Paths +# --------------------------------------------------------------------------- + +def get_state_dir() -> Path: + """State dir — separate from ``$HERMES_HOME/logs/``.""" + return get_hermes_home() / "disk-cleanup" + + +def get_tracked_file() -> Path: + return get_state_dir() / "tracked.json" + + +def get_log_file() -> Path: + """Audit log — intentionally NOT under ``$HERMES_HOME/logs/``.""" + return get_state_dir() / "cleanup.log" + + +# --------------------------------------------------------------------------- +# Path safety +# --------------------------------------------------------------------------- + +def is_safe_path(path: Path) -> bool: + """Accept only paths under HERMES_HOME or ``/tmp/hermes-*``. + + Rejects Windows mounts (``/mnt/c`` etc.) and any system directory. + """ + hermes_home = get_hermes_home() + try: + path.resolve().relative_to(hermes_home) + return True + except (ValueError, OSError): + pass + # Allow /tmp/hermes-* explicitly + parts = path.parts + if len(parts) >= 3 and parts[1] == "tmp" and parts[2].startswith("hermes-"): + return True + return False + + +# --------------------------------------------------------------------------- +# Audit log +# --------------------------------------------------------------------------- + +def _log(message: str) -> None: + try: + log_file = get_log_file() + log_file.parent.mkdir(parents=True, exist_ok=True) + ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S") + with open(log_file, "a") as f: + f.write(f"[{ts}] {message}\n") + except OSError: + # Never let the audit log break the agent loop. + pass + + +# --------------------------------------------------------------------------- +# tracked.json — atomic read/write, backup scoped to tracked.json only +# --------------------------------------------------------------------------- + +def load_tracked() -> List[Dict[str, Any]]: + """Load tracked.json. Restores from ``.bak`` on corruption.""" + tf = get_tracked_file() + tf.parent.mkdir(parents=True, exist_ok=True) + + if not tf.exists(): + return [] + + try: + return json.loads(tf.read_text()) + except (json.JSONDecodeError, ValueError): + bak = tf.with_suffix(".json.bak") + if bak.exists(): + try: + data = json.loads(bak.read_text()) + _log("WARN: tracked.json corrupted — restored from .bak") + return data + except Exception: + pass + _log("WARN: tracked.json corrupted, no backup — starting fresh") + return [] + + +def save_tracked(tracked: List[Dict[str, Any]]) -> None: + """Atomic write: ``.tmp`` → backup old → rename.""" + tf = get_tracked_file() + tf.parent.mkdir(parents=True, exist_ok=True) + tmp = tf.with_suffix(".json.tmp") + tmp.write_text(json.dumps(tracked, indent=2)) + if tf.exists(): + shutil.copy2(tf, tf.with_suffix(".json.bak")) + tmp.replace(tf) + + +# --------------------------------------------------------------------------- +# Categories +# --------------------------------------------------------------------------- + +ALLOWED_CATEGORIES = { + "temp", "test", "research", "download", + "chrome-profile", "cron-output", "other", +} + + +def fmt_size(n: float) -> str: + for unit in ("B", "KB", "MB", "GB", "TB"): + if n < 1024: + return f"{n:.1f} {unit}" + n /= 1024 + return f"{n:.1f} PB" + + +# --------------------------------------------------------------------------- +# Track / forget +# --------------------------------------------------------------------------- + +def track(path_str: str, category: str, silent: bool = False) -> bool: + """Register a file for tracking. Returns True if newly tracked.""" + if category not in ALLOWED_CATEGORIES: + _log(f"WARN: unknown category '{category}', using 'other'") + category = "other" + + path = Path(path_str).resolve() + + if not path.exists(): + _log(f"SKIP: {path} (does not exist)") + return False + + if not is_safe_path(path): + _log(f"REJECT: {path} (outside HERMES_HOME)") + return False + + size = path.stat().st_size if path.is_file() else 0 + tracked = load_tracked() + + # Deduplicate + if any(item["path"] == str(path) for item in tracked): + return False + + tracked.append({ + "path": str(path), + "timestamp": datetime.now(timezone.utc).isoformat(), + "category": category, + "size": size, + }) + save_tracked(tracked) + _log(f"TRACKED: {path} ({category}, {fmt_size(size)})") + if not silent: + print(f"Tracked: {path} ({category}, {fmt_size(size)})") + return True + + +def forget(path_str: str) -> int: + """Remove a path from tracking without deleting the file.""" + p = Path(path_str).resolve() + tracked = load_tracked() + before = len(tracked) + tracked = [i for i in tracked if Path(i["path"]).resolve() != p] + removed = before - len(tracked) + if removed: + save_tracked(tracked) + _log(f"FORGOT: {p} ({removed} entries)") + return removed + + +# --------------------------------------------------------------------------- +# Dry run +# --------------------------------------------------------------------------- + +def dry_run() -> Tuple[List[Dict], List[Dict]]: + """Return (auto_delete_list, needs_prompt_list) without touching files.""" + tracked = load_tracked() + now = datetime.now(timezone.utc) + + auto: List[Dict] = [] + prompt: List[Dict] = [] + + for item in tracked: + p = Path(item["path"]) + if not p.exists(): + continue + age = (now - datetime.fromisoformat(item["timestamp"])).days + cat = item["category"] + size = item["size"] + + if cat == "test": + auto.append(item) + elif cat == "temp" and age > 7: + auto.append(item) + elif cat == "cron-output" and age > 14: + auto.append(item) + elif cat == "research" and age > 30: + prompt.append(item) + elif cat == "chrome-profile" and age > 14: + prompt.append(item) + elif size > 500 * 1024 * 1024: + prompt.append(item) + + return auto, prompt + + +# --------------------------------------------------------------------------- +# Quick cleanup +# --------------------------------------------------------------------------- + +def quick() -> Dict[str, Any]: + """Safe deterministic cleanup — no prompts. + + Returns: ``{"deleted": N, "empty_dirs": N, "freed": bytes, + "errors": [str, ...]}``. + """ + tracked = load_tracked() + now = datetime.now(timezone.utc) + deleted = 0 + freed = 0 + new_tracked: List[Dict] = [] + errors: List[str] = [] + + for item in tracked: + p = Path(item["path"]) + cat = item["category"] + + if not p.exists(): + _log(f"STALE: {p} (removed from tracking)") + continue + + age = (now - datetime.fromisoformat(item["timestamp"])).days + + should_delete = ( + cat == "test" + or (cat == "temp" and age > 7) + or (cat == "cron-output" and age > 14) + ) + + if should_delete: + try: + if p.is_file(): + p.unlink() + elif p.is_dir(): + shutil.rmtree(p) + freed += item["size"] + deleted += 1 + _log(f"DELETED: {p} ({cat}, {fmt_size(item['size'])})") + except OSError as e: + _log(f"ERROR deleting {p}: {e}") + errors.append(f"{p}: {e}") + new_tracked.append(item) + else: + new_tracked.append(item) + + # Remove empty dirs under HERMES_HOME (but leave HERMES_HOME itself and + # a short list of well-known top-level state dirs alone — a fresh install + # has these empty, and deleting them would surprise the user). + hermes_home = get_hermes_home() + _PROTECTED_TOP_LEVEL = { + "logs", "memories", "sessions", "cron", "cronjobs", + "cache", "skills", "plugins", "disk-cleanup", "optional-skills", + "hermes-agent", "backups", "profiles", ".worktrees", + } + empty_removed = 0 + try: + for dirpath in sorted(hermes_home.rglob("*"), reverse=True): + if not dirpath.is_dir() or dirpath == hermes_home: + continue + try: + rel_parts = dirpath.relative_to(hermes_home).parts + except ValueError: + continue + # Skip the well-known top-level state dirs themselves. + if len(rel_parts) == 1 and rel_parts[0] in _PROTECTED_TOP_LEVEL: + continue + try: + if not any(dirpath.iterdir()): + dirpath.rmdir() + empty_removed += 1 + _log(f"DELETED: {dirpath} (empty dir)") + except OSError: + pass + except OSError: + pass + + save_tracked(new_tracked) + _log( + f"QUICK_SUMMARY: {deleted} files, {empty_removed} dirs, " + f"{fmt_size(freed)}" + ) + return { + "deleted": deleted, + "empty_dirs": empty_removed, + "freed": freed, + "errors": errors, + } + + +# --------------------------------------------------------------------------- +# Deep cleanup (interactive — not called from plugin hooks) +# --------------------------------------------------------------------------- + +def deep( + confirm: Optional[callable] = None, +) -> Dict[str, Any]: + """Deep cleanup. + + Runs :func:`quick` first, then asks the *confirm* callable for each + risky item (research > 30d beyond 10 newest, chrome-profile > 14d, + any file > 500 MB). *confirm(item)* must return True to delete. + + Returns: ``{"quick": {...}, "deep_deleted": N, "deep_freed": bytes}``. + """ + quick_result = quick() + + if confirm is None: + # No interactive confirmer — deep stops after the quick pass. + return {"quick": quick_result, "deep_deleted": 0, "deep_freed": 0} + + tracked = load_tracked() + now = datetime.now(timezone.utc) + research, chrome, large = [], [], [] + + for item in tracked: + p = Path(item["path"]) + if not p.exists(): + continue + age = (now - datetime.fromisoformat(item["timestamp"])).days + cat = item["category"] + + if cat == "research" and age > 30: + research.append(item) + elif cat == "chrome-profile" and age > 14: + chrome.append(item) + elif item["size"] > 500 * 1024 * 1024: + large.append(item) + + research.sort(key=lambda x: x["timestamp"], reverse=True) + old_research = research[10:] + + freed, count = 0, 0 + to_remove: List[Dict] = [] + + for group in (old_research, chrome, large): + for item in group: + if confirm(item): + try: + p = Path(item["path"]) + if p.is_file(): + p.unlink() + elif p.is_dir(): + shutil.rmtree(p) + to_remove.append(item) + freed += item["size"] + count += 1 + _log( + f"DELETED: {p} ({item['category']}, " + f"{fmt_size(item['size'])})" + ) + except OSError as e: + _log(f"ERROR deleting {item['path']}: {e}") + + if to_remove: + remove_paths = {i["path"] for i in to_remove} + save_tracked([i for i in tracked if i["path"] not in remove_paths]) + + return {"quick": quick_result, "deep_deleted": count, "deep_freed": freed} + + +# --------------------------------------------------------------------------- +# Status +# --------------------------------------------------------------------------- + +def status() -> Dict[str, Any]: + """Return per-category breakdown and top 10 largest tracked files.""" + tracked = load_tracked() + cats: Dict[str, Dict] = {} + for item in tracked: + c = item["category"] + cats.setdefault(c, {"count": 0, "size": 0}) + cats[c]["count"] += 1 + cats[c]["size"] += item["size"] + + existing = [ + (i["path"], i["size"], i["category"]) + for i in tracked if Path(i["path"]).exists() + ] + existing.sort(key=lambda x: x[1], reverse=True) + + return { + "categories": cats, + "top10": existing[:10], + "total_tracked": len(tracked), + } + + +def format_status(s: Dict[str, Any]) -> str: + """Human-readable status string (for slash command output).""" + lines = [f"{'Category':<20} {'Files':>6} {'Size':>10}", "-" * 40] + cats = s["categories"] + for cat, d in sorted(cats.items(), key=lambda x: x[1]["size"], reverse=True): + lines.append(f"{cat:<20} {d['count']:>6} {fmt_size(d['size']):>10}") + + if not cats: + lines.append("(nothing tracked yet)") + + lines.append("") + lines.append("Top 10 largest tracked files:") + if not s["top10"]: + lines.append(" (none)") + else: + for rank, (path, size, cat) in enumerate(s["top10"], 1): + lines.append(f" {rank:>2}. {fmt_size(size):>8} [{cat}] {path}") + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Auto-categorisation from tool-call inspection +# --------------------------------------------------------------------------- + +_TEST_PATTERNS = ("test_", "tmp_") +_TEST_SUFFIXES = (".test.py", ".test.js", ".test.ts", ".test.md") + + +def guess_category(path: Path) -> Optional[str]: + """Return a category label for *path*, or None if we shouldn't track it. + + Used by the ``post_tool_call`` hook to auto-track ephemeral files. + """ + if not is_safe_path(path): + return None + + # Skip the state dir itself, logs, memory files, sessions, config. + hermes_home = get_hermes_home() + try: + rel = path.resolve().relative_to(hermes_home) + top = rel.parts[0] if rel.parts else "" + if top in { + "disk-cleanup", "logs", "memories", "sessions", "config.yaml", + "skills", "plugins", ".env", "USER.md", "MEMORY.md", "SOUL.md", + "auth.json", "hermes-agent", + }: + return None + if top == "cron" or top == "cronjobs": + return "cron-output" + if top == "cache": + return "temp" + except ValueError: + # Path isn't under HERMES_HOME (e.g. /tmp/hermes-*) — fall through. + pass + + name = path.name + if name.startswith(_TEST_PATTERNS): + return "test" + if any(name.endswith(sfx) for sfx in _TEST_SUFFIXES): + return "test" + return None diff --git a/plugins/disk-cleanup/plugin.yaml b/plugins/disk-cleanup/plugin.yaml new file mode 100644 index 000000000000..fe005c88491b --- /dev/null +++ b/plugins/disk-cleanup/plugin.yaml @@ -0,0 +1,7 @@ +name: disk-cleanup +version: 2.0.0 +description: "Auto-track and clean up ephemeral files (test scripts, temp outputs, cron logs) created during Hermes sessions. Runs via plugin hooks — no agent action required." +author: "@LVT382009 (original), NousResearch (plugin port)" +hooks: + - post_tool_call + - on_session_end diff --git a/plugins/example-dashboard/dashboard/dist/index.js b/plugins/example-dashboard/dashboard/dist/index.js new file mode 100644 index 000000000000..a54916be41d9 --- /dev/null +++ b/plugins/example-dashboard/dashboard/dist/index.js @@ -0,0 +1,94 @@ +/** + * Example Dashboard Plugin + * + * Demonstrates how to build a dashboard plugin using the Hermes Plugin SDK. + * No build step needed — this is a plain IIFE that uses globals from the SDK. + */ +(function () { + "use strict"; + + const SDK = window.__HERMES_PLUGIN_SDK__; + const { React } = SDK; + const { Card, CardHeader, CardTitle, CardContent, Badge, Button } = SDK.components; + const { useState, useEffect } = SDK.hooks; + const { cn } = SDK.utils; + + function ExamplePage() { + const [greeting, setGreeting] = useState(null); + const [loading, setLoading] = useState(false); + + function fetchGreeting() { + setLoading(true); + SDK.fetchJSON("/api/plugins/example/hello") + .then(function (data) { setGreeting(data.message); }) + .catch(function () { setGreeting("(backend not available)"); }) + .finally(function () { setLoading(false); }); + } + + return React.createElement("div", { className: "flex flex-col gap-6" }, + // Header card + React.createElement(Card, null, + React.createElement(CardHeader, null, + React.createElement("div", { className: "flex items-center gap-3" }, + React.createElement(CardTitle, { className: "text-lg" }, "Example Plugin"), + React.createElement(Badge, { variant: "outline" }, "v1.0.0"), + ), + ), + React.createElement(CardContent, { className: "flex flex-col gap-4" }, + React.createElement("p", { className: "text-sm text-muted-foreground" }, + "This is an example dashboard plugin. It demonstrates using the Plugin SDK to build ", + "custom tabs with React components, connect to backend API routes, and integrate with ", + "the existing Hermes UI system.", + ), + React.createElement("div", { className: "flex items-center gap-3" }, + React.createElement(Button, { + onClick: fetchGreeting, + disabled: loading, + className: cn( + "inline-flex items-center gap-2 border border-border bg-background/40 px-4 py-2", + "text-sm font-courier transition-colors hover:bg-foreground/10 cursor-pointer", + ), + }, loading ? "Loading..." : "Call Backend API"), + greeting && React.createElement("span", { + className: "text-sm font-courier text-muted-foreground", + }, greeting), + ), + ), + ), + + // Info card about the SDK + React.createElement(Card, null, + React.createElement(CardHeader, null, + React.createElement(CardTitle, { className: "text-base" }, "Plugin SDK Reference"), + ), + React.createElement(CardContent, null, + React.createElement("div", { className: "grid gap-3 text-sm" }, + React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" }, + React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.React"), + React.createElement("span", { className: "text-muted-foreground text-xs" }, "React instance — use instead of importing react"), + ), + React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" }, + React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.hooks"), + React.createElement("span", { className: "text-muted-foreground text-xs" }, "useState, useEffect, useCallback, useMemo, useRef, useContext, createContext"), + ), + React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" }, + React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.components"), + React.createElement("span", { className: "text-muted-foreground text-xs" }, "Card, Badge, Button, Input, Label, Select, Separator, Tabs, etc."), + ), + React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" }, + React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.api"), + React.createElement("span", { className: "text-muted-foreground text-xs" }, "Hermes API client — getStatus(), getSessions(), etc."), + ), + React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" }, + React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.utils"), + React.createElement("span", { className: "text-muted-foreground text-xs" }, "cn(), timeAgo(), isoTimeAgo()"), + ), + ), + ), + ), + ); + } + + // Register this plugin — the dashboard picks it up automatically. + window.__HERMES_PLUGINS__.register("example", ExamplePage); +})(); diff --git a/plugins/example-dashboard/dashboard/manifest.json b/plugins/example-dashboard/dashboard/manifest.json new file mode 100644 index 000000000000..2111bff5e71f --- /dev/null +++ b/plugins/example-dashboard/dashboard/manifest.json @@ -0,0 +1,13 @@ +{ + "name": "example", + "label": "Example", + "description": "Example dashboard plugin — demonstrates the plugin SDK", + "icon": "Sparkles", + "version": "1.0.0", + "tab": { + "path": "/example", + "position": "after:skills" + }, + "entry": "dist/index.js", + "api": "plugin_api.py" +} diff --git a/plugins/example-dashboard/dashboard/plugin_api.py b/plugins/example-dashboard/dashboard/plugin_api.py new file mode 100644 index 000000000000..20aed76e26fe --- /dev/null +++ b/plugins/example-dashboard/dashboard/plugin_api.py @@ -0,0 +1,14 @@ +"""Example dashboard plugin — backend API routes. + +Mounted at /api/plugins/example/ by the dashboard plugin system. +""" + +from fastapi import APIRouter + +router = APIRouter() + + +@router.get("/hello") +async def hello(): + """Simple greeting endpoint to demonstrate plugin API routes.""" + return {"message": "Hello from the example plugin!", "plugin": "example", "version": "1.0.0"} diff --git a/plugins/image_gen/openai-codex/__init__.py b/plugins/image_gen/openai-codex/__init__.py new file mode 100644 index 000000000000..ab524dbdd759 --- /dev/null +++ b/plugins/image_gen/openai-codex/__init__.py @@ -0,0 +1,378 @@ +"""OpenAI image generation backend — ChatGPT/Codex OAuth variant. + +Identical model catalog and tier semantics to the ``openai`` image-gen plugin +(``gpt-image-2`` at low/medium/high quality), but routes the request through +the Codex Responses API ``image_generation`` tool instead of the +``images.generate`` REST endpoint. This lets users who are already +authenticated with Codex/ChatGPT generate images without configuring a +separate ``OPENAI_API_KEY``. + +Selection precedence for the tier (first hit wins): + +1. ``OPENAI_IMAGE_MODEL`` env var (escape hatch for scripts / tests) +2. ``image_gen.openai-codex.model`` in ``config.yaml`` +3. ``image_gen.model`` in ``config.yaml`` (when it's one of our tier IDs) +4. :data:`DEFAULT_MODEL` — ``gpt-image-2-medium`` + +Output is saved as PNG under ``$HERMES_HOME/cache/images/``. +""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, List, Optional, Tuple + +from agent.image_gen_provider import ( + DEFAULT_ASPECT_RATIO, + ImageGenProvider, + error_response, + resolve_aspect_ratio, + save_b64_image, + success_response, +) + +logger = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# Model catalog — mirrors the ``openai`` plugin so the picker UX is identical. +# --------------------------------------------------------------------------- + +API_MODEL = "gpt-image-2" + +_MODELS: Dict[str, Dict[str, Any]] = { + "gpt-image-2-low": { + "display": "GPT Image 2 (Low)", + "speed": "~15s", + "strengths": "Fast iteration, lowest cost", + "quality": "low", + }, + "gpt-image-2-medium": { + "display": "GPT Image 2 (Medium)", + "speed": "~40s", + "strengths": "Balanced — default", + "quality": "medium", + }, + "gpt-image-2-high": { + "display": "GPT Image 2 (High)", + "speed": "~2min", + "strengths": "Highest fidelity, strongest prompt adherence", + "quality": "high", + }, +} + +DEFAULT_MODEL = "gpt-image-2-medium" + +_SIZES = { + "landscape": "1536x1024", + "square": "1024x1024", + "portrait": "1024x1536", +} + +# Codex Responses surface used for the request. The chat model itself is only +# the host that calls the ``image_generation`` tool; the actual image work is +# done by ``API_MODEL``. +_CODEX_CHAT_MODEL = "gpt-5.4" +_CODEX_BASE_URL = "https://chatgpt.com/backend-api/codex" +_CODEX_INSTRUCTIONS = ( + "You are an assistant that must fulfill image generation requests by " + "using the image_generation tool when provided." +) + + +# --------------------------------------------------------------------------- +# Config + auth helpers +# --------------------------------------------------------------------------- + + +def _load_image_gen_config() -> Dict[str, Any]: + """Read ``image_gen`` from config.yaml (returns {} on any failure).""" + try: + from hermes_cli.config import load_config + + cfg = load_config() + section = cfg.get("image_gen") if isinstance(cfg, dict) else None + return section if isinstance(section, dict) else {} + except Exception as exc: + logger.debug("Could not load image_gen config: %s", exc) + return {} + + +def _resolve_model() -> Tuple[str, Dict[str, Any]]: + """Decide which tier to use and return ``(model_id, meta)``.""" + import os + + env_override = os.environ.get("OPENAI_IMAGE_MODEL") + if env_override and env_override in _MODELS: + return env_override, _MODELS[env_override] + + cfg = _load_image_gen_config() + sub = cfg.get("openai-codex") if isinstance(cfg.get("openai-codex"), dict) else {} + candidate: Optional[str] = None + if isinstance(sub, dict): + value = sub.get("model") + if isinstance(value, str) and value in _MODELS: + candidate = value + if candidate is None: + top = cfg.get("model") + if isinstance(top, str) and top in _MODELS: + candidate = top + + if candidate is not None: + return candidate, _MODELS[candidate] + + return DEFAULT_MODEL, _MODELS[DEFAULT_MODEL] + + +def _read_codex_access_token() -> Optional[str]: + """Return a usable Codex OAuth token, or None. + + Delegates to the canonical reader in ``agent.auxiliary_client`` so token + expiry, credential pool selection, and JWT decoding stay in one place. + """ + try: + from agent.auxiliary_client import _read_codex_access_token as _reader + + token = _reader() + if isinstance(token, str) and token.strip(): + return token.strip() + return None + except Exception as exc: + logger.debug("Could not resolve Codex access token: %s", exc) + return None + + +def _build_codex_client(): + """Return an OpenAI client pointed at the ChatGPT/Codex backend, or None.""" + token = _read_codex_access_token() + if not token: + return None + try: + import openai + from agent.auxiliary_client import _codex_cloudflare_headers + + return openai.OpenAI( + api_key=token, + base_url=_CODEX_BASE_URL, + default_headers=_codex_cloudflare_headers(token), + ) + except Exception as exc: + logger.debug("Could not build Codex image client: %s", exc) + return None + + +def _collect_image_b64(client: Any, *, prompt: str, size: str, quality: str) -> Optional[str]: + """Stream a Codex Responses image_generation call and return the b64 image.""" + image_b64: Optional[str] = None + + with client.responses.stream( + model=_CODEX_CHAT_MODEL, + store=False, + instructions=_CODEX_INSTRUCTIONS, + input=[{ + "type": "message", + "role": "user", + "content": [{"type": "input_text", "text": prompt}], + }], + tools=[{ + "type": "image_generation", + "model": API_MODEL, + "size": size, + "quality": quality, + "output_format": "png", + "background": "opaque", + "partial_images": 1, + }], + tool_choice={ + "type": "allowed_tools", + "mode": "required", + "tools": [{"type": "image_generation"}], + }, + ) as stream: + for event in stream: + event_type = getattr(event, "type", "") + if event_type == "response.output_item.done": + item = getattr(event, "item", None) + if getattr(item, "type", None) == "image_generation_call": + result = getattr(item, "result", None) + if isinstance(result, str) and result: + image_b64 = result + elif event_type == "response.image_generation_call.partial_image": + partial = getattr(event, "partial_image_b64", None) + if isinstance(partial, str) and partial: + image_b64 = partial + final = stream.get_final_response() + + # Final-response sweep covers the case where the stream finished before + # we observed the ``output_item.done`` event for the image call. + for item in getattr(final, "output", None) or []: + if getattr(item, "type", None) == "image_generation_call": + result = getattr(item, "result", None) + if isinstance(result, str) and result: + image_b64 = result + + return image_b64 + + +# --------------------------------------------------------------------------- +# Provider +# --------------------------------------------------------------------------- + + +class OpenAICodexImageGenProvider(ImageGenProvider): + """gpt-image-2 routed through ChatGPT/Codex OAuth instead of an API key.""" + + @property + def name(self) -> str: + return "openai-codex" + + @property + def display_name(self) -> str: + return "OpenAI (Codex auth)" + + def is_available(self) -> bool: + if not _read_codex_access_token(): + return False + try: + import openai # noqa: F401 + except ImportError: + return False + return True + + def list_models(self) -> List[Dict[str, Any]]: + return [ + { + "id": model_id, + "display": meta["display"], + "speed": meta["speed"], + "strengths": meta["strengths"], + "price": "varies", + } + for model_id, meta in _MODELS.items() + ] + + def default_model(self) -> Optional[str]: + return DEFAULT_MODEL + + def get_setup_schema(self) -> Dict[str, Any]: + return { + "name": "OpenAI (Codex auth)", + "badge": "free", + "tag": "gpt-image-2 via ChatGPT/Codex OAuth — no API key required", + "env_vars": [], + "post_setup_hint": ( + "Sign in with `hermes auth codex` (or `hermes setup` → Codex) " + "if you haven't already. No API key needed." + ), + } + + def generate( + self, + prompt: str, + aspect_ratio: str = DEFAULT_ASPECT_RATIO, + **kwargs: Any, + ) -> Dict[str, Any]: + prompt = (prompt or "").strip() + aspect = resolve_aspect_ratio(aspect_ratio) + + if not prompt: + return error_response( + error="Prompt is required and must be a non-empty string", + error_type="invalid_argument", + provider="openai-codex", + aspect_ratio=aspect, + ) + + if not _read_codex_access_token(): + return error_response( + error=( + "No Codex/ChatGPT OAuth credentials available. Run " + "`hermes auth codex` (or `hermes setup` → Codex) to sign in." + ), + error_type="auth_required", + provider="openai-codex", + aspect_ratio=aspect, + ) + + try: + import openai # noqa: F401 + except ImportError: + return error_response( + error="openai Python package not installed (pip install openai)", + error_type="missing_dependency", + provider="openai-codex", + aspect_ratio=aspect, + ) + + tier_id, meta = _resolve_model() + size = _SIZES.get(aspect, _SIZES["square"]) + + client = _build_codex_client() + if client is None: + return error_response( + error="Could not initialize Codex image client", + error_type="auth_required", + provider="openai-codex", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + try: + b64 = _collect_image_b64( + client, + prompt=prompt, + size=size, + quality=meta["quality"], + ) + except Exception as exc: + logger.debug("Codex image generation failed", exc_info=True) + return error_response( + error=f"OpenAI image generation via Codex auth failed: {exc}", + error_type="api_error", + provider="openai-codex", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + if not b64: + return error_response( + error="Codex response contained no image_generation_call result", + error_type="empty_response", + provider="openai-codex", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + try: + saved_path = save_b64_image(b64, prefix=f"openai_codex_{tier_id}") + except Exception as exc: + return error_response( + error=f"Could not save image to cache: {exc}", + error_type="io_error", + provider="openai-codex", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + return success_response( + image=str(saved_path), + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + provider="openai-codex", + extra={"size": size, "quality": meta["quality"]}, + ) + + +# --------------------------------------------------------------------------- +# Plugin entry point +# --------------------------------------------------------------------------- + + +def register(ctx) -> None: + """Plugin entry point — register the Codex-backed image-gen provider.""" + ctx.register_image_gen_provider(OpenAICodexImageGenProvider()) diff --git a/plugins/image_gen/openai-codex/plugin.yaml b/plugins/image_gen/openai-codex/plugin.yaml new file mode 100644 index 000000000000..61757773e19c --- /dev/null +++ b/plugins/image_gen/openai-codex/plugin.yaml @@ -0,0 +1,5 @@ +name: openai-codex +version: 1.0.0 +description: "OpenAI image generation backed by ChatGPT/Codex OAuth (gpt-image-2 via the Responses image_generation tool). Saves generated images to $HERMES_HOME/cache/images/." +author: NousResearch +kind: backend diff --git a/plugins/image_gen/openai/__init__.py b/plugins/image_gen/openai/__init__.py new file mode 100644 index 000000000000..c1a719f91022 --- /dev/null +++ b/plugins/image_gen/openai/__init__.py @@ -0,0 +1,303 @@ +"""OpenAI image generation backend. + +Exposes OpenAI's ``gpt-image-2`` model at three quality tiers as an +:class:`ImageGenProvider` implementation. The tiers are implemented as +three virtual model IDs so the ``hermes tools`` model picker and the +``image_gen.model`` config key behave like any other multi-model backend: + + gpt-image-2-low ~15s fastest, good for iteration + gpt-image-2-medium ~40s default — balanced + gpt-image-2-high ~2min slowest, highest fidelity + +All three hit the same underlying API model (``gpt-image-2``) with a +different ``quality`` parameter. Output is base64 JSON → saved under +``$HERMES_HOME/cache/images/``. + +Selection precedence (first hit wins): + +1. ``OPENAI_IMAGE_MODEL`` env var (escape hatch for scripts / tests) +2. ``image_gen.openai.model`` in ``config.yaml`` +3. ``image_gen.model`` in ``config.yaml`` (when it's one of our tier IDs) +4. :data:`DEFAULT_MODEL` — ``gpt-image-2-medium`` +""" + +from __future__ import annotations + +import logging +import os +from typing import Any, Dict, List, Optional, Tuple + +from agent.image_gen_provider import ( + DEFAULT_ASPECT_RATIO, + ImageGenProvider, + error_response, + resolve_aspect_ratio, + save_b64_image, + success_response, +) + +logger = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# Model catalog +# --------------------------------------------------------------------------- +# +# All three IDs resolve to the same underlying API model with a different +# ``quality`` setting. ``api_model`` is what gets sent to OpenAI; +# ``quality`` is the knob that changes generation time and output fidelity. + +API_MODEL = "gpt-image-2" + +_MODELS: Dict[str, Dict[str, Any]] = { + "gpt-image-2-low": { + "display": "GPT Image 2 (Low)", + "speed": "~15s", + "strengths": "Fast iteration, lowest cost", + "quality": "low", + }, + "gpt-image-2-medium": { + "display": "GPT Image 2 (Medium)", + "speed": "~40s", + "strengths": "Balanced — default", + "quality": "medium", + }, + "gpt-image-2-high": { + "display": "GPT Image 2 (High)", + "speed": "~2min", + "strengths": "Highest fidelity, strongest prompt adherence", + "quality": "high", + }, +} + +DEFAULT_MODEL = "gpt-image-2-medium" + +_SIZES = { + "landscape": "1536x1024", + "square": "1024x1024", + "portrait": "1024x1536", +} + + +def _load_openai_config() -> Dict[str, Any]: + """Read ``image_gen`` from config.yaml (returns {} on any failure).""" + try: + from hermes_cli.config import load_config + + cfg = load_config() + section = cfg.get("image_gen") if isinstance(cfg, dict) else None + return section if isinstance(section, dict) else {} + except Exception as exc: + logger.debug("Could not load image_gen config: %s", exc) + return {} + + +def _resolve_model() -> Tuple[str, Dict[str, Any]]: + """Decide which tier to use and return ``(model_id, meta)``.""" + env_override = os.environ.get("OPENAI_IMAGE_MODEL") + if env_override and env_override in _MODELS: + return env_override, _MODELS[env_override] + + cfg = _load_openai_config() + openai_cfg = cfg.get("openai") if isinstance(cfg.get("openai"), dict) else {} + candidate: Optional[str] = None + if isinstance(openai_cfg, dict): + value = openai_cfg.get("model") + if isinstance(value, str) and value in _MODELS: + candidate = value + if candidate is None: + top = cfg.get("model") + if isinstance(top, str) and top in _MODELS: + candidate = top + + if candidate is not None: + return candidate, _MODELS[candidate] + + return DEFAULT_MODEL, _MODELS[DEFAULT_MODEL] + + +# --------------------------------------------------------------------------- +# Provider +# --------------------------------------------------------------------------- + + +class OpenAIImageGenProvider(ImageGenProvider): + """OpenAI ``images.generate`` backend — gpt-image-2 at low/medium/high.""" + + @property + def name(self) -> str: + return "openai" + + @property + def display_name(self) -> str: + return "OpenAI" + + def is_available(self) -> bool: + if not os.environ.get("OPENAI_API_KEY"): + return False + try: + import openai # noqa: F401 + except ImportError: + return False + return True + + def list_models(self) -> List[Dict[str, Any]]: + return [ + { + "id": model_id, + "display": meta["display"], + "speed": meta["speed"], + "strengths": meta["strengths"], + "price": "varies", + } + for model_id, meta in _MODELS.items() + ] + + def default_model(self) -> Optional[str]: + return DEFAULT_MODEL + + def get_setup_schema(self) -> Dict[str, Any]: + return { + "name": "OpenAI", + "badge": "paid", + "tag": "gpt-image-2 at low/medium/high quality tiers", + "env_vars": [ + { + "key": "OPENAI_API_KEY", + "prompt": "OpenAI API key", + "url": "https://platform.openai.com/api-keys", + }, + ], + } + + def generate( + self, + prompt: str, + aspect_ratio: str = DEFAULT_ASPECT_RATIO, + **kwargs: Any, + ) -> Dict[str, Any]: + prompt = (prompt or "").strip() + aspect = resolve_aspect_ratio(aspect_ratio) + + if not prompt: + return error_response( + error="Prompt is required and must be a non-empty string", + error_type="invalid_argument", + provider="openai", + aspect_ratio=aspect, + ) + + if not os.environ.get("OPENAI_API_KEY"): + return error_response( + error=( + "OPENAI_API_KEY not set. Run `hermes tools` → Image " + "Generation → OpenAI to configure, or `hermes setup` " + "to add the key." + ), + error_type="auth_required", + provider="openai", + aspect_ratio=aspect, + ) + + try: + import openai + except ImportError: + return error_response( + error="openai Python package not installed (pip install openai)", + error_type="missing_dependency", + provider="openai", + aspect_ratio=aspect, + ) + + tier_id, meta = _resolve_model() + size = _SIZES.get(aspect, _SIZES["square"]) + + # gpt-image-2 returns b64_json unconditionally and REJECTS + # ``response_format`` as an unknown parameter. Don't send it. + payload: Dict[str, Any] = { + "model": API_MODEL, + "prompt": prompt, + "size": size, + "n": 1, + "quality": meta["quality"], + } + + try: + client = openai.OpenAI() + response = client.images.generate(**payload) + except Exception as exc: + logger.debug("OpenAI image generation failed", exc_info=True) + return error_response( + error=f"OpenAI image generation failed: {exc}", + error_type="api_error", + provider="openai", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + data = getattr(response, "data", None) or [] + if not data: + return error_response( + error="OpenAI returned no image data", + error_type="empty_response", + provider="openai", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + first = data[0] + b64 = getattr(first, "b64_json", None) + url = getattr(first, "url", None) + revised_prompt = getattr(first, "revised_prompt", None) + + if b64: + try: + saved_path = save_b64_image(b64, prefix=f"openai_{tier_id}") + except Exception as exc: + return error_response( + error=f"Could not save image to cache: {exc}", + error_type="io_error", + provider="openai", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + image_ref = str(saved_path) + elif url: + # Defensive — gpt-image-2 returns b64 today, but fall back + # gracefully if the API ever changes. + image_ref = url + else: + return error_response( + error="OpenAI response contained neither b64_json nor URL", + error_type="empty_response", + provider="openai", + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + extra: Dict[str, Any] = {"size": size, "quality": meta["quality"]} + if revised_prompt: + extra["revised_prompt"] = revised_prompt + + return success_response( + image=image_ref, + model=tier_id, + prompt=prompt, + aspect_ratio=aspect, + provider="openai", + extra=extra, + ) + + +# --------------------------------------------------------------------------- +# Plugin entry point +# --------------------------------------------------------------------------- + + +def register(ctx) -> None: + """Plugin entry point — wire ``OpenAIImageGenProvider`` into the registry.""" + ctx.register_image_gen_provider(OpenAIImageGenProvider()) diff --git a/plugins/image_gen/openai/plugin.yaml b/plugins/image_gen/openai/plugin.yaml new file mode 100644 index 000000000000..18e4d86390db --- /dev/null +++ b/plugins/image_gen/openai/plugin.yaml @@ -0,0 +1,7 @@ +name: openai +version: 1.0.0 +description: "OpenAI image generation backend (gpt-image-2). Saves generated images to $HERMES_HOME/cache/images/." +author: NousResearch +kind: backend +requires_env: + - OPENAI_API_KEY diff --git a/plugins/image_gen/xai/__init__.py b/plugins/image_gen/xai/__init__.py new file mode 100644 index 000000000000..b1ec4368efab --- /dev/null +++ b/plugins/image_gen/xai/__init__.py @@ -0,0 +1,313 @@ +"""xAI image generation backend. + +Exposes xAI's ``grok-imagine-image`` model as an +:class:`ImageGenProvider` implementation. + +Features: +- Text-to-image generation +- Multiple aspect ratios (1:1, 16:9, 9:16, etc.) +- Multiple resolutions (1K, 2K) +- Base64 output saved to cache + +Selection precedence (first hit wins): +1. ``XAI_IMAGE_MODEL`` env var +2. ``image_gen.xai.model`` in ``config.yaml`` +3. :data:`DEFAULT_MODEL` +""" + +from __future__ import annotations + +import logging +import os +from typing import Any, Dict, List, Optional, Tuple + +import requests + +from agent.image_gen_provider import ( + DEFAULT_ASPECT_RATIO, + ImageGenProvider, + error_response, + resolve_aspect_ratio, + save_b64_image, + success_response, +) +from tools.xai_http import hermes_xai_user_agent + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Model catalog +# --------------------------------------------------------------------------- + +API_MODEL = "grok-imagine-image" + +_MODELS: Dict[str, Dict[str, Any]] = { + "grok-imagine-image": { + "display": "Grok Imagine Image", + "speed": "~5-10s", + "strengths": "Fast, high-quality", + }, +} + +DEFAULT_MODEL = "grok-imagine-image" + +# xAI aspect ratios (more options than FAL/OpenAI) +_XAI_ASPECT_RATIOS = { + "landscape": "16:9", + "square": "1:1", + "portrait": "9:16", + "4:3": "4:3", + "3:4": "3:4", + "3:2": "3:2", + "2:3": "2:3", +} + +# xAI resolutions +_XAI_RESOLUTIONS = { + "1k": "1024", + "2k": "2048", +} + +DEFAULT_RESOLUTION = "1k" + + +# --------------------------------------------------------------------------- +# Config +# --------------------------------------------------------------------------- + + +def _load_xai_config() -> Dict[str, Any]: + """Read ``image_gen.xai`` from config.yaml.""" + try: + from hermes_cli.config import load_config + + cfg = load_config() + section = cfg.get("image_gen") if isinstance(cfg, dict) else None + xai_section = section.get("xai") if isinstance(section, dict) else None + return xai_section if isinstance(xai_section, dict) else {} + except Exception as exc: + logger.debug("Could not load image_gen.xai config: %s", exc) + return {} + + +def _resolve_model() -> Tuple[str, Dict[str, Any]]: + """Decide which model to use and return ``(model_id, meta)``.""" + env_override = os.environ.get("XAI_IMAGE_MODEL") + if env_override and env_override in _MODELS: + return env_override, _MODELS[env_override] + + cfg = _load_xai_config() + candidate = cfg.get("model") if isinstance(cfg.get("model"), str) else None + if candidate and candidate in _MODELS: + return candidate, _MODELS[candidate] + + return DEFAULT_MODEL, _MODELS[DEFAULT_MODEL] + + +def _resolve_resolution() -> str: + """Get configured resolution.""" + cfg = _load_xai_config() + res = cfg.get("resolution") if isinstance(cfg.get("resolution"), str) else None + if res and res in _XAI_RESOLUTIONS: + return res + return DEFAULT_RESOLUTION + + +# --------------------------------------------------------------------------- +# Provider +# --------------------------------------------------------------------------- + + +class XAIImageGenProvider(ImageGenProvider): + """xAI ``grok-imagine-image`` backend.""" + + @property + def name(self) -> str: + return "xai" + + @property + def display_name(self) -> str: + return "xAI (Grok)" + + def is_available(self) -> bool: + return bool(os.getenv("XAI_API_KEY")) + + def list_models(self) -> List[Dict[str, Any]]: + return [ + { + "id": model_id, + "display": meta.get("display", model_id), + "speed": meta.get("speed", ""), + "strengths": meta.get("strengths", ""), + } + for model_id, meta in _MODELS.items() + ] + + def get_setup_schema(self) -> Dict[str, Any]: + return { + "name": "xAI (Grok)", + "badge": "paid", + "tag": "Native xAI image generation via grok-imagine-image", + "env_vars": [ + { + "key": "XAI_API_KEY", + "prompt": "xAI API key", + "url": "https://console.x.ai/", + }, + ], + } + + def generate( + self, + prompt: str, + aspect_ratio: str = DEFAULT_ASPECT_RATIO, + **kwargs: Any, + ) -> Dict[str, Any]: + """Generate an image using xAI's grok-imagine-image.""" + api_key = os.getenv("XAI_API_KEY", "").strip() + if not api_key: + return error_response( + error="XAI_API_KEY not set. Get one at https://console.x.ai/", + error_type="missing_api_key", + provider="xai", + aspect_ratio=aspect_ratio, + ) + + model_id, meta = _resolve_model() + aspect = resolve_aspect_ratio(aspect_ratio) + xai_ar = _XAI_ASPECT_RATIOS.get(aspect, "1:1") + resolution = _resolve_resolution() + xai_res = _XAI_RESOLUTIONS.get(resolution, "1024") + + payload: Dict[str, Any] = { + "model": API_MODEL, + "prompt": prompt, + "aspect_ratio": xai_ar, + "resolution": xai_res, + } + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + "User-Agent": hermes_xai_user_agent(), + } + + base_url = (os.getenv("XAI_BASE_URL") or "https://api.x.ai/v1").strip().rstrip("/") + + try: + response = requests.post( + f"{base_url}/images/generations", + headers=headers, + json=payload, + timeout=120, + ) + response.raise_for_status() + except requests.HTTPError as exc: + status = exc.response.status_code if exc.response else 0 + try: + err_msg = exc.response.json().get("error", {}).get("message", exc.response.text[:300]) + except Exception: + err_msg = exc.response.text[:300] if exc.response else str(exc) + logger.error("xAI image gen failed (%d): %s", status, err_msg) + return error_response( + error=f"xAI image generation failed ({status}): {err_msg}", + error_type="api_error", + provider="xai", + model=model_id, + prompt=prompt, + aspect_ratio=aspect, + ) + except requests.Timeout: + return error_response( + error="xAI image generation timed out (120s)", + error_type="timeout", + provider="xai", + model=model_id, + prompt=prompt, + aspect_ratio=aspect, + ) + except requests.ConnectionError as exc: + return error_response( + error=f"xAI connection error: {exc}", + error_type="connection_error", + provider="xai", + model=model_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + try: + result = response.json() + except Exception as exc: + return error_response( + error=f"xAI returned invalid JSON: {exc}", + error_type="invalid_response", + provider="xai", + model=model_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + # Parse response — xAI returns data[0].b64_json or data[0].url + data = result.get("data", []) + if not data: + return error_response( + error="xAI returned no image data", + error_type="empty_response", + provider="xai", + model=model_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + first = data[0] + b64 = first.get("b64_json") + url = first.get("url") + + if b64: + try: + saved_path = save_b64_image(b64, prefix=f"xai_{model_id}") + except Exception as exc: + return error_response( + error=f"Could not save image to cache: {exc}", + error_type="io_error", + provider="xai", + model=model_id, + prompt=prompt, + aspect_ratio=aspect, + ) + image_ref = str(saved_path) + elif url: + image_ref = url + else: + return error_response( + error="xAI response contained neither b64_json nor URL", + error_type="empty_response", + provider="xai", + model=model_id, + prompt=prompt, + aspect_ratio=aspect, + ) + + extra: Dict[str, Any] = { + "resolution": xai_res, + } + + return success_response( + image=image_ref, + model=model_id, + prompt=prompt, + aspect_ratio=aspect, + provider="xai", + extra=extra, + ) + + +# --------------------------------------------------------------------------- +# Plugin registration +# --------------------------------------------------------------------------- + + +def register(ctx: Any) -> None: + """Register this provider with the image gen registry.""" + ctx.register_image_gen_provider(XAIImageGenProvider()) diff --git a/plugins/image_gen/xai/plugin.yaml b/plugins/image_gen/xai/plugin.yaml new file mode 100644 index 000000000000..1bebc7d725b0 --- /dev/null +++ b/plugins/image_gen/xai/plugin.yaml @@ -0,0 +1,7 @@ +name: xai +version: 1.0.0 +description: "xAI image generation backend (grok-imagine-image). Text-to-image." +author: Julien Talbot +kind: backend +requires_env: + - XAI_API_KEY diff --git a/plugins/memory/__init__.py b/plugins/memory/__init__.py index cd583e6d8dfd..0ae65a25d563 100644 --- a/plugins/memory/__init__.py +++ b/plugins/memory/__init__.py @@ -1,18 +1,22 @@ """Memory provider plugin discovery. -Scans ``plugins/memory//`` directories for memory provider plugins. +Scans two directories for memory provider plugins: + +1. Bundled providers: ``plugins/memory//`` (shipped with hermes-agent) +2. User-installed providers: ``$HERMES_HOME/plugins//`` + Each subdirectory must contain ``__init__.py`` with a class implementing -the MemoryProvider ABC. +the MemoryProvider ABC. On name collisions, bundled providers take +precedence. -Memory providers are separate from the general plugin system — they live -in the repo and are always available without user installation. Only ONE -can be active at a time, selected via ``memory.provider`` in config.yaml. +Only ONE provider can be active at a time, selected via +``memory.provider`` in config.yaml. Usage: from plugins.memory import discover_memory_providers, load_memory_provider available = discover_memory_providers() # [(name, desc, available), ...] - provider = load_memory_provider("openviking") # MemoryProvider instance + provider = load_memory_provider("mnemosyne") # MemoryProvider instance """ from __future__ import annotations @@ -29,24 +33,101 @@ _MEMORY_PLUGINS_DIR = Path(__file__).parent +# --------------------------------------------------------------------------- +# Directory helpers +# --------------------------------------------------------------------------- + +def _get_user_plugins_dir() -> Optional[Path]: + """Return ``$HERMES_HOME/plugins/`` or None if unavailable.""" + try: + from hermes_constants import get_hermes_home + d = get_hermes_home() / "plugins" + return d if d.is_dir() else None + except Exception: + return None + + +def _is_memory_provider_dir(path: Path) -> bool: + """Heuristic: does *path* look like a memory provider plugin? + + Checks for ``register_memory_provider`` or ``MemoryProvider`` in the + ``__init__.py`` source. Cheap text scan — no import needed. + """ + init_file = path / "__init__.py" + if not init_file.exists(): + return False + try: + source = init_file.read_text(errors="replace")[:8192] + return "register_memory_provider" in source or "MemoryProvider" in source + except Exception: + return False + + +def _iter_provider_dirs() -> List[Tuple[str, Path]]: + """Yield ``(name, path)`` for all discovered provider directories. + + Scans bundled first, then user-installed. Bundled takes precedence + on name collisions (first-seen wins via ``seen`` set). + """ + seen: set = set() + dirs: List[Tuple[str, Path]] = [] + + # 1. Bundled providers (plugins/memory//) + if _MEMORY_PLUGINS_DIR.is_dir(): + for child in sorted(_MEMORY_PLUGINS_DIR.iterdir()): + if not child.is_dir() or child.name.startswith(("_", ".")): + continue + if not (child / "__init__.py").exists(): + continue + seen.add(child.name) + dirs.append((child.name, child)) + + # 2. User-installed providers ($HERMES_HOME/plugins//) + user_dir = _get_user_plugins_dir() + if user_dir: + for child in sorted(user_dir.iterdir()): + if not child.is_dir() or child.name.startswith(("_", ".")): + continue + if child.name in seen: + continue # bundled takes precedence + if not _is_memory_provider_dir(child): + continue # skip non-memory plugins + dirs.append((child.name, child)) + + return dirs + + +def find_provider_dir(name: str) -> Optional[Path]: + """Resolve a provider name to its directory. + + Checks bundled first, then user-installed. + """ + # Bundled + bundled = _MEMORY_PLUGINS_DIR / name + if bundled.is_dir() and (bundled / "__init__.py").exists(): + return bundled + # User-installed + user_dir = _get_user_plugins_dir() + if user_dir: + user = user_dir / name + if user.is_dir() and _is_memory_provider_dir(user): + return user + return None + + +# --------------------------------------------------------------------------- +# Public API +# --------------------------------------------------------------------------- + def discover_memory_providers() -> List[Tuple[str, str, bool]]: - """Scan plugins/memory/ for available providers. + """Scan bundled and user-installed directories for available providers. Returns list of (name, description, is_available) tuples. - Does NOT import the providers — just reads plugin.yaml for metadata - and does a lightweight availability check. + Bundled providers take precedence on name collisions. """ results = [] - if not _MEMORY_PLUGINS_DIR.is_dir(): - return results - - for child in sorted(_MEMORY_PLUGINS_DIR.iterdir()): - if not child.is_dir() or child.name.startswith(("_", ".")): - continue - init_file = child / "__init__.py" - if not init_file.exists(): - continue + for name, child in _iter_provider_dirs(): # Read description from plugin.yaml if available desc = "" yaml_file = child / "plugin.yaml" @@ -70,7 +151,7 @@ def discover_memory_providers() -> List[Tuple[str, str, bool]]: except Exception: available = False - results.append((child.name, desc, available)) + results.append((name, desc, available)) return results @@ -78,11 +159,15 @@ def discover_memory_providers() -> List[Tuple[str, str, bool]]: def load_memory_provider(name: str) -> Optional["MemoryProvider"]: """Load and return a MemoryProvider instance by name. + Checks both bundled (``plugins/memory//``) and user-installed + (``$HERMES_HOME/plugins//``) directories. Bundled takes + precedence on name collisions. + Returns None if the provider is not found or fails to load. """ - provider_dir = _MEMORY_PLUGINS_DIR / name - if not provider_dir.is_dir(): - logger.debug("Memory provider '%s' not found in %s", name, _MEMORY_PLUGINS_DIR) + provider_dir = find_provider_dir(name) + if not provider_dir: + logger.debug("Memory provider '%s' not found in bundled or user plugins", name) return None try: @@ -104,7 +189,10 @@ def _load_provider_from_dir(provider_dir: Path) -> Optional["MemoryProvider"]: - A top-level class that extends MemoryProvider — we instantiate it """ name = provider_dir.name - module_name = f"plugins.memory.{name}" + # Use a separate namespace for user-installed plugins so they don't + # collide with bundled providers in sys.modules. + _is_bundled = _MEMORY_PLUGINS_DIR in provider_dir.parents or provider_dir.parent == _MEMORY_PLUGINS_DIR + module_name = f"plugins.memory.{name}" if _is_bundled else f"_hermes_user_memory.{name}" init_file = provider_dir / "__init__.py" if not init_file.exists(): @@ -257,15 +345,16 @@ def discover_plugin_cli_commands() -> List[dict]: return results # Only look at the active provider's directory - plugin_dir = _MEMORY_PLUGINS_DIR / active_provider - if not plugin_dir.is_dir(): + plugin_dir = find_provider_dir(active_provider) + if not plugin_dir: return results cli_file = plugin_dir / "cli.py" if not cli_file.exists(): return results - module_name = f"plugins.memory.{active_provider}.cli" + _is_bundled = _MEMORY_PLUGINS_DIR in plugin_dir.parents or plugin_dir.parent == _MEMORY_PLUGINS_DIR + module_name = f"plugins.memory.{active_provider}.cli" if _is_bundled else f"_hermes_user_memory.{active_provider}.cli" try: # Import the CLI module (lightweight — no SDK needed) if module_name in sys.modules: diff --git a/plugins/memory/hindsight/README.md b/plugins/memory/hindsight/README.md index 024a99303125..3fbdc2aba43e 100644 --- a/plugins/memory/hindsight/README.md +++ b/plugins/memory/hindsight/README.md @@ -84,7 +84,10 @@ Config file: `~/.hermes/hindsight/config.json` | `retain_async` | `true` | Process retain asynchronously on the Hindsight server | | `retain_every_n_turns` | `1` | Retain every N turns (1 = every turn) | | `retain_context` | `conversation between Hermes Agent and the User` | Context label for retained memories | -| `tags` | — | Tags applied when storing memories | +| `retain_tags` | — | Default tags applied to retained memories; merged with per-call tool tags | +| `retain_source` | — | Optional `metadata.source` attached to retained memories | +| `retain_user_prefix` | `User` | Label used before user turns in auto-retained transcripts | +| `retain_assistant_prefix` | `Assistant` | Label used before assistant turns in auto-retained transcripts | ### Integration @@ -113,7 +116,7 @@ Available in `hybrid` and `tools` memory modes: | Tool | Description | |------|-------------| -| `hindsight_retain` | Store information with auto entity extraction | +| `hindsight_retain` | Store information with auto entity extraction; supports optional per-call `tags` | | `hindsight_recall` | Multi-strategy search (semantic + entity graph) | | `hindsight_reflect` | Cross-memory synthesis (LLM-powered) | diff --git a/plugins/memory/hindsight/__init__.py b/plugins/memory/hindsight/__init__.py index c39679b73c8f..2b233e265caa 100644 --- a/plugins/memory/hindsight/__init__.py +++ b/plugins/memory/hindsight/__init__.py @@ -6,11 +6,15 @@ Original PR #1811 by benfrank241, adapted to MemoryProvider ABC. Config via environment variables: - HINDSIGHT_API_KEY — API key for Hindsight Cloud - HINDSIGHT_BANK_ID — memory bank identifier (default: hermes) - HINDSIGHT_BUDGET — recall budget: low/mid/high (default: mid) - HINDSIGHT_API_URL — API endpoint - HINDSIGHT_MODE — cloud or local (default: cloud) + HINDSIGHT_API_KEY — API key for Hindsight Cloud + HINDSIGHT_BANK_ID — memory bank identifier (default: hermes) + HINDSIGHT_BUDGET — recall budget: low/mid/high (default: mid) + HINDSIGHT_API_URL — API endpoint + HINDSIGHT_MODE — cloud or local (default: cloud) + HINDSIGHT_RETAIN_TAGS — comma-separated tags attached to retained memories + HINDSIGHT_RETAIN_SOURCE — metadata source value attached to retained memories + HINDSIGHT_RETAIN_USER_PREFIX — label used before user turns in retained transcripts + HINDSIGHT_RETAIN_ASSISTANT_PREFIX — label used before assistant turns in retained transcripts Or via $HERMES_HOME/hindsight/config.json (profile-scoped), falling back to ~/.hindsight/config.json (legacy, shared) for backward compatibility. @@ -24,7 +28,7 @@ import os import threading -from hermes_constants import get_hermes_home +from datetime import datetime, timezone from typing import Any, Dict, List from agent.memory_provider import MemoryProvider @@ -99,6 +103,11 @@ def _run_sync(coro, timeout: float = 120.0): "properties": { "content": {"type": "string", "description": "The information to store."}, "context": {"type": "string", "description": "Short label (e.g. 'user preference', 'project decision')."}, + "tags": { + "type": "array", + "items": {"type": "string"}, + "description": "Optional per-call tags to merge with configured default retain tags.", + }, }, "required": ["content"], }, @@ -168,6 +177,10 @@ def _load_config() -> dict: return { "mode": os.environ.get("HINDSIGHT_MODE", "cloud"), "apiKey": os.environ.get("HINDSIGHT_API_KEY", ""), + "retain_tags": os.environ.get("HINDSIGHT_RETAIN_TAGS", ""), + "retain_source": os.environ.get("HINDSIGHT_RETAIN_SOURCE", ""), + "retain_user_prefix": os.environ.get("HINDSIGHT_RETAIN_USER_PREFIX", "User"), + "retain_assistant_prefix": os.environ.get("HINDSIGHT_RETAIN_ASSISTANT_PREFIX", "Assistant"), "banks": { "hermes": { "bankId": os.environ.get("HINDSIGHT_BANK_ID", "hermes"), @@ -178,6 +191,48 @@ def _load_config() -> dict: } +def _normalize_retain_tags(value: Any) -> List[str]: + """Normalize tag config/tool values to a deduplicated list of strings.""" + if value is None: + return [] + + raw_items: list[Any] + if isinstance(value, list): + raw_items = value + elif isinstance(value, str): + text = value.strip() + if not text: + return [] + if text.startswith("["): + try: + parsed = json.loads(text) + except Exception: + parsed = None + if isinstance(parsed, list): + raw_items = parsed + else: + raw_items = text.split(",") + else: + raw_items = text.split(",") + else: + raw_items = [value] + + normalized = [] + seen = set() + for item in raw_items: + tag = str(item).strip() + if not tag or tag in seen: + continue + seen.add(tag) + normalized.append(tag) + return normalized + + +def _utc_timestamp() -> str: + """Return current UTC timestamp in ISO-8601 with milliseconds and Z suffix.""" + return datetime.now(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z") + + # --------------------------------------------------------------------------- # MemoryProvider implementation # --------------------------------------------------------------------------- @@ -195,6 +250,19 @@ def __init__(self): self._llm_base_url = "" self._memory_mode = "hybrid" # "context", "tools", or "hybrid" self._prefetch_method = "recall" # "recall" or "reflect" + self._retain_tags: List[str] = [] + self._retain_source = "" + self._retain_user_prefix = "User" + self._retain_assistant_prefix = "Assistant" + self._platform = "" + self._user_id = "" + self._user_name = "" + self._chat_id = "" + self._chat_name = "" + self._chat_type = "" + self._thread_id = "" + self._agent_identity = "" + self._turn_index = 0 self._client = None self._prefetch_result = "" self._prefetch_lock = threading.Lock() @@ -210,6 +278,7 @@ def __init__(self): # Retain controls self._auto_retain = True self._retain_every_n_turns = 1 + self._retain_async = True self._retain_context = "conversation between Hermes Agent and the User" self._turn_counter = 0 self._session_turns: list[str] = [] # accumulates ALL turns for the session @@ -224,7 +293,6 @@ def __init__(self): # Bank self._bank_mission = "" self._bank_retain_mission: str | None = None - self._retain_async = True @property def name(self) -> str: @@ -423,7 +491,10 @@ def get_config_schema(self): {"key": "recall_budget", "description": "Recall thoroughness", "default": "mid", "choices": ["low", "mid", "high"]}, {"key": "memory_mode", "description": "Memory integration mode", "default": "hybrid", "choices": ["hybrid", "context", "tools"]}, {"key": "recall_prefetch_method", "description": "Auto-recall method", "default": "recall", "choices": ["recall", "reflect"]}, - {"key": "tags", "description": "Tags applied when storing memories (comma-separated)", "default": ""}, + {"key": "retain_tags", "description": "Default tags applied to retained memories (comma-separated)", "default": ""}, + {"key": "retain_source", "description": "Metadata source value attached to retained memories", "default": ""}, + {"key": "retain_user_prefix", "description": "Label used before user turns in retained transcripts", "default": "User"}, + {"key": "retain_assistant_prefix", "description": "Label used before assistant turns in retained transcripts", "default": "Assistant"}, {"key": "recall_tags", "description": "Tags to filter when searching memories (comma-separated)", "default": ""}, {"key": "recall_tags_match", "description": "Tag matching mode for recall", "default": "any", "choices": ["any", "all", "any_strict", "all_strict"]}, {"key": "auto_recall", "description": "Automatically recall memories before each turn", "default": True}, @@ -467,7 +538,7 @@ def _get_client(self): return self._client def initialize(self, session_id: str, **kwargs) -> None: - self._session_id = session_id + self._session_id = str(session_id or "").strip() # Check client version and auto-upgrade if needed try: @@ -496,6 +567,16 @@ def initialize(self, session_id: str, **kwargs) -> None: pass # packaging not available or other issue — proceed anyway self._config = _load_config() + self._platform = str(kwargs.get("platform") or "").strip() + self._user_id = str(kwargs.get("user_id") or "").strip() + self._user_name = str(kwargs.get("user_name") or "").strip() + self._chat_id = str(kwargs.get("chat_id") or "").strip() + self._chat_name = str(kwargs.get("chat_name") or "").strip() + self._chat_type = str(kwargs.get("chat_type") or "").strip() + self._thread_id = str(kwargs.get("thread_id") or "").strip() + self._agent_identity = str(kwargs.get("agent_identity") or "").strip() + self._turn_index = 0 + self._session_turns = [] self._mode = self._config.get("mode", "cloud") # "local" is a legacy alias for "local_embedded" if self._mode == "local": @@ -513,7 +594,7 @@ def initialize(self, session_id: str, **kwargs) -> None: memory_mode = self._config.get("memory_mode", "hybrid") self._memory_mode = memory_mode if memory_mode in ("context", "tools", "hybrid") else "hybrid" - prefetch_method = self._config.get("recall_prefetch_method", "recall") + prefetch_method = self._config.get("recall_prefetch_method") or self._config.get("prefetch_method", "recall") self._prefetch_method = prefetch_method if prefetch_method in ("recall", "reflect") else "recall" # Bank options @@ -521,9 +602,22 @@ def initialize(self, session_id: str, **kwargs) -> None: self._bank_retain_mission = self._config.get("bank_retain_mission") or None # Tags - self._tags = self._config.get("tags") or None + self._retain_tags = _normalize_retain_tags( + self._config.get("retain_tags") + or os.environ.get("HINDSIGHT_RETAIN_TAGS", "") + ) + self._tags = self._retain_tags or None self._recall_tags = self._config.get("recall_tags") or None self._recall_tags_match = self._config.get("recall_tags_match", "any") + self._retain_source = str( + self._config.get("retain_source") or os.environ.get("HINDSIGHT_RETAIN_SOURCE", "") + ).strip() + self._retain_user_prefix = str( + self._config.get("retain_user_prefix") or os.environ.get("HINDSIGHT_RETAIN_USER_PREFIX", "User") + ).strip() or "User" + self._retain_assistant_prefix = str( + self._config.get("retain_assistant_prefix") or os.environ.get("HINDSIGHT_RETAIN_ASSISTANT_PREFIX", "Assistant") + ).strip() or "Assistant" # Retain controls self._auto_retain = self._config.get("auto_retain", True) @@ -547,11 +641,9 @@ def initialize(self, session_id: str, **kwargs) -> None: logger.info("Hindsight initialized: mode=%s, api_url=%s, bank=%s, budget=%s, memory_mode=%s, prefetch_method=%s, client=%s", self._mode, self._api_url, self._bank_id, self._budget, self._memory_mode, self._prefetch_method, _client_version) logger.debug("Hindsight config: auto_retain=%s, auto_recall=%s, retain_every_n=%d, " - "retain_async=%s, retain_context=%s, " - "recall_max_tokens=%d, recall_max_input_chars=%d, tags=%s, recall_tags=%s", + "retain_async=%s, retain_context=%s, recall_max_tokens=%d, recall_max_input_chars=%d, tags=%s, recall_tags=%s", self._auto_retain, self._auto_recall, self._retain_every_n_turns, - self._retain_async, self._retain_context, - self._recall_max_tokens, self._recall_max_input_chars, + self._retain_async, self._retain_context, self._recall_max_tokens, self._recall_max_input_chars, self._tags, self._recall_tags) # For local mode, start the embedded daemon in the background so it @@ -712,6 +804,78 @@ def _run(): self._prefetch_thread = threading.Thread(target=_run, daemon=True, name="hindsight-prefetch") self._prefetch_thread.start() + def _build_turn_messages(self, user_content: str, assistant_content: str) -> List[Dict[str, str]]: + now = datetime.now(timezone.utc).isoformat() + return [ + { + "role": "user", + "content": f"{self._retain_user_prefix}: {user_content}", + "timestamp": now, + }, + { + "role": "assistant", + "content": f"{self._retain_assistant_prefix}: {assistant_content}", + "timestamp": now, + }, + ] + + def _build_metadata(self, *, message_count: int, turn_index: int) -> Dict[str, str]: + metadata: Dict[str, str] = { + "retained_at": _utc_timestamp(), + "message_count": str(message_count), + "turn_index": str(turn_index), + } + if self._retain_source: + metadata["source"] = self._retain_source + if self._session_id: + metadata["session_id"] = self._session_id + if self._platform: + metadata["platform"] = self._platform + if self._user_id: + metadata["user_id"] = self._user_id + if self._user_name: + metadata["user_name"] = self._user_name + if self._chat_id: + metadata["chat_id"] = self._chat_id + if self._chat_name: + metadata["chat_name"] = self._chat_name + if self._chat_type: + metadata["chat_type"] = self._chat_type + if self._thread_id: + metadata["thread_id"] = self._thread_id + if self._agent_identity: + metadata["agent_identity"] = self._agent_identity + return metadata + + def _build_retain_kwargs( + self, + content: str, + *, + context: str | None = None, + document_id: str | None = None, + metadata: Dict[str, str] | None = None, + tags: List[str] | None = None, + retain_async: bool | None = None, + ) -> Dict[str, Any]: + kwargs: Dict[str, Any] = { + "bank_id": self._bank_id, + "content": content, + "metadata": metadata or self._build_metadata(message_count=1, turn_index=self._turn_index), + } + if context is not None: + kwargs["context"] = context + if document_id: + kwargs["document_id"] = document_id + if retain_async is not None: + kwargs["retain_async"] = retain_async + merged_tags = _normalize_retain_tags(self._retain_tags) + for tag in _normalize_retain_tags(tags): + if tag not in merged_tags: + merged_tags.append(tag) + if merged_tags: + kwargs["tags"] = merged_tags + return kwargs + def sync_turn(self, user_content: str, assistant_content: str, *, session_id: str = "") -> None: """Retain conversation turn in background (non-blocking). @@ -721,19 +885,14 @@ def sync_turn(self, user_content: str, assistant_content: str, *, session_id: st logger.debug("sync_turn: skipped (auto_retain disabled)") return - from datetime import datetime, timezone - now = datetime.now(timezone.utc).isoformat() + if session_id: + self._session_id = str(session_id).strip() - messages = [ - {"role": "user", "content": user_content, "timestamp": now}, - {"role": "assistant", "content": assistant_content, "timestamp": now}, - ] - - turn = json.dumps(messages) + turn = json.dumps(self._build_turn_messages(user_content, assistant_content)) self._session_turns.append(turn) self._turn_counter += 1 + self._turn_index = self._turn_counter - # Only retain every N turns if self._turn_counter % self._retain_every_n_turns != 0: logger.debug("sync_turn: buffered turn %d (will retain at turn %d)", self._turn_counter, self._turn_counter + (self._retain_every_n_turns - self._turn_counter % self._retain_every_n_turns)) @@ -741,19 +900,21 @@ def sync_turn(self, user_content: str, assistant_content: str, *, session_id: st logger.debug("sync_turn: retaining %d turns, total session content %d chars", len(self._session_turns), sum(len(t) for t in self._session_turns)) - # Send the ENTIRE session as a single JSON array (document_id deduplicates). - # Each element in _session_turns is a JSON string of that turn's messages. content = "[" + ",".join(self._session_turns) + "]" def _sync(): try: client = self._get_client() - item: dict = { - "content": content, - "context": self._retain_context, - } - if self._tags: - item["tags"] = self._tags + item = self._build_retain_kwargs( + content, + context=self._retain_context, + metadata=self._build_metadata( + message_count=len(self._session_turns) * 2, + turn_index=self._turn_index, + ), + ) + item.pop("bank_id", None) + item.pop("retain_async", None) logger.debug("Hindsight retain: bank=%s, doc=%s, async=%s, content_len=%d, num_turns=%d", self._bank_id, self._session_id, self._retain_async, len(content), len(self._session_turns)) _run_sync(client.aretain_batch( @@ -789,11 +950,11 @@ def handle_tool_call(self, tool_name: str, args: dict, **kwargs) -> str: return tool_error("Missing required parameter: content") context = args.get("context") try: - retain_kwargs: dict = { - "bank_id": self._bank_id, "content": content, "context": context, - } - if self._tags: - retain_kwargs["tags"] = self._tags + retain_kwargs = self._build_retain_kwargs( + content, + context=context, + tags=args.get("tags"), + ) logger.debug("Tool hindsight_retain: bank=%s, content_len=%d, context=%s", self._bank_id, len(content), context) _run_sync(client.aretain(**retain_kwargs)) diff --git a/plugins/memory/honcho/README.md b/plugins/memory/honcho/README.md index 80cc5a70aa03..4f8d10ea9ecb 100644 --- a/plugins/memory/honcho/README.md +++ b/plugins/memory/honcho/README.md @@ -1,6 +1,6 @@ # Honcho Memory Provider -AI-native cross-session user modeling with dialectic Q&A, semantic search, peer cards, and persistent conclusions. +AI-native cross-session user modeling with multi-pass dialectic reasoning, session summaries, bidirectional peer tools, and persistent conclusions. > **Honcho docs:** @@ -19,9 +19,86 @@ hermes memory setup # generic picker, also works Or manually: ```bash hermes config set memory.provider honcho -echo "HONCHO_API_KEY=your-key" >> ~/.hermes/.env +echo "HONCHO_API_KEY=***" >> ~/.hermes/.env ``` +## Architecture Overview + +### Two-Layer Context Injection + +Context is injected into the **user message** at API-call time (not the system prompt) to preserve prompt caching. Only a static mode header goes in the system prompt. The injected block is wrapped in `` fences with a system note clarifying it's background data, not new user input. + +Two independent layers, each on its own cadence: + +**Layer 1 — Base context** (refreshed every `contextCadence` turns): +1. **SESSION SUMMARY** — from `session.context(summary=True)`, placed first +2. **User Representation** — Honcho's evolving model of the user +3. **User Peer Card** — key facts snapshot +4. **AI Self-Representation** — Honcho's model of the AI peer +5. **AI Identity Card** — AI peer facts + +**Layer 2 — Dialectic supplement** (fired every `dialecticCadence` turns): +Multi-pass `.chat()` reasoning about the user, appended after base context. + +Both layers are joined, then truncated to fit `contextTokens` budget via `_truncate_to_budget` (tokens × 4 chars, word-boundary safe). + +### Cold Start vs Warm Session Prompts + +Dialectic pass 0 automatically selects its prompt based on session state: + +- **Cold** (no base context cached): "Who is this person? What are their preferences, goals, and working style? Focus on facts that would help an AI assistant be immediately useful." +- **Warm** (base context exists): "Given what's been discussed in this session so far, what context about this user is most relevant to the current conversation? Prioritize active context over biographical facts." + +Not configurable — determined automatically. + +### Dialectic Depth (Multi-Pass Reasoning) + +`dialecticDepth` (1–3, clamped) controls how many `.chat()` calls fire per dialectic cycle: + +| Depth | Passes | Behavior | +|-------|--------|----------| +| 1 | single `.chat()` | Base query only (cold or warm prompt) | +| 2 | audit + synthesis | Pass 0 result is self-audited; pass 1 does targeted synthesis. Conditional bail-out if pass 0 returns strong signal (>300 chars or structured with bullets/sections >100 chars) | +| 3 | audit + synthesis + reconciliation | Pass 2 reconciles contradictions across prior passes into a final synthesis | + +### Proportional Reasoning Levels + +When `dialecticDepthLevels` is not set, each pass uses a proportional level relative to `dialecticReasoningLevel` (the "base"): + +| Depth | Pass levels | +|-------|-------------| +| 1 | [base] | +| 2 | [minimal, base] | +| 3 | [minimal, base, low] | + +Override with `dialecticDepthLevels`: an explicit array of reasoning level strings per pass. + +### Three Orthogonal Dialectic Knobs + +| Knob | Controls | Type | +|------|----------|------| +| `dialecticCadence` | How often — minimum turns between dialectic firings | int | +| `dialecticDepth` | How many — passes per firing (1–3) | int | +| `dialecticReasoningLevel` | How hard — reasoning ceiling per `.chat()` call | string | + +### Input Sanitization + +`run_conversation` strips leaked `` blocks from user input before processing. When `saveMessages` persists a turn that included injected context, the block can reappear in subsequent turns via message history. The sanitizer removes `` blocks plus associated system notes. + +## Tools + +Five bidirectional tools. All accept an optional `peer` parameter (`"user"` or `"ai"`, default `"user"`). + +| Tool | LLM call? | Description | +|------|-----------|-------------| +| `honcho_profile` | No | Peer card — key facts snapshot | +| `honcho_search` | No | Semantic search over stored context (800 tok default, 2000 max) | +| `honcho_context` | No | Full session context: summary, representation, card, messages | +| `honcho_reasoning` | Yes | LLM-synthesized answer via dialectic `.chat()` | +| `honcho_conclude` | No | Write a persistent fact/conclusion about the user | + +Tool visibility depends on `recallMode`: hidden in `context` mode, always present in `tools` and `hybrid`. + ## Config Resolution Config is read from the first file that exists: @@ -34,125 +111,153 @@ Config is read from the first file that exists: Host key is derived from the active Hermes profile: `hermes` (default) or `hermes.`. -## Tools - -| Tool | LLM call? | Description | -|------|-----------|-------------| -| `honcho_profile` | No | User's peer card -- key facts snapshot | -| `honcho_search` | No | Semantic search over stored context (800 tok default, 2000 max) | -| `honcho_context` | Yes | LLM-synthesized answer via dialectic reasoning | -| `honcho_conclude` | No | Write a persistent fact about the user | - -Tool availability depends on `recallMode`: hidden in `context` mode, always present in `tools` and `hybrid`. +For every key, resolution order is: **host block > root > env var > default**. ## Full Configuration Reference ### Identity & Connection -| Key | Type | Default | Scope | Description | -|-----|------|---------|-------|-------------| -| `apiKey` | string | -- | root / host | API key. Falls back to `HONCHO_API_KEY` env var | -| `baseUrl` | string | -- | root | Base URL for self-hosted Honcho. Local URLs (`localhost`, `127.0.0.1`, `::1`) auto-skip API key auth | -| `environment` | string | `"production"` | root / host | SDK environment mapping | -| `enabled` | bool | auto | root / host | Master toggle. Auto-enables when `apiKey` or `baseUrl` present | -| `workspace` | string | host key | root / host | Honcho workspace ID | -| `peerName` | string | -- | root / host | User peer identity | -| `aiPeer` | string | host key | root / host | AI peer identity | +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `apiKey` | string | — | API key. Falls back to `HONCHO_API_KEY` env var | +| `baseUrl` | string | — | Base URL for self-hosted Honcho. Local URLs auto-skip API key auth | +| `environment` | string | `"production"` | SDK environment mapping | +| `enabled` | bool | auto | Master toggle. Auto-enables when `apiKey` or `baseUrl` present | +| `workspace` | string | host key | Honcho workspace ID. Shared environment — all profiles in the same workspace can see the same user identity and related memories | +| `peerName` | string | — | User peer identity | +| `aiPeer` | string | host key | AI peer identity | ### Memory & Recall -| Key | Type | Default | Scope | Description | -|-----|------|---------|-------|-------------| -| `recallMode` | string | `"hybrid"` | root / host | `"hybrid"` (auto-inject + tools), `"context"` (auto-inject only, tools hidden), `"tools"` (tools only, no injection). Legacy `"auto"` normalizes to `"hybrid"` | -| `observationMode` | string | `"directional"` | root / host | Shorthand preset: `"directional"` (all on) or `"unified"` (shared pool). Use `observation` object for granular control | -| `observation` | object | -- | root / host | Per-peer observation config (see below) | +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `recallMode` | string | `"hybrid"` | `"hybrid"` (auto-inject + tools), `"context"` (auto-inject only, tools hidden), `"tools"` (tools only, no injection). Legacy `"auto"` → `"hybrid"` | +| `observationMode` | string | `"directional"` | Preset: `"directional"` (all on) or `"unified"` (shared pool). Use `observation` object for granular control | +| `observation` | object | — | Per-peer observation config (see Observation section) | -#### Observation (granular) +### Write Behavior -Maps 1:1 to Honcho's per-peer `SessionPeerConfig`. Set at root or per host block -- each profile can have different observation settings. When present, overrides `observationMode` preset. +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `writeFrequency` | string/int | `"async"` | `"async"` (background), `"turn"` (sync per turn), `"session"` (batch on end), or integer N (every N turns) | +| `saveMessages` | bool | `true` | Persist messages to Honcho API | -```json -"observation": { - "user": { "observeMe": true, "observeOthers": true }, - "ai": { "observeMe": true, "observeOthers": true } -} -``` +### Session Resolution -| Field | Default | Description | -|-------|---------|-------------| -| `user.observeMe` | `true` | User peer self-observation (Honcho builds user representation) | -| `user.observeOthers` | `true` | User peer observes AI messages | -| `ai.observeMe` | `true` | AI peer self-observation (Honcho builds AI representation) | -| `ai.observeOthers` | `true` | AI peer observes user messages (enables cross-peer dialectic) | +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `sessionStrategy` | string | `"per-directory"` | `"per-directory"`, `"per-session"`, `"per-repo"` (git root), `"global"` | +| `sessionPeerPrefix` | bool | `false` | Prepend peer name to session keys | +| `sessions` | object | `{}` | Manual directory-to-session-name mappings | -Presets for `observationMode`: -- `"directional"` (default): all four booleans `true` -- `"unified"`: user `observeMe=true`, AI `observeOthers=true`, rest `false` +#### Session Name Resolution + +The Honcho session name determines which conversation bucket memory lands in. Resolution follows a priority chain — first match wins: + +| Priority | Source | Example session name | +|----------|--------|---------------------| +| 1 | Manual map (`sessions` config) | `"myproject-main"` | +| 2 | `/title` command (mid-session rename) | `"refactor-auth"` | +| 3 | Gateway session key (Telegram, Discord, etc.) | `"agent-main-telegram-dm-8439114563"` | +| 4 | `per-session` strategy | Hermes session ID (`20260415_a3f2b1`) | +| 5 | `per-repo` strategy | Git root directory name (`hermes-agent`) | +| 6 | `per-directory` strategy | Current directory basename (`src`) | +| 7 | `global` strategy | Workspace name (`hermes`) | + +Gateway platforms always resolve via priority 3 (per-chat isolation) regardless of `sessionStrategy`. The strategy setting only affects CLI sessions. -Per-profile example -- coder profile observes the user but user doesn't observe coder: +If `sessionPeerPrefix` is `true`, the peer name is prepended: `eri-hermes-agent`. + +#### What each strategy produces + +- **`per-directory`** — basename of `$PWD`. Opening hermes in `~/code/myapp` and `~/code/other` gives two separate sessions. Same directory = same session across runs. +- **`per-repo`** — git root directory name. All subdirectories within a repo share one session. Falls back to `per-directory` if not inside a git repo. +- **`per-session`** — Hermes session ID (timestamp + hex). Every `hermes` invocation starts a fresh Honcho session. Falls back to `per-directory` if no session ID is available. +- **`global`** — workspace name. One session for everything. Memory accumulates across all directories and runs. + +### Multi-Profile Pattern + +Multiple Hermes profiles can share one workspace while maintaining separate AI identities. Config resolution is **host block > root > env var > default** — host blocks inherit from root, so shared settings only need to be declared once: ```json -"hosts": { - "hermes.coder": { - "observation": { - "user": { "observeMe": true, "observeOthers": false }, - "ai": { "observeMe": true, "observeOthers": true } +{ + "apiKey": "***", + "workspace": "hermes", + "peerName": "yourname", + "hosts": { + "hermes": { + "aiPeer": "hermes", + "recallMode": "hybrid", + "sessionStrategy": "per-directory" + }, + "hermes.coder": { + "aiPeer": "coder", + "recallMode": "tools", + "sessionStrategy": "per-repo" } } } ``` -Settings changed in the [Honcho dashboard](https://app.honcho.dev) are synced back on session init. - -### Write Behavior +Both profiles see the same user (`yourname`) in the same shared environment (`hermes`), but each AI peer builds its own observations, conclusions, and behavior patterns. The coder's memory stays code-oriented; the main agent's stays broad. -| Key | Type | Default | Scope | Description | -|-----|------|---------|-------|-------------| -| `writeFrequency` | string or int | `"async"` | root / host | `"async"` (background thread), `"turn"` (sync per turn), `"session"` (batch on end), or integer N (every N turns) | -| `saveMessages` | bool | `true` | root / host | Whether to persist messages to Honcho API | +Host key is derived from the active Hermes profile: `hermes` (default) or `hermes.` (e.g. `hermes -p coder` → host key `hermes.coder`). -### Session Resolution +### Dialectic & Reasoning -| Key | Type | Default | Scope | Description | -|-----|------|---------|-------|-------------| -| `sessionStrategy` | string | `"per-directory"` | root / host | `"per-directory"`, `"per-session"` (new each run), `"per-repo"` (git root name), `"global"` (single session) | -| `sessionPeerPrefix` | bool | `false` | root / host | Prepend peer name to session keys | -| `sessions` | object | `{}` | root | Manual directory-to-session-name mappings: `{"/path/to/project": "my-session"}` | - -### Token Budgets & Dialectic +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `dialecticDepth` | int | `1` | Passes per dialectic cycle (1–3, clamped). 1=single query, 2=audit+synthesis, 3=audit+synthesis+reconciliation | +| `dialecticDepthLevels` | array | — | Optional array of reasoning level strings per pass. Overrides proportional defaults. Example: `["minimal", "low", "medium"]` | +| `dialecticReasoningLevel` | string | `"low"` | Base reasoning level for `.chat()`: `"minimal"`, `"low"`, `"medium"`, `"high"`, `"max"` | +| `dialecticDynamic` | bool | `true` | When `true`, model can override reasoning level per-call via `honcho_reasoning` tool. When `false`, always uses `dialecticReasoningLevel` | +| `dialecticMaxChars` | int | `600` | Max chars of dialectic result injected into system prompt | +| `dialecticMaxInputChars` | int | `10000` | Max chars for dialectic query input to `.chat()`. Honcho cloud limit: 10k | -| Key | Type | Default | Scope | Description | -|-----|------|---------|-------|-------------| -| `contextTokens` | int | SDK default | root / host | Token budget for `context()` API calls. Also gates prefetch truncation (tokens x 4 chars) | -| `dialecticReasoningLevel` | string | `"low"` | root / host | Base reasoning level for `peer.chat()`: `"minimal"`, `"low"`, `"medium"`, `"high"`, `"max"` | -| `dialecticDynamic` | bool | `true` | root / host | Auto-bump reasoning based on query length: `<120` chars = base level, `120-400` = +1, `>400` = +2 (capped at `"high"`). Set `false` to always use `dialecticReasoningLevel` as-is | -| `dialecticMaxChars` | int | `600` | root / host | Max chars of dialectic result injected into system prompt | -| `dialecticMaxInputChars` | int | `10000` | root / host | Max chars for dialectic query input to `peer.chat()`. Honcho cloud limit: 10k | -| `messageMaxChars` | int | `25000` | root / host | Max chars per message sent via `add_messages()`. Messages exceeding this are chunked with `[continued]` markers. Honcho cloud limit: 25k | +### Token Budgets -### Cost Awareness (Advanced) +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `contextTokens` | int | SDK default | Token budget for `context()` API calls. Also gates prefetch truncation (tokens × 4 chars) | +| `messageMaxChars` | int | `25000` | Max chars per message sent via `add_messages()`. Exceeding this triggers chunking with `[continued]` markers. Honcho cloud limit: 25k | -These are read from the root config object, not the host block. Must be set manually in `honcho.json`. +### Cadence (Cost Control) | Key | Type | Default | Description | |-----|------|---------|-------------| -| `injectionFrequency` | string | `"every-turn"` | `"every-turn"` or `"first-turn"` (inject context only on turn 0) | -| `contextCadence` | int | `1` | Minimum turns between `context()` API calls | -| `dialecticCadence` | int | `1` | Minimum turns between `peer.chat()` API calls | -| `reasoningLevelCap` | string | -- | Hard cap on auto-bumped reasoning: `"minimal"`, `"low"`, `"mid"`, `"high"` | +| `contextCadence` | int | `1` | Minimum turns between base context refreshes (session summary + representation + card) | +| `dialecticCadence` | int | `1` | Minimum turns between dialectic `.chat()` firings | +| `injectionFrequency` | string | `"every-turn"` | `"every-turn"` or `"first-turn"` (inject context on the first user message only, skip from turn 2 onward) | +| `reasoningLevelCap` | string | — | Hard cap on reasoning level: `"minimal"`, `"low"`, `"medium"`, `"high"` | -### Hardcoded Limits (Not Configurable) +### Observation (Granular) -| Limit | Value | Location | -|-------|-------|----------| -| Search tool max tokens | 2000 (hard cap), 800 (default) | `__init__.py` handle_tool_call | -| Peer card fetch tokens | 200 | `session.py` get_peer_card | +Maps 1:1 to Honcho's per-peer `SessionPeerConfig`. When present, overrides `observationMode` preset. -## Config Precedence +```json +"observation": { + "user": { "observeMe": true, "observeOthers": true }, + "ai": { "observeMe": true, "observeOthers": true } +} +``` -For every key, resolution order is: **host block > root > env var > default**. +| Field | Default | Description | +|-------|---------|-------------| +| `user.observeMe` | `true` | User peer self-observation (Honcho builds user representation) | +| `user.observeOthers` | `true` | User peer observes AI messages | +| `ai.observeMe` | `true` | AI peer self-observation (Honcho builds AI representation) | +| `ai.observeOthers` | `true` | AI peer observes user messages (enables cross-peer dialectic) | + +Presets: +- `"directional"` (default): all four `true` +- `"unified"`: user `observeMe=true`, AI `observeOthers=true`, rest `false` + +### Hardcoded Limits -Host key derivation: `HERMES_HONCHO_HOST` env > active profile (`hermes.`) > `"hermes"`. +| Limit | Value | +|-------|-------| +| Search tool max tokens | 2000 (hard cap), 800 (default) | +| Peer card fetch tokens | 200 | ## Environment Variables @@ -182,15 +287,16 @@ Host key derivation: `HERMES_HONCHO_HOST` env > active profile (`hermes. active profile (`hermes. None: logger.debug("Honcho not configured — plugin inactive") return - # Override peer_name with gateway user_id for per-user memory scoping. - # Only when no explicit peerName was configured — an explicit peerName - # means the user chose their identity; a raw user_id (e.g. Telegram - # chat ID) should not silently replace it. - _gw_user_id = kwargs.get("user_id") - if _gw_user_id and not cfg.peer_name: - cfg.peer_name = _gw_user_id - self._config = cfg # ----- B1: recall_mode from config ----- @@ -236,10 +304,16 @@ def initialize(self, session_id: str, **kwargs) -> None: raw = cfg.raw or {} self._injection_frequency = raw.get("injectionFrequency", "every-turn") self._context_cadence = int(raw.get("contextCadence", 1)) + # Backwards-compat: unset dialecticCadence falls back to 1 + # (every turn) so existing honcho.json configs without the key + # behave as they did before. New setups via `hermes honcho setup` + # get dialecticCadence=2 written explicitly by the wizard. self._dialectic_cadence = int(raw.get("dialecticCadence", 1)) - cap = raw.get("reasoningLevelCap") - if cap and cap in ("minimal", "low", "mid", "high"): - self._reasoning_level_cap = cap + self._dialectic_depth = max(1, min(cfg.dialectic_depth, 3)) + self._dialectic_depth_levels = cfg.dialectic_depth_levels + self._reasoning_heuristic = cfg.reasoning_heuristic + if cfg.reasoning_level_cap in self._LEVEL_ORDER: + self._reasoning_level_cap = cfg.reasoning_level_cap except Exception as e: logger.debug("Honcho cost-awareness config parse error: %s", e) @@ -251,9 +325,7 @@ def initialize(self, session_id: str, **kwargs) -> None: # ----- Port #1957: lazy session init for tools-only mode ----- if self._recall_mode == "tools": if cfg.init_on_session_start: - # Eager init: create session now so sync_turn() works from turn 1. - # Does NOT enable auto-injection — prefetch() still returns empty. - logger.debug("Honcho tools-only mode — eager session init (initOnSessionStart=true)") + # Eager init even in tools mode (opt-in) self._do_session_init(cfg, session_id, **kwargs) return # Defer actual session creation until first tool call @@ -283,12 +355,18 @@ def _do_session_init(self, cfg, session_id: str, **kwargs) -> None: honcho=client, config=cfg, context_tokens=cfg.context_tokens, + runtime_user_peer_name=kwargs.get("user_id") or None, ) # ----- B3: resolve_session_name ----- session_title = kwargs.get("session_title") + gateway_session_key = kwargs.get("gateway_session_key") self._session_key = ( - cfg.resolve_session_name(session_title=session_title, session_id=session_id) + cfg.resolve_session_name( + session_title=session_title, + session_id=session_id, + gateway_session_key=gateway_session_key, + ) or session_id or "hermes-default" ) @@ -299,23 +377,63 @@ def _do_session_init(self, cfg, session_id: str, **kwargs) -> None: self._session_initialized = True # ----- B6: Memory file migration (one-time, for new sessions) ----- + # Skip under per-session strategy: every Hermes run creates a fresh + # Honcho session by design, so uploading MEMORY.md/USER.md/SOUL.md to + # each one would flood the backend with short-lived duplicates instead + # of performing a one-time migration. try: - if not session.messages: + if not session.messages and cfg.session_strategy != "per-session": from hermes_constants import get_hermes_home mem_dir = str(get_hermes_home() / "memories") self._manager.migrate_memory_files(self._session_key, mem_dir) logger.debug("Honcho memory file migration attempted for new session: %s", self._session_key) + elif cfg.session_strategy == "per-session": + logger.debug( + "Honcho memory file migration skipped: per-session strategy creates a fresh session per run (%s)", + self._session_key, + ) except Exception as e: logger.debug("Honcho memory file migration skipped: %s", e) - # ----- B7: Pre-warming context at init ----- + # ----- B7: Pre-warming at init ----- + # Context prewarm warms peer.context() (base layer), consumed via + # pop_context_result() in prefetch(). Dialectic prewarm runs the + # full configured depth and writes into _prefetch_result so turn 1 + # consumes the result directly. if self._recall_mode in ("context", "hybrid"): try: self._manager.prefetch_context(self._session_key) - self._manager.prefetch_dialectic(self._session_key, "What should I know about this user?") - logger.debug("Honcho pre-warm threads started for session: %s", self._session_key) except Exception as e: - logger.debug("Honcho pre-warm failed: %s", e) + logger.debug("Honcho context prewarm failed: %s", e) + + _prewarm_query = ( + "Summarize what you know about this user. " + "Focus on preferences, current projects, and working style." + ) + + def _prewarm_dialectic() -> None: + try: + r = self._run_dialectic_depth(_prewarm_query) + except Exception as exc: + logger.debug("Honcho dialectic prewarm failed: %s", exc) + self._dialectic_empty_streak += 1 + return + if r and r.strip(): + with self._prefetch_lock: + self._prefetch_result = r + self._prefetch_result_fired_at = 0 + # Treat prewarm as turn 0 so cadence gating starts clean. + self._last_dialectic_turn = 0 + self._dialectic_empty_streak = 0 + else: + self._dialectic_empty_streak += 1 + + self._prefetch_thread_started_at = time.monotonic() + self._prefetch_thread = threading.Thread( + target=_prewarm_dialectic, daemon=True, name="honcho-prewarm-dialectic" + ) + self._prefetch_thread.start() + logger.debug("Honcho pre-warm started for session: %s", self._session_key) def _ensure_session(self) -> bool: """Lazily initialize the Honcho session (for tools-only mode). @@ -347,6 +465,11 @@ def _format_first_turn_context(self, ctx: dict) -> str: """Format the prefetch context dict into a readable system prompt block.""" parts = [] + # Session summary — session-scoped context, placed first for relevance + summary = ctx.get("summary", "") + if summary: + parts.append(f"## Session Summary\n{summary}") + rep = ctx.get("representation", "") if rep: parts.append(f"## User Representation\n{rep}") @@ -370,9 +493,9 @@ def _format_first_turn_context(self, ctx: dict) -> str: def system_prompt_block(self) -> str: """Return system prompt text, adapted by recall_mode. - B4: On the FIRST call, fetch and bake the full Honcho context - (user representation, peer card, AI representation, continuity synthesis). - Subsequent calls return the cached block for prompt caching stability. + Returns only the mode header and tool instructions — static text + that doesn't change between turns (prompt-cache friendly). + Live context (representation, card) is injected via prefetch(). """ if self._cron_skipped: return "" @@ -382,24 +505,10 @@ def system_prompt_block(self) -> str: return ( "# Honcho Memory\n" "Active (tools-only mode). Use honcho_profile, honcho_search, " - "honcho_context, and honcho_conclude tools to access user memory." + "honcho_reasoning, honcho_context, and honcho_conclude tools to access user memory." ) return "" - # ----- B4: First-turn context baking ----- - first_turn_block = "" - if self._recall_mode in ("context", "hybrid"): - with self._first_turn_lock: - if self._first_turn_context is None: - # First call — fetch and cache - try: - ctx = self._manager.get_prefetch_context(self._session_key) - self._first_turn_context = self._format_first_turn_context(ctx) if ctx else "" - except Exception as e: - logger.debug("Honcho first-turn context fetch failed: %s", e) - self._first_turn_context = "" - first_turn_block = self._first_turn_context - # ----- B1: adapt text based on recall_mode ----- if self._recall_mode == "context": header = ( @@ -412,7 +521,9 @@ def system_prompt_block(self) -> str: header = ( "# Honcho Memory\n" "Active (tools-only mode). Use honcho_profile for a quick factual snapshot, " - "honcho_search for raw excerpts, honcho_context for synthesized answers, " + "honcho_search for raw excerpts, honcho_context for raw peer context, " + "honcho_reasoning for synthesized answers (pass reasoning_level " + "minimal/low/medium/high/max — you pick the depth per call), " "honcho_conclude to save facts about the user. " "No automatic context injection — you must use tools to access memory." ) @@ -421,16 +532,20 @@ def system_prompt_block(self) -> str: "# Honcho Memory\n" "Active (hybrid mode). Relevant context is auto-injected AND memory tools are available. " "Use honcho_profile for a quick factual snapshot, " - "honcho_search for raw excerpts, honcho_context for synthesized answers, " + "honcho_search for raw excerpts, honcho_context for raw peer context, " + "honcho_reasoning for synthesized answers (pass reasoning_level " + "minimal/low/medium/high/max — you pick the depth per call), " "honcho_conclude to save facts about the user." ) - if first_turn_block: - return f"{header}\n\n{first_turn_block}" return header def prefetch(self, query: str, *, session_id: str = "") -> str: - """Return prefetched dialectic context from background thread. + """Return base context (representation + card) plus dialectic supplement. + + Assembles two layers: + 1. Base context from peer.context() — cached, refreshed on context_cadence + 2. Dialectic supplement — cached, refreshed on dialectic_cadence B1: Returns empty when recall_mode is "tools" (no injection). B5: Respects injection_frequency — "first-turn" returns cached/empty after turn 0. @@ -443,22 +558,128 @@ def prefetch(self, query: str, *, session_id: str = "") -> str: if self._recall_mode == "tools": return "" - # B5: injection_frequency — if "first-turn" and past first turn, return empty - if self._injection_frequency == "first-turn" and self._turn_count > 0: + # B5: injection_frequency — if "first-turn" and past first turn, return empty. + # _turn_count is 1-indexed (first user message = 1), so > 1 means "past first". + if self._injection_frequency == "first-turn" and self._turn_count > 1: return "" + # Trivial prompts ("ok", "yes", slash commands) carry no semantic signal. + if self._is_trivial_prompt(query): + return "" + + parts = [] + + # ----- Layer 1: Base context (representation + card) ----- + # On first call, fetch synchronously so turn 1 isn't empty. + # After that, serve from cache and refresh in background on cadence. + with self._base_context_lock: + if self._base_context_cache is None: + # First call — synchronous fetch + try: + ctx = self._manager.get_prefetch_context(self._session_key) + self._base_context_cache = self._format_first_turn_context(ctx) if ctx else "" + self._last_context_turn = self._turn_count + except Exception as e: + logger.debug("Honcho base context fetch failed: %s", e) + self._base_context_cache = "" + base_context = self._base_context_cache + + # Check if background context prefetch has a fresher result + if self._manager: + fresh_ctx = self._manager.pop_context_result(self._session_key) + if fresh_ctx: + formatted = self._format_first_turn_context(fresh_ctx) + if formatted: + with self._base_context_lock: + self._base_context_cache = formatted + base_context = formatted + + if base_context: + parts.append(base_context) + + # ----- Layer 2: Dialectic supplement ----- + # On the very first turn, no queue_prefetch() has run yet so the + # dialectic result is empty. Run with a bounded timeout so a slow + # Honcho connection doesn't block the first response indefinitely. + # On timeout we let the thread keep running and write its result into + # _prefetch_result under the lock, so the next turn picks it up. + # + # Skip if the session-start prewarm already filled _prefetch_result — + # firing another .chat() would be duplicate work. + with self._prefetch_lock: + _prewarm_landed = bool(self._prefetch_result) + if _prewarm_landed and self._last_dialectic_turn == -999: + self._last_dialectic_turn = self._turn_count + + if self._last_dialectic_turn == -999 and query: + _first_turn_timeout = ( + self._config.timeout if self._config and self._config.timeout else 8.0 + ) + _fired_at = self._turn_count + + def _run_first_turn() -> None: + try: + r = self._run_dialectic_depth(query) + except Exception as exc: + logger.debug("Honcho first-turn dialectic failed: %s", exc) + self._dialectic_empty_streak += 1 + return + if r and r.strip(): + with self._prefetch_lock: + self._prefetch_result = r + self._prefetch_result_fired_at = _fired_at + # Advance cadence only on a non-empty result so the next + # turn retries when the call returned nothing. + self._last_dialectic_turn = _fired_at + self._dialectic_empty_streak = 0 + else: + self._dialectic_empty_streak += 1 + + self._prefetch_thread_started_at = time.monotonic() + self._prefetch_thread = threading.Thread( + target=_run_first_turn, daemon=True, name="honcho-prefetch-first" + ) + self._prefetch_thread.start() + self._prefetch_thread.join(timeout=_first_turn_timeout) + if self._prefetch_thread.is_alive(): + logger.debug( + "Honcho first-turn dialectic still running after %.1fs — " + "will surface on next turn", + _first_turn_timeout, + ) + if self._prefetch_thread and self._prefetch_thread.is_alive(): self._prefetch_thread.join(timeout=3.0) with self._prefetch_lock: - result = self._prefetch_result + dialectic_result = self._prefetch_result + fired_at = self._prefetch_result_fired_at self._prefetch_result = "" - if not result: + self._prefetch_result_fired_at = -999 + + # Discard stale pending results: if the fire happened more than + # cadence × multiplier turns ago (e.g. a run of trivial-prompt turns + # passed without consumption), the content likely no longer tracks + # the current conversational pivot. + stale_limit = self._dialectic_cadence * self._STALE_RESULT_MULTIPLIER + if dialectic_result and fired_at >= 0 and (self._turn_count - fired_at) > stale_limit: + logger.debug( + "Honcho pending dialectic discarded as stale: fired_at=%d, " + "turn=%d, limit=%d", fired_at, self._turn_count, stale_limit, + ) + dialectic_result = "" + + if dialectic_result and dialectic_result.strip(): + parts.append(dialectic_result) + + if not parts: return "" + result = "\n\n".join(parts) + # ----- Port #3265: token budget enforcement ----- result = self._truncate_to_budget(result) - return f"## Honcho Context\n{result}" + return result def _truncate_to_budget(self, text: str) -> str: """Truncate text to fit within context_tokens budget if set.""" @@ -475,9 +696,11 @@ def _truncate_to_budget(self, text: str) -> str: return truncated + " …" def queue_prefetch(self, query: str, *, session_id: str = "") -> None: - """Fire a background dialectic query for the upcoming turn. + """Fire background prefetch threads for the upcoming turn. - B5: Checks cadence before firing background threads. + B5: Checks cadence independently for dialectic and context refresh. + Context refresh updates the base layer (representation + card). + Dialectic fires the LLM reasoning supplement. """ if self._cron_skipped: return @@ -488,38 +711,301 @@ def queue_prefetch(self, query: str, *, session_id: str = "") -> None: if self._recall_mode == "tools": return - # B5: cadence check — skip if too soon since last dialectic call - if self._dialectic_cadence > 1: - if (self._turn_count - self._last_dialectic_turn) < self._dialectic_cadence: - logger.debug("Honcho dialectic prefetch skipped: cadence %d, turns since last: %d", - self._dialectic_cadence, self._turn_count - self._last_dialectic_turn) - return + # Trivial prompts don't warrant either a context refresh or a dialectic call. + if self._is_trivial_prompt(query): + return + + # ----- Context refresh (base layer) — independent cadence ----- + if self._context_cadence <= 1 or (self._turn_count - self._last_context_turn) >= self._context_cadence: + self._last_context_turn = self._turn_count + try: + self._manager.prefetch_context(self._session_key, query) + except Exception as e: + logger.debug("Honcho context prefetch failed: %s", e) + + # ----- Dialectic prefetch (supplement layer) ----- + # Thread-alive guard with stale-thread recovery: a hung Honcho call + # older than timeout × multiplier is treated as dead so it can't + # block subsequent fires. + if self._thread_is_live(): + logger.debug("Honcho dialectic prefetch skipped: prior thread still running") + return - self._last_dialectic_turn = self._turn_count + # Cadence gate, widened by the empty-streak backoff so a persistently + # silent backend doesn't retry every turn forever. + effective = self._effective_cadence() + if (self._turn_count - self._last_dialectic_turn) < effective: + logger.debug( + "Honcho dialectic prefetch skipped: effective cadence %d " + "(base %d, empty streak %d), turns since last: %d", + effective, self._dialectic_cadence, self._dialectic_empty_streak, + self._turn_count - self._last_dialectic_turn, + ) + return + + # Cadence advances only on a non-empty result so empty returns + # (transient API error, sparse representation) retry next turn. + _fired_at = self._turn_count def _run(): try: - result = self._manager.dialectic_query( - self._session_key, query, peer="user" - ) - if result and result.strip(): - with self._prefetch_lock: - self._prefetch_result = result + result = self._run_dialectic_depth(query) except Exception as e: logger.debug("Honcho prefetch failed: %s", e) - + self._dialectic_empty_streak += 1 + return + if result and result.strip(): + with self._prefetch_lock: + self._prefetch_result = result + self._prefetch_result_fired_at = _fired_at + self._last_dialectic_turn = _fired_at + self._dialectic_empty_streak = 0 + else: + self._dialectic_empty_streak += 1 + + self._prefetch_thread_started_at = time.monotonic() self._prefetch_thread = threading.Thread( target=_run, daemon=True, name="honcho-prefetch" ) self._prefetch_thread.start() - # Also fire context prefetch if cadence allows - if self._context_cadence <= 1 or (self._turn_count - self._last_context_turn) >= self._context_cadence: - self._last_context_turn = self._turn_count - try: - self._manager.prefetch_context(self._session_key, query) - except Exception as e: - logger.debug("Honcho context prefetch failed: %s", e) + # ----- Dialectic depth: multi-pass .chat() with cold/warm prompts ----- + + # Proportional reasoning levels per depth/pass when dialecticDepthLevels + # is not configured. The base level is dialecticReasoningLevel. + # Index: (depth, pass) → level relative to base. + _PROPORTIONAL_LEVELS: dict[tuple[int, int], str] = { + # depth 1: single pass at base level + (1, 0): "base", + # depth 2: pass 0 lighter, pass 1 at base + (2, 0): "minimal", + (2, 1): "base", + # depth 3: pass 0 lighter, pass 1 at base, pass 2 one above minimal + (3, 0): "minimal", + (3, 1): "base", + (3, 2): "low", + } + + _LEVEL_ORDER = ("minimal", "low", "medium", "high", "max") + + # Char-count thresholds for the query-length reasoning heuristic. + _HEURISTIC_LENGTH_MEDIUM = 120 + _HEURISTIC_LENGTH_HIGH = 400 + + # Liveness constants. A thread older than timeout × multiplier is treated + # as dead so a hung Honcho call can't block future retries indefinitely. + _STALE_THREAD_MULTIPLIER = 2.0 + # Pending result whose fire-turn is older than cadence × multiplier is + # discarded on read so we don't inject context for a stale conversational + # pivot after a gap of trivial-prompt turns. + _STALE_RESULT_MULTIPLIER = 2 + # Cap on the empty-streak backoff so a persistently silent backend + # eventually settles on a ceiling instead of unbounded widening. + _BACKOFF_MAX = 8 + + def _thread_is_live(self) -> bool: + """Thread-alive guard that treats threads older than the stale + threshold as dead, so a hung Honcho request can't block new fires.""" + if not self._prefetch_thread or not self._prefetch_thread.is_alive(): + return False + timeout = (self._config.timeout if self._config and self._config.timeout else 8.0) + age = time.monotonic() - self._prefetch_thread_started_at + if age > timeout * self._STALE_THREAD_MULTIPLIER: + logger.debug( + "Honcho prefetch thread age %.1fs exceeds stale threshold " + "%.1fs — treating as dead", age, timeout * self._STALE_THREAD_MULTIPLIER, + ) + return False + return True + + def _effective_cadence(self) -> int: + """Cadence plus empty-streak backoff, capped at _BACKOFF_MAX × base.""" + if self._dialectic_empty_streak <= 0: + return self._dialectic_cadence + widened = self._dialectic_cadence + self._dialectic_empty_streak + ceiling = self._dialectic_cadence * self._BACKOFF_MAX + return min(widened, ceiling) + + def liveness_snapshot(self) -> dict: + """In-process snapshot of dialectic liveness state for diagnostics. + + Returns current turn, last successful dialectic turn, pending-result + fire turn, empty streak, effective cadence, and thread status. + """ + thread_age = None + if self._prefetch_thread and self._prefetch_thread.is_alive(): + thread_age = time.monotonic() - self._prefetch_thread_started_at + return { + "turn_count": self._turn_count, + "last_dialectic_turn": self._last_dialectic_turn, + "pending_result_fired_at": self._prefetch_result_fired_at, + "empty_streak": self._dialectic_empty_streak, + "effective_cadence": self._effective_cadence(), + "thread_alive": thread_age is not None, + "thread_age_seconds": thread_age, + } + + def _apply_reasoning_heuristic(self, base: str, query: str) -> str: + """Scale `base` up by query length, clamped at reasoning_level_cap. + + Char-count heuristic: +1 at >=120 chars, +2 at >=400. + """ + if not self._reasoning_heuristic or not query: + return base + if base not in self._LEVEL_ORDER: + return base + n = len(query) + if n < self._HEURISTIC_LENGTH_MEDIUM: + bump = 0 + elif n < self._HEURISTIC_LENGTH_HIGH: + bump = 1 + else: + bump = 2 + base_idx = self._LEVEL_ORDER.index(base) + cap_idx = self._LEVEL_ORDER.index(self._reasoning_level_cap) + return self._LEVEL_ORDER[min(base_idx + bump, cap_idx)] + + def _resolve_pass_level(self, pass_idx: int, query: str = "") -> str: + """Resolve reasoning level for a given pass index. + + Precedence: + 1. dialecticDepthLevels (explicit per-pass) — wins absolutely + 2. _PROPORTIONAL_LEVELS table (depth>1 lighter-early passes) + 3. Base level = dialecticReasoningLevel, optionally scaled by the + reasoning heuristic when the mapping falls through to 'base' + """ + if self._dialectic_depth_levels and pass_idx < len(self._dialectic_depth_levels): + return self._dialectic_depth_levels[pass_idx] + + base = (self._config.dialectic_reasoning_level if self._config else "low") + mapping = self._PROPORTIONAL_LEVELS.get((self._dialectic_depth, pass_idx)) + if mapping is None or mapping == "base": + return self._apply_reasoning_heuristic(base, query) + return mapping + + def _build_dialectic_prompt(self, pass_idx: int, prior_results: list[str], is_cold: bool) -> str: + """Build the prompt for a given dialectic pass. + + Pass 0: cold start (general user query) or warm (session-scoped). + Pass 1: self-audit / targeted synthesis against gaps from pass 0. + Pass 2: reconciliation / contradiction check across prior passes. + """ + if pass_idx == 0: + if is_cold: + return ( + "Who is this person? What are their preferences, goals, " + "and working style? Focus on facts that would help an AI " + "assistant be immediately useful." + ) + return ( + "Given what's been discussed in this session so far, what " + "context about this user is most relevant to the current " + "conversation? Prioritize active context over biographical facts." + ) + elif pass_idx == 1: + prior = prior_results[-1] if prior_results else "" + return ( + f"Given this initial assessment:\n\n{prior}\n\n" + "What gaps remain in your understanding that would help " + "going forward? Synthesize what you actually know about " + "the user's current state and immediate needs, grounded " + "in evidence from recent sessions." + ) + else: + # pass 2: reconciliation + return ( + f"Prior passes produced:\n\n" + f"Pass 1:\n{prior_results[0] if len(prior_results) > 0 else '(empty)'}\n\n" + f"Pass 2:\n{prior_results[1] if len(prior_results) > 1 else '(empty)'}\n\n" + "Do these assessments cohere? Reconcile any contradictions " + "and produce a final, concise synthesis of what matters most " + "for the current conversation." + ) + + @staticmethod + def _signal_sufficient(result: str) -> bool: + """Check if a dialectic pass returned enough signal to skip further passes. + + Heuristic: a response longer than 100 chars with some structure + (section headers, bullets, or an ordered list) is considered sufficient. + """ + if not result or len(result.strip()) < 100: + return False + # Structured output with sections/bullets is strong signal + if "\n" in result and ( + "##" in result + or "•" in result + or re.search(r"^[*-] ", result, re.MULTILINE) + or re.search(r"^\s*\d+\. ", result, re.MULTILINE) + ): + return True + # Long enough even without structure + return len(result.strip()) > 300 + + def _run_dialectic_depth(self, query: str) -> str: + """Execute up to dialecticDepth .chat() calls with conditional bail-out. + + Cold start (no base context): general user-oriented query. + Warm session (base context exists): session-scoped query. + Each pass is conditional — bails early if prior pass returned strong signal. + Returns the best (usually last) result. + """ + if not self._manager or not self._session_key: + return "" + + is_cold = not self._base_context_cache + results: list[str] = [] + + for i in range(self._dialectic_depth): + if i == 0: + prompt = self._build_dialectic_prompt(0, results, is_cold) + else: + # Skip further passes if prior pass delivered strong signal + if results and self._signal_sufficient(results[-1]): + logger.debug("Honcho dialectic depth %d: pass %d skipped, prior signal sufficient", + self._dialectic_depth, i) + break + prompt = self._build_dialectic_prompt(i, results, is_cold) + + level = self._resolve_pass_level(i, query=query) + logger.debug("Honcho dialectic depth %d: pass %d, level=%s, cold=%s", + self._dialectic_depth, i, level, is_cold) + + result = self._manager.dialectic_query( + self._session_key, prompt, + reasoning_level=level, + peer="user", + ) + results.append(result or "") + + # Return the last non-empty result (deepest pass that ran) + for r in reversed(results): + if r and r.strip(): + return r + return "" + + # Prompts that carry no semantic signal — trivial acknowledgements, slash + # commands, empty input. Skipping injection here saves tokens and prevents + # stale user-model context from derailing one-word replies. + _TRIVIAL_PROMPT_RE = re.compile( + r'^(yes|no|ok|okay|sure|thanks|thank you|y|n|yep|nope|yeah|nah|' + r'continue|go ahead|do it|proceed|got it|cool|nice|great|done|next|lgtm|k)$', + re.IGNORECASE, + ) + + @classmethod + def _is_trivial_prompt(cls, text: str) -> bool: + """Return True if the prompt is too trivial to warrant context injection.""" + if not text: + return True + stripped = text.strip() + if not stripped: + return True + if stripped.startswith("/"): + return True + if cls._TRIVIAL_PROMPT_RE.match(stripped): + return True + return False def on_turn_start(self, turn_number: int, message: str, **kwargs) -> None: """Track turn count for cadence and injection_frequency logic.""" @@ -659,7 +1145,14 @@ def handle_tool_call(self, tool_name: str, args: dict, **kwargs) -> str: try: if tool_name == "honcho_profile": - card = self._manager.get_peer_card(self._session_key) + peer = args.get("peer", "user") + card_update = args.get("card") + if card_update: + result = self._manager.set_peer_card(self._session_key, card_update, peer=peer) + if result is None: + return tool_error("Failed to update peer card.") + return json.dumps({"result": f"Peer card updated ({len(result)} facts).", "card": result}) + card = self._manager.get_peer_card(self._session_key, peer=peer) if not card: return json.dumps({"result": "No profile facts available yet."}) return json.dumps({"result": card}) @@ -669,30 +1162,68 @@ def handle_tool_call(self, tool_name: str, args: dict, **kwargs) -> str: if not query: return tool_error("Missing required parameter: query") max_tokens = min(int(args.get("max_tokens", 800)), 2000) + peer = args.get("peer", "user") result = self._manager.search_context( - self._session_key, query, max_tokens=max_tokens + self._session_key, query, max_tokens=max_tokens, peer=peer ) if not result: return json.dumps({"result": "No relevant context found."}) return json.dumps({"result": result}) - elif tool_name == "honcho_context": + elif tool_name == "honcho_reasoning": query = args.get("query", "") if not query: return tool_error("Missing required parameter: query") peer = args.get("peer", "user") + reasoning_level = args.get("reasoning_level") result = self._manager.dialectic_query( - self._session_key, query, peer=peer + self._session_key, query, + reasoning_level=reasoning_level, + peer=peer, ) + # Update cadence tracker so auto-injection respects the gap after an explicit call + self._last_dialectic_turn = self._turn_count return json.dumps({"result": result or "No result from Honcho."}) + elif tool_name == "honcho_context": + peer = args.get("peer", "user") + ctx = self._manager.get_session_context(self._session_key, peer=peer) + if not ctx: + return json.dumps({"result": "No context available yet."}) + parts = [] + if ctx.get("summary"): + parts.append(f"## Summary\n{ctx['summary']}") + if ctx.get("representation"): + parts.append(f"## Representation\n{ctx['representation']}") + if ctx.get("card"): + parts.append(f"## Card\n{ctx['card']}") + if ctx.get("recent_messages"): + msgs = ctx["recent_messages"] + msg_str = "\n".join( + f" [{m['role']}] {m['content'][:200]}" + for m in msgs[-5:] # last 5 for brevity + ) + parts.append(f"## Recent messages\n{msg_str}") + return json.dumps({"result": "\n\n".join(parts) or "No context available."}) + elif tool_name == "honcho_conclude": - conclusion = args.get("conclusion", "") - if not conclusion: - return tool_error("Missing required parameter: conclusion") - ok = self._manager.create_conclusion(self._session_key, conclusion) + delete_id = (args.get("delete_id") or "").strip() + conclusion = args.get("conclusion", "").strip() + peer = args.get("peer", "user") + + has_delete_id = bool(delete_id) + has_conclusion = bool(conclusion) + if has_delete_id == has_conclusion: + return tool_error("Exactly one of conclusion or delete_id must be provided.") + + if has_delete_id: + ok = self._manager.delete_conclusion(self._session_key, delete_id, peer=peer) + if ok: + return json.dumps({"result": f"Conclusion {delete_id} deleted."}) + return tool_error(f"Failed to delete conclusion {delete_id}.") + ok = self._manager.create_conclusion(self._session_key, conclusion, peer=peer) if ok: - return json.dumps({"result": f"Conclusion saved: {conclusion}"}) + return json.dumps({"result": f"Conclusion saved for {peer}: {conclusion}"}) return tool_error("Failed to save conclusion.") return tool_error(f"Unknown tool: {tool_name}") diff --git a/plugins/memory/honcho/cli.py b/plugins/memory/honcho/cli.py index dff4b386a5cd..5c829a4c989a 100644 --- a/plugins/memory/honcho/cli.py +++ b/plugins/memory/honcho/cli.py @@ -440,11 +440,63 @@ def cmd_setup(args) -> None: if new_recall in ("hybrid", "context", "tools"): hermes_host["recallMode"] = new_recall - # --- 7. Session strategy --- - current_strat = hermes_host.get("sessionStrategy") or cfg.get("sessionStrategy", "per-directory") + # --- 7. Context token budget --- + current_ctx_tokens = hermes_host.get("contextTokens") or cfg.get("contextTokens") + current_display = str(current_ctx_tokens) if current_ctx_tokens else "uncapped" + print("\n Context injection per turn (hybrid/context recall modes only):") + print(" uncapped -- no limit (default)") + print(" N -- token limit per turn (e.g. 1200)") + new_ctx_tokens = _prompt("Context tokens", default=current_display) + if new_ctx_tokens.strip().lower() in ("none", "uncapped", "no limit"): + hermes_host.pop("contextTokens", None) + elif new_ctx_tokens.strip() == "": + pass # keep current + else: + try: + val = int(new_ctx_tokens) + if val >= 0: + hermes_host["contextTokens"] = val + except (ValueError, TypeError): + pass # keep current + + # --- 7b. Dialectic cadence --- + current_dialectic = str(hermes_host.get("dialecticCadence") or cfg.get("dialecticCadence") or "2") + print("\n Dialectic cadence:") + print(" How often Honcho rebuilds its user model (LLM call on Honcho backend).") + print(" 1 = every turn, 2 = every other turn, 3+ = sparser.") + print(" Recommended: 1-5.") + new_dialectic = _prompt("Dialectic cadence", default=current_dialectic) + try: + val = int(new_dialectic) + if val >= 1: + hermes_host["dialecticCadence"] = val + except (ValueError, TypeError): + hermes_host["dialecticCadence"] = 2 + + # --- 7c. Dialectic reasoning level --- + current_reasoning = ( + hermes_host.get("dialecticReasoningLevel") + or cfg.get("dialecticReasoningLevel") + or "low" + ) + print("\n Dialectic reasoning level:") + print(" Depth Honcho uses when synthesizing user context on auto-injected calls.") + print(" minimal -- quick factual lookups") + print(" low -- straightforward questions (default)") + print(" medium -- multi-aspect synthesis") + print(" high -- complex behavioral patterns") + print(" max -- thorough audit-level analysis") + new_reasoning = _prompt("Reasoning level", default=current_reasoning) + if new_reasoning in ("minimal", "low", "medium", "high", "max"): + hermes_host["dialecticReasoningLevel"] = new_reasoning + else: + hermes_host["dialecticReasoningLevel"] = "low" + + # --- 8. Session strategy --- + current_strat = hermes_host.get("sessionStrategy") or cfg.get("sessionStrategy", "per-session") print("\n Session strategy:") - print(" per-directory -- one session per working directory (default)") - print(" per-session -- new Honcho session each run") + print(" per-session -- each run starts clean, Honcho injects context automatically") + print(" per-directory -- reuses session per dir, prior context auto-injected each run") print(" per-repo -- one session per git repository") print(" global -- single session across all directories") new_strat = _prompt("Session strategy", default=current_strat) @@ -490,10 +542,11 @@ def cmd_setup(args) -> None: print(f" Recall: {hcfg.recall_mode}") print(f" Sessions: {hcfg.session_strategy}") print("\n Honcho tools available in chat:") - print(" honcho_context -- ask Honcho about the user (LLM-synthesized)") - print(" honcho_search -- semantic search over history (no LLM)") - print(" honcho_profile -- peer card, key facts (no LLM)") - print(" honcho_conclude -- persist a user fact to memory (no LLM)") + print(" honcho_context -- session context: summary, representation, card, messages") + print(" honcho_search -- semantic search over history") + print(" honcho_profile -- peer card, key facts") + print(" honcho_reasoning -- ask Honcho a question, synthesized answer") + print(" honcho_conclude -- persist a user fact to memory") print("\n Other commands:") print(" hermes honcho status -- show full config") print(" hermes honcho mode -- change recall/observation mode") @@ -585,13 +638,29 @@ def cmd_status(args) -> None: print(f" Enabled: {hcfg.enabled}") print(f" API key: {masked}") print(f" Workspace: {hcfg.workspace_id}") - print(f" Config path: {active_path}") + + # Config paths — show where config was read from and where writes go + global_path = Path.home() / ".honcho" / "config.json" + print(f" Config: {active_path}") if write_path != active_path: - print(f" Write path: {write_path} (instance-local)") + print(f" Write to: {write_path} (profile-local)") + if active_path == global_path: + print(f" Fallback: (none — using global ~/.honcho/config.json)") + elif global_path.exists(): + print(f" Fallback: {global_path} (exists, cross-app interop)") + print(f" AI peer: {hcfg.ai_peer}") print(f" User peer: {hcfg.peer_name or 'not set'}") print(f" Session key: {hcfg.resolve_session_name()}") + print(f" Session strat: {hcfg.session_strategy}") print(f" Recall mode: {hcfg.recall_mode}") + print(f" Context budget: {hcfg.context_tokens or '(uncapped)'} tokens") + raw = getattr(hcfg, "raw", None) or {} + dialectic_cadence = raw.get("dialecticCadence") or 1 + print(f" Dialectic cad: every {dialectic_cadence} turn{'s' if dialectic_cadence != 1 else ''}") + reasoning_cap = raw.get("reasoningLevelCap") or hcfg.reasoning_level_cap + heuristic_on = "on" if hcfg.reasoning_heuristic else "off" + print(f" Reasoning: base={hcfg.dialectic_reasoning_level}, cap={reasoning_cap}, heuristic={heuristic_on}") print(f" Observation: user(me={hcfg.user_observe_me},others={hcfg.user_observe_others}) ai(me={hcfg.ai_observe_me},others={hcfg.ai_observe_others})") print(f" Write freq: {hcfg.write_frequency}") @@ -599,8 +668,8 @@ def cmd_status(args) -> None: print("\n Connection... ", end="", flush=True) try: client = get_honcho_client(hcfg) - print("OK") _show_peer_cards(hcfg, client) + print("OK") except Exception as e: print(f"FAILED ({e})\n") else: @@ -824,6 +893,41 @@ def cmd_mode(args) -> None: print(f" {label}Recall mode -> {mode_arg} ({MODES[mode_arg]})\n") +def cmd_strategy(args) -> None: + """Show or set the session strategy.""" + STRATEGIES = { + "per-session": "each run starts clean, Honcho injects context automatically", + "per-directory": "reuses session per dir, prior context auto-injected each run", + "per-repo": "one session per git repository", + "global": "single session across all directories", + } + cfg = _read_config() + strat_arg = getattr(args, "strategy", None) + + if strat_arg is None: + current = ( + (cfg.get("hosts") or {}).get(_host_key(), {}).get("sessionStrategy") + or cfg.get("sessionStrategy") + or "per-session" + ) + print("\nHoncho session strategy\n" + "─" * 40) + for s, desc in STRATEGIES.items(): + marker = " <-" if s == current else "" + print(f" {s:<15} {desc}{marker}") + print(f"\n Set with: hermes honcho strategy [per-session|per-directory|per-repo|global]\n") + return + + if strat_arg not in STRATEGIES: + print(f" Invalid strategy '{strat_arg}'. Options: {', '.join(STRATEGIES)}\n") + return + + host = _host_key() + label = f"[{host}] " if host != "hermes" else "" + cfg.setdefault("hosts", {}).setdefault(host, {})["sessionStrategy"] = strat_arg + _write_config(cfg) + print(f" {label}Session strategy -> {strat_arg} ({STRATEGIES[strat_arg]})\n") + + def cmd_tokens(args) -> None: """Show or set token budget settings.""" cfg = _read_config() @@ -1143,10 +1247,11 @@ def cmd_migrate(args) -> None: print(" automatically. Files become the seed, not the live store.") print() print(" Honcho tools (available to the agent during conversation)") - print(" honcho_context — ask Honcho a question, get a synthesized answer (LLM)") - print(" honcho_search — semantic search over stored context (no LLM)") - print(" honcho_profile — fast peer card snapshot (no LLM)") - print(" honcho_conclude — write a conclusion/fact back to memory (no LLM)") + print(" honcho_context — session context: summary, representation, card, messages") + print(" honcho_search — semantic search over stored context") + print(" honcho_profile — fast peer card snapshot") + print(" honcho_reasoning — ask Honcho a question, synthesized answer") + print(" honcho_conclude — write a conclusion/fact back to memory") print() print(" Session naming") print(" OpenClaw: no persistent session concept — files are global.") @@ -1197,6 +1302,8 @@ def honcho_command(args) -> None: cmd_peer(args) elif sub == "mode": cmd_mode(args) + elif sub == "strategy": + cmd_strategy(args) elif sub == "tokens": cmd_tokens(args) elif sub == "identity": @@ -1211,7 +1318,7 @@ def honcho_command(args) -> None: cmd_sync(args) else: print(f" Unknown honcho command: {sub}") - print(" Available: status, sessions, map, peer, mode, tokens, identity, migrate, enable, disable, sync\n") + print(" Available: status, sessions, map, peer, mode, strategy, tokens, identity, migrate, enable, disable, sync\n") def register_cli(subparser) -> None: @@ -1270,6 +1377,15 @@ def register_cli(subparser) -> None: help="Recall mode to set (hybrid/context/tools). Omit to show current.", ) + strategy_parser = subs.add_parser( + "strategy", help="Show or set session strategy (per-session/per-directory/per-repo/global)", + ) + strategy_parser.add_argument( + "strategy", nargs="?", metavar="STRATEGY", + choices=("per-session", "per-directory", "per-repo", "global"), + help="Session strategy to set. Omit to show current.", + ) + tokens_parser = subs.add_parser( "tokens", help="Show or set token budget for context and dialectic", ) diff --git a/plugins/memory/honcho/client.py b/plugins/memory/honcho/client.py index 3c779f64fea1..fef2e2d58f1e 100644 --- a/plugins/memory/honcho/client.py +++ b/plugins/memory/honcho/client.py @@ -94,6 +94,68 @@ def _resolve_bool(host_val, root_val, *, default: bool) -> bool: return default +def _parse_context_tokens(host_val, root_val) -> int | None: + """Parse contextTokens: host wins, then root, then None (uncapped).""" + for val in (host_val, root_val): + if val is not None: + try: + return int(val) + except (ValueError, TypeError): + pass + return None + + +def _parse_dialectic_depth(host_val, root_val) -> int: + """Parse dialecticDepth: host wins, then root, then 1. Clamped to 1-3.""" + for val in (host_val, root_val): + if val is not None: + try: + return max(1, min(int(val), 3)) + except (ValueError, TypeError): + pass + return 1 + + +_VALID_REASONING_LEVELS = ("minimal", "low", "medium", "high", "max") + + +def _parse_dialectic_depth_levels(host_val, root_val, depth: int) -> list[str] | None: + """Parse dialecticDepthLevels: optional array of reasoning levels per pass. + + Returns None when not configured (use proportional defaults). + When configured, validates each level and truncates/pads to match depth. + """ + for val in (host_val, root_val): + if val is not None and isinstance(val, list): + levels = [ + lvl if lvl in _VALID_REASONING_LEVELS else "low" + for lvl in val[:depth] + ] + # Pad with "low" if array is shorter than depth + while len(levels) < depth: + levels.append("low") + return levels + return None + + +def _resolve_optional_float(*values: Any) -> float | None: + """Return the first non-empty value coerced to a positive float.""" + for value in values: + if value is None: + continue + if isinstance(value, str): + value = value.strip() + if not value: + continue + try: + parsed = float(value) + except (TypeError, ValueError): + continue + if parsed > 0: + return parsed + return None + + _VALID_OBSERVATION_MODES = {"unified", "directional"} _OBSERVATION_MODE_ALIASES = {"shared": "unified", "separate": "directional", "cross": "directional"} @@ -159,6 +221,8 @@ class HonchoClientConfig: environment: str = "production" # Optional base URL for self-hosted Honcho (overrides environment mapping) base_url: str | None = None + # Optional request timeout in seconds for Honcho SDK HTTP calls + timeout: float | None = None # Identity peer_name: str | None = None ai_peer: str = "hermes" @@ -168,17 +232,30 @@ class HonchoClientConfig: # Write frequency: "async" (background thread), "turn" (sync per turn), # "session" (flush on session end), or int (every N turns) write_frequency: str | int = "async" - # Prefetch budget + # Prefetch budget (None = no cap; set to an integer to bound auto-injected context) context_tokens: int | None = None # Dialectic (peer.chat) settings # reasoning_level: "minimal" | "low" | "medium" | "high" | "max" dialectic_reasoning_level: str = "low" - # dynamic: auto-bump reasoning level based on query length - # true — low->medium (120+ chars), low->high (400+ chars), capped at "high" - # false — always use dialecticReasoningLevel as-is + # When true, the model can override reasoning_level per-call via the + # honcho_reasoning tool param (agentic). When false, always uses + # dialecticReasoningLevel and ignores model-provided overrides. dialectic_dynamic: bool = True # Max chars of dialectic result to inject into Hermes system prompt dialectic_max_chars: int = 600 + # Dialectic depth: how many .chat() calls per dialectic cycle (1-3). + # Depth 1: single call. Depth 2: self-audit + targeted synthesis. + # Depth 3: self-audit + synthesis + reconciliation. + dialectic_depth: int = 1 + # Optional per-pass reasoning level override. Array of reasoning levels + # matching dialectic_depth length. When None, uses proportional defaults + # derived from dialectic_reasoning_level. + dialectic_depth_levels: list[str] | None = None + # When true, the auto-injected dialectic scales reasoning level up on + # longer queries. See HonchoMemoryProvider for thresholds. + reasoning_heuristic: bool = True + # Ceiling for the heuristic-selected reasoning level. + reasoning_level_cap: str = "high" # Honcho API limits — configurable for self-hosted instances # Max chars per message sent via add_messages() (Honcho cloud: 25000) message_max_chars: int = 25000 @@ -189,10 +266,8 @@ class HonchoClientConfig: # "context" — auto-injected context only, Honcho tools removed # "tools" — Honcho tools only, no auto-injected context recall_mode: str = "hybrid" - # When True and recallMode is "tools", create the Honcho session eagerly - # during initialize() instead of deferring to the first tool call. - # This ensures sync_turn() can write from the very first turn. - # Does NOT enable automatic context injection — only changes init timing. + # Eager init in tools mode — when true, initializes session during + # initialize() instead of deferring to first tool call init_on_session_start: bool = False # Observation mode: legacy string shorthand ("directional" or "unified"). # Kept for backward compat; granular per-peer booleans below are preferred. @@ -224,12 +299,14 @@ def from_env( resolved_host = host or resolve_active_host() api_key = os.environ.get("HONCHO_API_KEY") base_url = os.environ.get("HONCHO_BASE_URL", "").strip() or None + timeout = _resolve_optional_float(os.environ.get("HONCHO_TIMEOUT")) return cls( host=resolved_host, workspace_id=workspace_id, api_key=api_key, environment=os.environ.get("HONCHO_ENVIRONMENT", "production"), base_url=base_url, + timeout=timeout, ai_peer=resolved_host, enabled=bool(api_key or base_url), ) @@ -290,6 +367,11 @@ def from_global_config( or os.environ.get("HONCHO_BASE_URL", "").strip() or None ) + timeout = _resolve_optional_float( + raw.get("timeout"), + raw.get("requestTimeout"), + os.environ.get("HONCHO_TIMEOUT"), + ) # Auto-enable when API key or base_url is present (unless explicitly disabled) # Host-level enabled wins, then root-level, then auto-enable if key/url exists. @@ -335,12 +417,16 @@ def from_global_config( api_key=api_key, environment=environment, base_url=base_url, + timeout=timeout, peer_name=host_block.get("peerName") or raw.get("peerName"), ai_peer=ai_peer, enabled=enabled, save_messages=save_messages, write_frequency=write_frequency, - context_tokens=host_block.get("contextTokens") or raw.get("contextTokens"), + context_tokens=_parse_context_tokens( + host_block.get("contextTokens"), + raw.get("contextTokens"), + ), dialectic_reasoning_level=( host_block.get("dialecticReasoningLevel") or raw.get("dialecticReasoningLevel") @@ -356,6 +442,25 @@ def from_global_config( or raw.get("dialecticMaxChars") or 600 ), + dialectic_depth=_parse_dialectic_depth( + host_block.get("dialecticDepth"), + raw.get("dialecticDepth"), + ), + dialectic_depth_levels=_parse_dialectic_depth_levels( + host_block.get("dialecticDepthLevels"), + raw.get("dialecticDepthLevels"), + depth=_parse_dialectic_depth(host_block.get("dialecticDepth"), raw.get("dialecticDepth")), + ), + reasoning_heuristic=_resolve_bool( + host_block.get("reasoningHeuristic"), + raw.get("reasoningHeuristic"), + default=True, + ), + reasoning_level_cap=( + host_block.get("reasoningLevelCap") + or raw.get("reasoningLevelCap") + or "high" + ), message_max_chars=int( host_block.get("messageMaxChars") or raw.get("messageMaxChars") @@ -422,16 +527,18 @@ def resolve_session_name( cwd: str | None = None, session_title: str | None = None, session_id: str | None = None, + gateway_session_key: str | None = None, ) -> str | None: """Resolve Honcho session name. Resolution order: 1. Manual directory override from sessions map 2. Hermes session title (from /title command) - 3. per-session strategy — Hermes session_id ({timestamp}_{hex}) - 4. per-repo strategy — git repo root directory name - 5. per-directory strategy — directory basename - 6. global strategy — workspace name + 3. Gateway session key (stable per-chat identifier from gateway platforms) + 4. per-session strategy — Hermes session_id ({timestamp}_{hex}) + 5. per-repo strategy — git repo root directory name + 6. per-directory strategy — directory basename + 7. global strategy — workspace name """ import re @@ -445,12 +552,22 @@ def resolve_session_name( # /title mid-session remap if session_title: - sanitized = re.sub(r'[^a-zA-Z0-9_-]', '-', session_title).strip('-') + sanitized = re.sub(r'[^a-zA-Z0-9_-]+', '-', session_title).strip('-') if sanitized: if self.session_peer_prefix and self.peer_name: return f"{self.peer_name}-{sanitized}" return sanitized + # Gateway session key: stable per-chat identifier passed by the gateway + # (e.g. "agent:main:telegram:dm:8439114563"). Sanitize colons to hyphens + # for Honcho session ID compatibility. This takes priority over strategy- + # based resolution because gateway platforms need per-chat isolation that + # cwd-based strategies cannot provide. + if gateway_session_key: + sanitized = re.sub(r'[^a-zA-Z0-9_-]+', '-', gateway_session_key).strip('-') + if sanitized: + return sanitized + # per-session: inherit Hermes session_id (new Honcho session each run) if self.session_strategy == "per-session" and session_id: if self.session_peer_prefix and self.peer_name: @@ -512,13 +629,20 @@ def get_honcho_client(config: HonchoClientConfig | None = None) -> Honcho: # mapping, enabling remote self-hosted Honcho deployments without # requiring the server to live on localhost. resolved_base_url = config.base_url - if not resolved_base_url: + resolved_timeout = config.timeout + if not resolved_base_url or resolved_timeout is None: try: from hermes_cli.config import load_config hermes_cfg = load_config() honcho_cfg = hermes_cfg.get("honcho", {}) if isinstance(honcho_cfg, dict): - resolved_base_url = honcho_cfg.get("base_url", "").strip() or None + if not resolved_base_url: + resolved_base_url = honcho_cfg.get("base_url", "").strip() or None + if resolved_timeout is None: + resolved_timeout = _resolve_optional_float( + honcho_cfg.get("timeout"), + honcho_cfg.get("request_timeout"), + ) except Exception: pass @@ -553,6 +677,8 @@ def get_honcho_client(config: HonchoClientConfig | None = None) -> Honcho: } if resolved_base_url: kwargs["base_url"] = resolved_base_url + if resolved_timeout is not None: + kwargs["timeout"] = resolved_timeout _honcho_client = Honcho(**kwargs) diff --git a/plugins/memory/honcho/session.py b/plugins/memory/honcho/session.py index 2cd4c5bd2f51..79625b5cd580 100644 --- a/plugins/memory/honcho/session.py +++ b/plugins/memory/honcho/session.py @@ -78,6 +78,7 @@ def __init__( honcho: Honcho | None = None, context_tokens: int | None = None, config: Any | None = None, + runtime_user_peer_name: str | None = None, ): """ Initialize the session manager. @@ -87,10 +88,12 @@ def __init__( context_tokens: Max tokens for context() calls (None = Honcho default). config: HonchoClientConfig from global config (provides peer_name, ai_peer, write_frequency, observation, etc.). + runtime_user_peer_name: Gateway user identity for per-user memory scoping. """ self._honcho = honcho self._context_tokens = context_tokens self._config = config + self._runtime_user_peer_name = runtime_user_peer_name self._cache: dict[str, HonchoSession] = {} self._peers_cache: dict[str, Any] = {} self._sessions_cache: dict[str, Any] = {} @@ -100,9 +103,11 @@ def __init__( self._write_frequency = write_frequency self._turn_counter: int = 0 - # Prefetch caches: session_key → last result (consumed once per turn) + # Prefetch cache: session_key → last context result (consumed once per turn). + # Dialectic results are cached on the plugin side (HonchoMemoryProvider + # ._prefetch_result) so session-start prewarm and turn-driven fires share + # one source of truth; see __init__.py _do_session_init for the prewarm. self._context_cache: dict[str, dict] = {} - self._dialectic_cache: dict[str, str] = {} self._prefetch_cache_lock = threading.Lock() self._dialectic_reasoning_level: str = ( config.dialectic_reasoning_level if config else "low" @@ -272,8 +277,10 @@ def get_or_create(self, key: str) -> HonchoSession: logger.debug("Local session cache hit: %s", key) return self._cache[key] - # Use peer names from global config when available - if self._config and self._config.peer_name: + # Gateway sessions should use the runtime user identity when available. + if self._runtime_user_peer_name: + user_peer_id = self._sanitize_id(self._runtime_user_peer_name) + elif self._config and self._config.peer_name: user_peer_id = self._sanitize_id(self._config.peer_name) else: # Fallback: derive from session key @@ -486,36 +493,9 @@ def new_session(self, key: str) -> HonchoSession: _REASONING_LEVELS = ("minimal", "low", "medium", "high", "max") - def _dynamic_reasoning_level(self, query: str) -> str: - """ - Pick a reasoning level for a dialectic query. - - When dialecticDynamic is true (default), auto-bumps based on query - length so Honcho applies more inference where it matters: - - < 120 chars -> configured default (typically "low") - 120-400 chars -> +1 level above default (cap at "high") - > 400 chars -> +2 levels above default (cap at "high") - - "max" is never selected automatically -- reserve it for explicit config. - - When dialecticDynamic is false, always returns the configured level. - """ - if not self._dialectic_dynamic: - return self._dialectic_reasoning_level - - levels = self._REASONING_LEVELS - default_idx = levels.index(self._dialectic_reasoning_level) if self._dialectic_reasoning_level in levels else 1 - n = len(query) - if n < 120: - bump = 0 - elif n < 400: - bump = 1 - else: - bump = 2 - # Cap at "high" (index 3) for auto-selection - idx = min(default_idx + bump, 3) - return levels[idx] + def _default_reasoning_level(self) -> str: + """Return the configured default reasoning level.""" + return self._dialectic_reasoning_level def dialectic_query( self, session_key: str, query: str, @@ -526,14 +506,15 @@ def dialectic_query( Query Honcho's dialectic endpoint about a peer. Runs an LLM on Honcho's backend against the target peer's full - representation. Higher latency than context() — call async via - prefetch_dialectic() to avoid blocking the response. + representation. Higher latency than context() — callers run this in + a background thread (see HonchoMemoryProvider) to avoid blocking. Args: session_key: The session key to query against. query: Natural language question. - reasoning_level: Override the config default. If None, uses - _dynamic_reasoning_level(query). + reasoning_level: Override the configured default (dialecticReasoningLevel). + Only honored when dialecticDynamic is true. + If None or dialecticDynamic is false, uses the configured default. peer: Which peer to query — "user" (default) or "ai". Returns: @@ -543,29 +524,34 @@ def dialectic_query( if not session: return "" + target_peer_id = self._resolve_peer_id(session, peer) + if target_peer_id is None: + return "" + # Guard: truncate query to Honcho's dialectic input limit if len(query) > self._dialectic_max_input_chars: query = query[:self._dialectic_max_input_chars].rsplit(" ", 1)[0] - level = reasoning_level or self._dynamic_reasoning_level(query) + if self._dialectic_dynamic and reasoning_level: + level = reasoning_level + else: + level = self._default_reasoning_level() try: if self._ai_observe_others: - # AI peer can observe user — use cross-observation routing - if peer == "ai": - ai_peer_obj = self._get_or_create_peer(session.assistant_peer_id) + # AI peer can observe other peers — use assistant as observer. + ai_peer_obj = self._get_or_create_peer(session.assistant_peer_id) + if target_peer_id == session.assistant_peer_id: result = ai_peer_obj.chat(query, reasoning_level=level) or "" else: - ai_peer_obj = self._get_or_create_peer(session.assistant_peer_id) result = ai_peer_obj.chat( query, - target=session.user_peer_id, + target=target_peer_id, reasoning_level=level, ) or "" else: - # AI can't observe others — each peer queries self - peer_id = session.assistant_peer_id if peer == "ai" else session.user_peer_id - target_peer = self._get_or_create_peer(peer_id) + # Without cross-observation, each peer queries its own context. + target_peer = self._get_or_create_peer(target_peer_id) result = target_peer.chat(query, reasoning_level=level) or "" # Apply Hermes-side char cap before caching @@ -576,42 +562,6 @@ def dialectic_query( logger.warning("Honcho dialectic query failed: %s", e) return "" - def prefetch_dialectic(self, session_key: str, query: str) -> None: - """ - Fire a dialectic_query in a background thread, caching the result. - - Non-blocking. The result is available via pop_dialectic_result() - on the next call (typically the following turn). Reasoning level - is selected dynamically based on query complexity. - - Args: - session_key: The session key to query against. - query: The user's current message, used as the query. - """ - def _run(): - result = self.dialectic_query(session_key, query) - if result: - self.set_dialectic_result(session_key, result) - - t = threading.Thread(target=_run, name="honcho-dialectic-prefetch", daemon=True) - t.start() - - def set_dialectic_result(self, session_key: str, result: str) -> None: - """Store a prefetched dialectic result in a thread-safe way.""" - if not result: - return - with self._prefetch_cache_lock: - self._dialectic_cache[session_key] = result - - def pop_dialectic_result(self, session_key: str) -> str: - """ - Return and clear the cached dialectic result for this session. - - Returns empty string if no result is ready yet. - """ - with self._prefetch_cache_lock: - return self._dialectic_cache.pop(session_key, "") - def prefetch_context(self, session_key: str, user_message: str | None = None) -> None: """ Fire get_prefetch_context in a background thread, caching the result. @@ -647,10 +597,11 @@ def get_prefetch_context(self, session_key: str, user_message: str | None = None """ Pre-fetch user and AI peer context from Honcho. - Fetches peer_representation and peer_card for both peers. search_query - is intentionally omitted — it would only affect additional excerpts - that this code does not consume, and passing the raw message exposes - conversation content in server access logs. + Fetches peer_representation and peer_card for both peers, plus the + session summary when available. search_query is intentionally omitted + — it would only affect additional excerpts that this code does not + consume, and passing the raw message exposes conversation content in + server access logs. Args: session_key: The session key to get context for. @@ -658,15 +609,29 @@ def get_prefetch_context(self, session_key: str, user_message: str | None = None Returns: Dictionary with 'representation', 'card', 'ai_representation', - and 'ai_card' keys. + 'ai_card', and optionally 'summary' keys. """ session = self._cache.get(session_key) if not session: return {} result: dict[str, str] = {} + + # Session summary — provides session-scoped context. + # Fresh sessions (per-session cold start, or first-ever per-directory) + # return null summary — the guard below handles that gracefully. + # Per-directory returning sessions get their accumulated summary. try: - user_ctx = self._fetch_peer_context(session.user_peer_id) + honcho_session = self._sessions_cache.get(session.honcho_session_id) + if honcho_session: + ctx = honcho_session.context(summary=True) + if ctx.summary and getattr(ctx.summary, "content", None): + result["summary"] = ctx.summary.content + except Exception as e: + logger.debug("Failed to fetch session summary from Honcho: %s", e) + + try: + user_ctx = self._fetch_peer_context(session.user_peer_id, target=session.user_peer_id) result["representation"] = user_ctx["representation"] result["card"] = "\n".join(user_ctx["card"]) except Exception as e: @@ -674,7 +639,7 @@ def get_prefetch_context(self, session_key: str, user_message: str | None = None # Also fetch AI peer's own representation so Hermes knows itself. try: - ai_ctx = self._fetch_peer_context(session.assistant_peer_id) + ai_ctx = self._fetch_peer_context(session.assistant_peer_id, target=session.assistant_peer_id) result["ai_representation"] = ai_ctx["representation"] result["ai_card"] = "\n".join(ai_ctx["card"]) except Exception as e: @@ -862,7 +827,7 @@ def _normalize_card(card: Any) -> list[str]: return [str(item) for item in card if item] return [str(card)] - def _fetch_peer_card(self, peer_id: str) -> list[str]: + def _fetch_peer_card(self, peer_id: str, *, target: str | None = None) -> list[str]: """Fetch a peer card directly from the peer object. This avoids relying on session.context(), which can return an empty @@ -872,22 +837,33 @@ def _fetch_peer_card(self, peer_id: str) -> list[str]: peer = self._get_or_create_peer(peer_id) getter = getattr(peer, "get_card", None) if callable(getter): - return self._normalize_card(getter()) + return self._normalize_card(getter(target=target) if target is not None else getter()) legacy_getter = getattr(peer, "card", None) if callable(legacy_getter): - return self._normalize_card(legacy_getter()) + return self._normalize_card(legacy_getter(target=target) if target is not None else legacy_getter()) return [] - def _fetch_peer_context(self, peer_id: str, search_query: str | None = None) -> dict[str, Any]: + def _fetch_peer_context( + self, + peer_id: str, + search_query: str | None = None, + *, + target: str | None = None, + ) -> dict[str, Any]: """Fetch representation + peer card directly from a peer object.""" peer = self._get_or_create_peer(peer_id) representation = "" card: list[str] = [] try: - ctx = peer.context(search_query=search_query) if search_query else peer.context() + context_kwargs: dict[str, Any] = {} + if target is not None: + context_kwargs["target"] = target + if search_query is not None: + context_kwargs["search_query"] = search_query + ctx = peer.context(**context_kwargs) if context_kwargs else peer.context() representation = ( getattr(ctx, "representation", None) or getattr(ctx, "peer_representation", None) @@ -899,24 +875,111 @@ def _fetch_peer_context(self, peer_id: str, search_query: str | None = None) -> if not representation: try: - representation = peer.representation() or "" + representation = ( + peer.representation(target=target) if target is not None else peer.representation() + ) or "" except Exception as e: logger.debug("Direct peer.representation() failed for '%s': %s", peer_id, e) if not card: try: - card = self._fetch_peer_card(peer_id) + card = self._fetch_peer_card(peer_id, target=target) except Exception as e: logger.debug("Direct peer card fetch failed for '%s': %s", peer_id, e) return {"representation": representation, "card": card} - def get_peer_card(self, session_key: str) -> list[str]: + def get_session_context(self, session_key: str, peer: str = "user") -> dict[str, Any]: + """Fetch full session context from Honcho including summary. + + Uses the session-level context() API which returns summary, + peer_representation, peer_card, and messages. """ - Fetch the user peer's card — a curated list of key facts. + session = self._cache.get(session_key) + if not session: + return {} + + honcho_session = self._sessions_cache.get(session.honcho_session_id) + if not honcho_session: + # Fall back to peer-level context, respecting the requested peer + peer_id = self._resolve_peer_id(session, peer) + if peer_id is None: + peer_id = session.user_peer_id + return self._fetch_peer_context(peer_id, target=peer_id) + + try: + peer_id = self._resolve_peer_id(session, peer) + ctx = honcho_session.context( + summary=True, + peer_target=peer_id, + peer_perspective=session.user_peer_id if peer == "user" else session.assistant_peer_id, + ) + + result: dict[str, Any] = {} + + # Summary + if ctx.summary: + result["summary"] = ctx.summary.content + + # Peer representation and card + if ctx.peer_representation: + result["representation"] = ctx.peer_representation + if ctx.peer_card: + result["card"] = "\n".join(ctx.peer_card) + + # Messages (last N for context) + if ctx.messages: + recent = ctx.messages[-10:] # last 10 messages + result["recent_messages"] = [ + {"role": getattr(m, "peer_id", "unknown"), "content": (m.content or "")[:500]} + for m in recent + ] + + return result + except Exception as e: + logger.debug("Session context fetch failed: %s", e) + return {} + + def _resolve_peer_id(self, session: HonchoSession, peer: str | None) -> str: + """Resolve a peer alias or explicit peer ID to a concrete Honcho peer ID. + + Always returns a non-empty string: either a known peer ID or a + sanitized version of the caller-supplied alias/ID. + """ + candidate = (peer or "user").strip() + if not candidate: + return session.user_peer_id + + normalized = self._sanitize_id(candidate) + if normalized == self._sanitize_id("user"): + return session.user_peer_id + if normalized == self._sanitize_id("ai"): + return session.assistant_peer_id + + return normalized + + def _resolve_observer_target( + self, + session: HonchoSession, + peer: str | None, + ) -> tuple[str, str | None]: + """Resolve observer and target peer IDs for context/search/profile queries.""" + target_peer_id = self._resolve_peer_id(session, peer) + + if target_peer_id == session.assistant_peer_id: + return session.assistant_peer_id, session.assistant_peer_id + + if self._ai_observe_others: + return session.assistant_peer_id, target_peer_id + + return target_peer_id, None + + def get_peer_card(self, session_key: str, peer: str = "user") -> list[str]: + """ + Fetch a peer card — a curated list of key facts. Fast, no LLM reasoning. Returns raw structured facts Honcho has - inferred about the user (name, role, preferences, patterns). + inferred about the target peer (name, role, preferences, patterns). Empty list if unavailable. """ session = self._cache.get(session_key) @@ -924,12 +987,19 @@ def get_peer_card(self, session_key: str) -> list[str]: return [] try: - return self._fetch_peer_card(session.user_peer_id) + observer_peer_id, target_peer_id = self._resolve_observer_target(session, peer) + return self._fetch_peer_card(observer_peer_id, target=target_peer_id) except Exception as e: logger.debug("Failed to fetch peer card from Honcho: %s", e) return [] - def search_context(self, session_key: str, query: str, max_tokens: int = 800) -> str: + def search_context( + self, + session_key: str, + query: str, + max_tokens: int = 800, + peer: str = "user", + ) -> str: """ Semantic search over Honcho session context. @@ -941,6 +1011,7 @@ def search_context(self, session_key: str, query: str, max_tokens: int = 800) -> session_key: Session to search against. query: Search query for semantic matching. max_tokens: Token budget for returned content. + peer: Peer alias or explicit peer ID to search about. Returns: Relevant context excerpts as a string, or empty string if none. @@ -950,7 +1021,13 @@ def search_context(self, session_key: str, query: str, max_tokens: int = 800) -> return "" try: - ctx = self._fetch_peer_context(session.user_peer_id, search_query=query) + observer_peer_id, target = self._resolve_observer_target(session, peer) + + ctx = self._fetch_peer_context( + observer_peer_id, + search_query=query, + target=target, + ) parts = [] if ctx["representation"]: parts.append(ctx["representation"]) @@ -962,16 +1039,17 @@ def search_context(self, session_key: str, query: str, max_tokens: int = 800) -> logger.debug("Honcho search_context failed: %s", e) return "" - def create_conclusion(self, session_key: str, content: str) -> bool: - """Write a conclusion about the user back to Honcho. + def create_conclusion(self, session_key: str, content: str, peer: str = "user") -> bool: + """Write a conclusion about a target peer back to Honcho. - Conclusions are facts the AI peer observes about the user — - preferences, corrections, clarifications, project context. - They feed into the user's peer card and representation. + Conclusions are facts a peer observes about another peer or itself — + preferences, corrections, clarifications, and project context. + They feed into the target peer's card and representation. Args: session_key: Session to associate the conclusion with. - content: The conclusion text (e.g. "User prefers dark mode"). + content: The conclusion text. + peer: Peer alias or explicit peer ID. "user" is the default alias. Returns: True on success, False on failure. @@ -985,25 +1063,90 @@ def create_conclusion(self, session_key: str, content: str) -> bool: return False try: - if self._ai_observe_others: - # AI peer creates conclusion about user (cross-observation) + target_peer_id = self._resolve_peer_id(session, peer) + if target_peer_id is None: + logger.warning("Could not resolve conclusion peer '%s' for session '%s'", peer, session_key) + return False + + if target_peer_id == session.assistant_peer_id: assistant_peer = self._get_or_create_peer(session.assistant_peer_id) - conclusions_scope = assistant_peer.conclusions_of(session.user_peer_id) + conclusions_scope = assistant_peer.conclusions_of(session.assistant_peer_id) + elif self._ai_observe_others: + assistant_peer = self._get_or_create_peer(session.assistant_peer_id) + conclusions_scope = assistant_peer.conclusions_of(target_peer_id) else: - # AI can't observe others — user peer creates self-conclusion - user_peer = self._get_or_create_peer(session.user_peer_id) - conclusions_scope = user_peer.conclusions_of(session.user_peer_id) + target_peer = self._get_or_create_peer(target_peer_id) + conclusions_scope = target_peer.conclusions_of(target_peer_id) conclusions_scope.create([{ "content": content.strip(), "session_id": session.honcho_session_id, }]) - logger.info("Created conclusion for %s: %s", session_key, content[:80]) + logger.info("Created conclusion about %s for %s: %s", target_peer_id, session_key, content[:80]) return True except Exception as e: logger.error("Failed to create conclusion: %s", e) return False + def delete_conclusion(self, session_key: str, conclusion_id: str, peer: str = "user") -> bool: + """Delete a conclusion by ID. Use only for PII removal. + + Args: + session_key: Session key for peer resolution. + conclusion_id: The conclusion ID to delete. + peer: Peer alias or explicit peer ID. + + Returns: + True on success, False on failure. + """ + session = self._cache.get(session_key) + if not session: + return False + try: + target_peer_id = self._resolve_peer_id(session, peer) + if target_peer_id == session.assistant_peer_id: + observer = self._get_or_create_peer(session.assistant_peer_id) + scope = observer.conclusions_of(session.assistant_peer_id) + elif self._ai_observe_others: + observer = self._get_or_create_peer(session.assistant_peer_id) + scope = observer.conclusions_of(target_peer_id) + else: + target_peer = self._get_or_create_peer(target_peer_id) + scope = target_peer.conclusions_of(target_peer_id) + scope.delete(conclusion_id) + logger.info("Deleted conclusion %s for %s", conclusion_id, session_key) + return True + except Exception as e: + logger.error("Failed to delete conclusion %s: %s", conclusion_id, e) + return False + + def set_peer_card(self, session_key: str, card: list[str], peer: str = "user") -> list[str] | None: + """Update a peer's card. + + Args: + session_key: Session key for peer resolution. + card: New peer card as list of fact strings. + peer: Peer alias or explicit peer ID. + + Returns: + Updated card on success, None on failure. + """ + session = self._cache.get(session_key) + if not session: + return None + try: + peer_id = self._resolve_peer_id(session, peer) + if peer_id is None: + logger.warning("Could not resolve peer '%s' for set_peer_card in session '%s'", peer, session_key) + return None + peer_obj = self._get_or_create_peer(peer_id) + result = peer_obj.set_card(card) + logger.info("Updated peer card for %s (%d facts)", peer_id, len(card)) + return result + except Exception as e: + logger.error("Failed to set peer card: %s", e) + return None + def seed_ai_identity(self, session_key: str, content: str, source: str = "manual") -> bool: """ Seed the AI peer's Honcho representation from text content. @@ -1061,7 +1204,7 @@ def get_ai_representation(self, session_key: str) -> dict[str, str]: return {"representation": "", "card": ""} try: - ctx = self._fetch_peer_context(session.assistant_peer_id) + ctx = self._fetch_peer_context(session.assistant_peer_id, target=session.assistant_peer_id) return { "representation": ctx["representation"] or "", "card": "\n".join(ctx["card"]), diff --git a/plugins/memory/openviking/__init__.py b/plugins/memory/openviking/__init__.py index f46d71321e6a..86d7ad5efb1f 100644 --- a/plugins/memory/openviking/__init__.py +++ b/plugins/memory/openviking/__init__.py @@ -10,8 +10,9 @@ Config via environment variables (profile-scoped via each profile's .env): OPENVIKING_ENDPOINT — Server URL (default: http://127.0.0.1:1933) OPENVIKING_API_KEY — API key (required for authenticated servers) - OPENVIKING_ACCOUNT — Tenant account (default: root) + OPENVIKING_ACCOUNT — Tenant account (default: default) OPENVIKING_USER — Tenant user (default: default) + OPENVIKING_AGENT — Tenant agent (default: hermes) Capabilities: - Automatic memory extraction on session commit (6 categories) @@ -80,11 +81,12 @@ class _VikingClient: """Thin HTTP client for the OpenViking REST API.""" def __init__(self, endpoint: str, api_key: str = "", - account: str = "", user: str = ""): + account: str = "", user: str = "", agent: str = ""): self._endpoint = endpoint.rstrip("/") self._api_key = api_key - self._account = account or os.environ.get("OPENVIKING_ACCOUNT", "root") + self._account = account or os.environ.get("OPENVIKING_ACCOUNT", "default") self._user = user or os.environ.get("OPENVIKING_USER", "default") + self._agent = agent or os.environ.get("OPENVIKING_AGENT", "hermes") self._httpx = _get_httpx() if self._httpx is None: raise ImportError("httpx is required for OpenViking: pip install httpx") @@ -94,6 +96,7 @@ def _headers(self) -> dict: "Content-Type": "application/json", "X-OpenViking-Account": self._account, "X-OpenViking-User": self._user, + "X-OpenViking-Agent": self._agent, } if self._api_key: h["X-API-Key"] = self._api_key @@ -282,20 +285,44 @@ def get_config_schema(self): }, { "key": "api_key", - "description": "OpenViking API key", + "description": "OpenViking API key (leave blank for local dev mode)", "secret": True, "env_var": "OPENVIKING_API_KEY", }, + { + "key": "account", + "description": "OpenViking tenant account ID ([default], used when local mode, OPENVIKING_API_KEY is empty)", + "default": "default", + "env_var": "OPENVIKING_ACCOUNT", + }, + { + "key": "user", + "description": "OpenViking user ID within the account ([default], used when local mode, OPENVIKING_API_KEY is empty)", + "default": "default", + "env_var": "OPENVIKING_USER", + }, + { + "key": "agent", + "description": "OpenViking agent ID within the account ([hermes], useful in multi-agent mode)", + "default": "hermes", + "env_var": "OPENVIKING_AGENT", + }, ] def initialize(self, session_id: str, **kwargs) -> None: self._endpoint = os.environ.get("OPENVIKING_ENDPOINT", _DEFAULT_ENDPOINT) self._api_key = os.environ.get("OPENVIKING_API_KEY", "") + self._account = os.environ.get("OPENVIKING_ACCOUNT", "default") + self._user = os.environ.get("OPENVIKING_USER", "default") + self._agent = os.environ.get("OPENVIKING_AGENT", "hermes") self._session_id = session_id self._turn_count = 0 try: - self._client = _VikingClient(self._endpoint, self._api_key) + self._client = _VikingClient( + self._endpoint, self._api_key, + account=self._account, user=self._user, agent=self._agent, + ) if not self._client.health(): logger.warning("OpenViking server at %s is not reachable", self._endpoint) self._client = None @@ -325,7 +352,8 @@ def system_prompt_block(self) -> str: "(abstract/overview/full), viking_browse to explore.\n" "Use viking_remember to store facts, viking_add_resource to index URLs/docs." ) - except Exception: + except Exception as e: + logger.warning("OpenViking system_prompt_block failed: %s", e) return ( "# OpenViking Knowledge Base\n" f"Active. Endpoint: {self._endpoint}\n" @@ -351,7 +379,10 @@ def queue_prefetch(self, query: str, *, session_id: str = "") -> None: def _run(): try: - client = _VikingClient(self._endpoint, self._api_key) + client = _VikingClient( + self._endpoint, self._api_key, + account=self._account, user=self._user, agent=self._agent, + ) resp = client.post("/api/v1/search/find", { "query": query, "top_k": 5, @@ -386,7 +417,10 @@ def sync_turn(self, user_content: str, assistant_content: str, *, session_id: st def _sync(): try: - client = _VikingClient(self._endpoint, self._api_key) + client = _VikingClient( + self._endpoint, self._api_key, + account=self._account, user=self._user, agent=self._agent, + ) sid = self._session_id # Add user message @@ -442,7 +476,10 @@ def on_memory_write(self, action: str, target: str, content: str) -> None: def _write(): try: - client = _VikingClient(self._endpoint, self._api_key) + client = _VikingClient( + self._endpoint, self._api_key, + account=self._account, user=self._user, agent=self._agent, + ) # Add as a user message with memory context so the commit # picks it up as an explicit memory during extraction client.post(f"/api/v1/sessions/{self._session_id}/messages", { @@ -509,19 +546,24 @@ def _tool_search(self, args: dict) -> str: result = resp.get("result", {}) # Format results for the model — keep it concise - formatted = [] + scored_entries = [] for ctx_type in ("memories", "resources", "skills"): items = result.get(ctx_type, []) for item in items: + raw_score = item.get("score") + sort_score = raw_score if raw_score is not None else 0.0 entry = { "uri": item.get("uri", ""), "type": ctx_type.rstrip("s"), - "score": round(item.get("score", 0), 3), + "score": round(raw_score, 3) if raw_score is not None else 0.0, "abstract": item.get("abstract", ""), } if item.get("relations"): entry["related"] = [r.get("uri") for r in item["relations"][:3]] - formatted.append(entry) + scored_entries.append((sort_score, entry)) + + scored_entries.sort(key=lambda x: x[0], reverse=True) + formatted = [entry for _, entry in scored_entries] return json.dumps({ "results": formatted, diff --git a/plugins/strike-freedom-cockpit/README.md b/plugins/strike-freedom-cockpit/README.md new file mode 100644 index 000000000000..c24c5e3882b0 --- /dev/null +++ b/plugins/strike-freedom-cockpit/README.md @@ -0,0 +1,70 @@ +# Strike Freedom Cockpit — dashboard skin demo + +Demonstrates how the dashboard skin+plugin system can be used to build a +fully custom cockpit-style reskin without touching the core dashboard. + +Two pieces: + +- `theme/strike-freedom.yaml` — a dashboard theme YAML that paints the + palette, typography, layout variant (`cockpit`), component chrome + (notched card corners, scanlines, accent colors), and declares asset + slots (`hero`, `crest`, `bg`). +- `dashboard/` — a plugin that populates the `sidebar`, `header-left`, + and `footer-right` slots reserved by the cockpit layout. The sidebar + renders an MS-STATUS panel with segmented telemetry bars driven by + real agent status; the header-left injects a COMPASS crest; the + footer-right replaces the default org tagline. + +## Install + +1. **Theme** — copy the theme YAML into your Hermes home: + + ``` + cp theme/strike-freedom.yaml ~/.hermes/dashboard-themes/ + ``` + +2. **Plugin** — the `dashboard/` directory gets auto-discovered because + it lives under `plugins/` in the repo. On a user install, copy the + whole plugin directory into `~/.hermes/plugins/`: + + ``` + cp -r . ~/.hermes/plugins/strike-freedom-cockpit + ``` + +3. Restart the web UI (or `GET /api/dashboard/plugins/rescan`), open it, + pick **Strike Freedom** from the theme switcher. + +## Customising the artwork + +The sidebar plugin reads `--theme-asset-hero` and `--theme-asset-crest` +from the active theme. Drop your own URLs into the theme YAML: + +```yaml +assets: + hero: "/my-images/strike-freedom.png" + crest: "/my-images/compass-crest.svg" + bg: "/my-images/cosmic-era-bg.jpg" +``` + +The plugin reads those at render time — no plugin code changes needed +to swap artwork across themes. + +## What this demo proves + +The dashboard skin+plugin system supports (ref: `web/src/themes/types.ts`, +`web/src/plugins/slots.ts`): + +- Palette, typography, font URLs, density, radius — already present +- **Asset URLs exposed as CSS vars** (bg / hero / crest / logo / + sidebar / header + arbitrary `custom.*`) +- **Raw `customCSS` blocks** injected as scoped ` + + +
+ +
+
+
+

[PROJECT NAME] Architecture

+
+

[Subtitle description]

+
+ + +
+ + + + + + + + + + + + + + + + + + + Users + Browser/Mobile + + + + Auth Provider + OAuth 2.0 + + + + AWS Region: us-west-2 + + + + CloudFront + CDN + + + + S3 Buckets + • bucket-one + • bucket-two + • bucket-three + OAI Protected + + + + sg-name :port + + + + Load Balancer + HTTPS :443 + + + + API Server + FastAPI :8000 + + + + Database + PostgreSQL + + + + Frontend + React + TypeScript + Additional detail + More info + domain.example.com + + + + + + HTTPS + + + + + + + OAI + + + + + TLS + + + + JWT + PKCE + + + Legend + + + Frontend + + + Backend + + + Cloud Service + + + Database + + + Security + + + Auth Flow + + + Security Group + +
+ + +
+
+
+
+

Card Title 1

+
+
    +
  • • Item one
    • +
  • • Item two
    • +
  • • Item three
    • +
  • • Item four
    • +
+
+ +
+
+
+

Card Title 2

+
+
    +
  • • Item one
    • +
  • • Item two
    • +
  • • Item three
    • +
  • • Item four
    • +
+
+ +
+
+
+

Card Title 3

+
+
    +
  • • Item one
    • +
  • • Item two
    • +
  • • Item three
    • +
  • • Item four
    • +
+
+
+ + +

+ [Project Name] • [Additional metadata] +

+
+ + diff --git a/skills/creative/baoyu-comic/PORT_NOTES.md b/skills/creative/baoyu-comic/PORT_NOTES.md new file mode 100644 index 000000000000..637b7befb54e --- /dev/null +++ b/skills/creative/baoyu-comic/PORT_NOTES.md @@ -0,0 +1,77 @@ +# Port Notes — baoyu-comic + +Ported from [JimLiu/baoyu-skills](https://github.com/JimLiu/baoyu-skills) v1.56.1. + +## Changes from upstream + +### SKILL.md adaptations + +| Change | Upstream | Hermes | +|--------|----------|--------| +| Metadata namespace | `openclaw` | `hermes` (with `tags` + `homepage`) | +| Trigger | Slash commands / CLI flags | Natural language skill matching | +| User config | EXTEND.md file (project/user/XDG paths) | Removed — not part of Hermes infra | +| User prompts | `AskUserQuestion` (batched) | `clarify` tool (one question at a time) | +| Image generation | baoyu-imagine (Bun/TypeScript, supports `--ref`) | `image_generate` — **prompt-only**, returns a URL; no reference image input; agent must download the URL to the output directory | +| PDF assembly | `scripts/merge-to-pdf.ts` (Bun + `pdf-lib`) | Removed — the PDF merge step is out of scope for this port; pages are delivered as PNGs only | +| Platform support | Linux/macOS/Windows/WSL/PowerShell | Linux/macOS only | +| File operations | Generic instructions | Hermes file tools (`write_file`, `read_file`) | + +### Structural removals + +- **`references/config/` directory** (removed entirely): + - `first-time-setup.md` — blocking first-time setup flow for EXTEND.md + - `preferences-schema.md` — EXTEND.md YAML schema + - `watermark-guide.md` — watermark config (tied to EXTEND.md) +- **`scripts/` directory** (removed entirely): upstream's `merge-to-pdf.ts` depended on `pdf-lib`, which is not declared anywhere in the Hermes repo. Rather than add a new dependency, the port drops PDF assembly and delivers per-page PNGs. +- **Workflow Step 8 (Merge to PDF)** removed from `workflow.md`; Step 9 (Completion report) renumbered to Step 8. +- **Workflow Step 1.1** — "Load Preferences (EXTEND.md)" section removed from `workflow.md`; steps 1.2/1.3 renumbered to 1.1/1.2. +- **Generic "User Input Tools" and "Image Generation Tools" preambles** — SKILL.md no longer lists fallback rules for multiple possible tools; it references `clarify` and `image_generate` directly. + +### Image generation strategy changes + +`image_generate`'s schema accepts only `prompt` and `aspect_ratio` (`landscape` | `portrait` | `square`). Upstream's reference-image flow (`--ref characters.png` for character consistency, plus user-supplied refs for style/palette/scene) does not map to this tool, so the workflow was restructured: + +- **Character sheet PNG** is still generated for multi-page comics, but it is repositioned as a **human-facing review artifact** (for visual verification) and a reference for later regenerations / manual prompt edits. Page prompts themselves are built from the **text descriptions** in `characters/characters.md` (embedded inline during Step 5). `image_generate` never sees the PNG as a visual input. +- **User-supplied reference images** are reduced to `style` / `palette` / `scene` trait extraction — traits are embedded in the prompt body; the image files themselves are kept only for provenance under `refs/`. +- **Page prompts** now mandate that character descriptions are embedded inline (copied from `characters/characters.md`) — this is the only mechanism left to enforce cross-page character consistency. +- **Download step** — after every `image_generate` call, the returned URL is fetched to disk (e.g., `curl -fsSL "" -o .png`) and verified before the workflow advances. + +### SKILL.md reductions + +- CLI option columns (`--art`, `--tone`, `--layout`, `--aspect`, `--lang`, `--ref`, `--storyboard-only`, `--prompts-only`, `--images-only`, `--regenerate`) converted to plain-English option descriptions. +- Preset files (`presets/*.md`) and `ohmsha-guide.md`: `` `--style X` `` / `` `--art X --tone Y` `` shorthand rewritten to `art=X, tone=Y` + natural-language references. +- `partial-workflows.md`: per-skill slash command invocations rewritten as user-intent cues; PDF-related outputs removed. +- `auto-selection.md`: priority order dropped the EXTEND.md tier. +- `analysis-framework.md`: language-priority comment updated (user option → conversation → source). + +### File naming convention + +Source content pasted by the user is saved as `source-{slug}.md`, where `{slug}` is the kebab-case topic slug used for the output directory. Backups follow the same pattern with a `-backup-YYYYMMDD-HHMMSS` suffix. SKILL.md and `workflow.md` now agree on this single convention. + +### What was preserved verbatim + +- All 6 art-style definitions (`references/art-styles/`) +- All 7 tone definitions (`references/tones/`) +- All 7 layout definitions (`references/layouts/`) +- Core templates: `character-template.md`, `storyboard-template.md`, `base-prompt.md` +- Preset bodies (only the first few intro lines adapted; special rules unchanged) +- Author, version, homepage attribution + +## Syncing with upstream + +To pull upstream updates: + +```bash +# Compare versions +curl -sL https://raw.githubusercontent.com/JimLiu/baoyu-skills/main/skills/baoyu-comic/SKILL.md | head -5 +# Look for the version: line + +# Diff a reference file +diff <(curl -sL https://raw.githubusercontent.com/JimLiu/baoyu-skills/main/skills/baoyu-comic/references/art-styles/manga.md) \ + references/art-styles/manga.md +``` + +Art-style, tone, and layout reference files can usually be overwritten directly (they're upstream-verbatim). `SKILL.md`, `references/workflow.md`, `references/partial-workflows.md`, `references/auto-selection.md`, `references/analysis-framework.md`, `references/ohmsha-guide.md`, and `references/presets/*.md` must be manually merged since they contain Hermes-specific adaptations. + +If upstream adds a Hermes-compatible PDF merge step (no extra npm deps), restore `scripts/` and reintroduce Step 8 in `workflow.md`. diff --git a/skills/creative/baoyu-comic/SKILL.md b/skills/creative/baoyu-comic/SKILL.md new file mode 100644 index 000000000000..d3c89ed4c7f3 --- /dev/null +++ b/skills/creative/baoyu-comic/SKILL.md @@ -0,0 +1,246 @@ +--- +name: baoyu-comic +description: Knowledge comic creator supporting multiple art styles and tones. Creates original educational comics with detailed panel layouts and sequential image generation. Use when user asks to create "知识漫画", "教育漫画", "biography comic", "tutorial comic", or "Logicomix-style comic". +version: 1.56.1 +author: 宝玉 (JimLiu) +license: MIT +metadata: + hermes: + tags: [comic, knowledge-comic, creative, image-generation] + homepage: https://github.com/JimLiu/baoyu-skills#baoyu-comic +--- + +# Knowledge Comic Creator + +Adapted from [baoyu-comic](https://github.com/JimLiu/baoyu-skills) for Hermes Agent's tool ecosystem. + +Create original knowledge comics with flexible art style × tone combinations. + +## When to Use + +Trigger this skill when the user asks to create a knowledge/educational comic, biography comic, tutorial comic, or uses terms like "知识漫画", "教育漫画", or "Logicomix-style". The user provides content (text, file path, URL, or topic) and optionally specifies art style, tone, layout, aspect ratio, or language. + +## Reference Images + +Hermes' `image_generate` tool is **prompt-only** — it accepts a text prompt and an aspect ratio, and returns an image URL. It does **NOT** accept reference images. When the user supplies a reference image, use it to **extract traits in text** that get embedded in every page prompt: + +**Intake**: Accept file paths when the user provides them (or pastes images in conversation). +- File path(s) → copy to `refs/NN-ref-{slug}.{ext}` alongside the comic output for provenance +- Pasted image with no path → ask the user for the path via `clarify`, or extract style traits verbally as a text fallback +- No reference → skip this section + +**Usage modes** (per reference): + +| Usage | Effect | +|-------|--------| +| `style` | Extract style traits (line treatment, texture, mood) and append to every page's prompt body | +| `palette` | Extract hex colors and append to every page's prompt body | +| `scene` | Extract scene composition or subject notes and append to the relevant page(s) | + +**Record in each page's prompt frontmatter** when refs exist: + +```yaml +references: + - ref_id: 01 + filename: 01-ref-scene.png + usage: style + traits: "muted earth tones, soft-edged ink wash, low-contrast backgrounds" +``` + +Character consistency is driven by **text descriptions** in `characters/characters.md` (written in Step 3) that get embedded inline in every page prompt (Step 5). The optional PNG character sheet generated in Step 7.1 is a human-facing review artifact, not an input to `image_generate`. + +## Options + +### Visual Dimensions + +| Option | Values | Description | +|--------|--------|-------------| +| Art | ligne-claire (default), manga, realistic, ink-brush, chalk, minimalist | Art style / rendering technique | +| Tone | neutral (default), warm, dramatic, romantic, energetic, vintage, action | Mood / atmosphere | +| Layout | standard (default), cinematic, dense, splash, mixed, webtoon, four-panel | Panel arrangement | +| Aspect | 3:4 (default, portrait), 4:3 (landscape), 16:9 (widescreen) | Page aspect ratio | +| Language | auto (default), zh, en, ja, etc. | Output language | +| Refs | File paths | Reference images used for style / palette trait extraction (not passed to the image model). See [Reference Images](#reference-images) above. | + +### Partial Workflow Options + +| Option | Description | +|--------|-------------| +| Storyboard only | Generate storyboard only, skip prompts and images | +| Prompts only | Generate storyboard + prompts, skip images | +| Images only | Generate images from existing prompts directory | +| Regenerate N | Regenerate specific page(s) only (e.g., `3` or `2,5,8`) | + +Details: [references/partial-workflows.md](references/partial-workflows.md) + +### Art, Tone & Preset Catalogue + +- **Art styles** (6): `ligne-claire`, `manga`, `realistic`, `ink-brush`, `chalk`, `minimalist`. Full definitions at `references/art-styles/