Skip to content

Commit 95c3bdb

Browse files
calvinmcleancalvinmclean
authored
Improve ability to list workflows and add prompt/tool testing setup (#286)
* Improve ability to list workflows * Add prompt/tool call testing setup * Minor improvements * Fix/improve README * Improvements from claude PR review * Add CLI flag to explicitly enable the new test * Add log if integration test is skipped * use build tags to skip tests completely
1 parent 8175e2f commit 95c3bdb

4 files changed

Lines changed: 361 additions & 2 deletions

File tree

‎integration_test/README.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Integration Tests
2+
3+
End-to-end tests that run against a real LLM. Excluded from `go test ./...` by default via the `//go:build integration` build tag.
4+
5+
## Running
6+
7+
```bash
8+
ANTHROPIC_API_KEY=... go test -tags integration ./integration_test/ -runs 5
9+
```
10+
11+
- `-tags integration` enables compilation of the tests (required; without it they are completely skipped).
12+
- `-runs` controls how many times each prompt is run per test (default: 5).
13+
14+
## Agent Configuration
15+
16+
The harness loads the builtin `nanobot` agent via `config.Load(".nanobot/", true)`. If a `.nanobot/` directory exists in the working directory when the tests run, its config is merged in — allowing you to override the agent's instructions, add MCP servers, or adjust other settings without changing test code.
17+
18+
If no `.nanobot/` directory exists, the tests run against the default builtin agent.
19+
20+
## Adding a Test
21+
22+
Tests use shared tooling:
23+
24+
- **`newTestRuntime(t, completer, recorder)`** — creates a `Runtime` wired with a recording/intercepting system server and returns a context and agent service ready for execution.
25+
- **`runAgent(ctx, svc, prompt)`** — runs the agent with the given user prompt.
26+
- **`newRecorder(handlers)`** — creates a `toolCallRecorder` with custom `ToolHandler`s for specific tools. By default, `config` and `getSkill` always pass through to the real server; all other tools return an error unless a handler is registered.
27+
- **`recorder.find(name)`** — returns the first recorded call for a tool, or nil.
28+
- **`recorder.summary()`** — returns a formatted list of all tool calls with arguments, useful in failure messages.
29+
30+
### Example
31+
32+
```go
33+
func TestMySkillBehavior(t *testing.T) {
34+
apiKey := os.Getenv("ANTHROPIC_API_KEY")
35+
if apiKey == "" {
36+
t.Fatal("ANTHROPIC_API_KEY must be set when running integration tests")
37+
}
38+
39+
completer := llm.NewClient(llm.Config{
40+
DefaultModel: "claude-sonnet-4-6",
41+
Anthropic: anthropic.Config{APIKey: apiKey},
42+
})
43+
44+
recorder := newRecorder(map[string]ToolHandler{
45+
// Register tools the agent is expected to call with mock responses.
46+
"bash": func(req mcp.CallToolRequest) *mcp.CallToolResult {
47+
return &mcp.CallToolResult{
48+
Content: []mcp.Content{{Type: "text", Text: "mock output"}},
49+
}
50+
},
51+
})
52+
53+
ctx, svc := newTestRuntime(t, completer, recorder)
54+
if err := runAgent(ctx, svc, "my prompt"); err != nil {
55+
t.Fatalf("Complete() failed: %v", err)
56+
}
57+
58+
call := recorder.find("bash")
59+
if call == nil {
60+
t.Fatalf("expected bash call\nall tool calls:\n%s", recorder.summary())
61+
}
62+
// assert call.Arguments as needed
63+
}
64+
```

‎integration_test/main_test.go‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
//go:build integration
2+
3+
// Package integration_test contains integration tests that require a real LLM.
4+
// Build with -tags integration to include them.
5+
package integration_test
6+
7+
import (
8+
"context"
9+
"encoding/json"
10+
"flag"
11+
"fmt"
12+
"strings"
13+
"sync"
14+
"testing"
15+
16+
"github.com/nanobot-ai/nanobot/pkg/agents"
17+
"github.com/nanobot-ai/nanobot/pkg/config"
18+
"github.com/nanobot-ai/nanobot/pkg/llm"
19+
"github.com/nanobot-ai/nanobot/pkg/mcp"
20+
"github.com/nanobot-ai/nanobot/pkg/runtime"
21+
"github.com/nanobot-ai/nanobot/pkg/servers/system"
22+
"github.com/nanobot-ai/nanobot/pkg/types"
23+
)
24+
25+
var numRuns = flag.Int("runs", 5, "number of times to run each prompt")
26+
27+
// ToolHandler is a function that handles a mocked tool call and returns a result.
28+
type ToolHandler func(mcp.CallToolRequest) *mcp.CallToolResult
29+
30+
// toolCallRecorder wraps an MCP handler and records every tools/call invocation.
31+
// By default, "config" and "getSkill" pass through to the real server; all other
32+
// tools return an error unless a custom ToolHandler is registered.
33+
type toolCallRecorder struct {
34+
mu sync.Mutex
35+
calls []mcp.CallToolRequest
36+
handlers map[string]ToolHandler
37+
}
38+
39+
func newRecorder(handlers map[string]ToolHandler) *toolCallRecorder {
40+
return &toolCallRecorder{handlers: handlers}
41+
}
42+
43+
func (r *toolCallRecorder) wrap(inner mcp.MessageHandler) mcp.MessageHandler {
44+
return mcp.MessageHandlerFunc(func(ctx context.Context, msg mcp.Message) {
45+
if msg.Method != "tools/call" {
46+
inner.OnMessage(ctx, msg)
47+
return
48+
}
49+
50+
var call mcp.CallToolRequest
51+
if err := json.Unmarshal(msg.Params, &call); err != nil {
52+
msg.SendError(ctx, mcp.ErrRPCInvalidParams.WithMessage("unmarshal tools/call params: %v", err))
53+
return
54+
}
55+
r.mu.Lock()
56+
r.calls = append(r.calls, call)
57+
r.mu.Unlock()
58+
59+
switch call.Name {
60+
case "config", "getSkill":
61+
// Always pass through: config drives agent setup, getSkill loads skills.
62+
inner.OnMessage(ctx, msg)
63+
default:
64+
if handler, ok := r.handlers[call.Name]; ok {
65+
mcp.Invoke(ctx, msg, func(_ context.Context, _ mcp.Message, req mcp.CallToolRequest) (*mcp.CallToolResult, error) {
66+
return handler(req), nil
67+
})
68+
} else {
69+
mcp.Invoke(ctx, msg, func(_ context.Context, _ mcp.Message, _ mcp.CallToolRequest) (*mcp.CallToolResult, error) {
70+
return &mcp.CallToolResult{
71+
IsError: true,
72+
Content: []mcp.Content{{Type: "text", Text: fmt.Sprintf("tool %q is not available in the test environment", call.Name)}},
73+
}, nil
74+
})
75+
}
76+
}
77+
})
78+
}
79+
80+
func (r *toolCallRecorder) find(name string) *mcp.CallToolRequest {
81+
r.mu.Lock()
82+
defer r.mu.Unlock()
83+
for i := range r.calls {
84+
if r.calls[i].Name == name {
85+
c := r.calls[i]
86+
return &c
87+
}
88+
}
89+
return nil
90+
}
91+
92+
// summary returns a multi-line description of all recorded tool calls with their
93+
// arguments, for use in test failure output.
94+
func (r *toolCallRecorder) summary() string {
95+
r.mu.Lock()
96+
defer r.mu.Unlock()
97+
if len(r.calls) == 0 {
98+
return " (no tool calls recorded)"
99+
}
100+
var sb strings.Builder
101+
for i, c := range r.calls {
102+
args, _ := json.MarshalIndent(c.Arguments, " ", " ")
103+
fmt.Fprintf(&sb, " [%d] %s\n args: %s\n", i+1, c.Name, args)
104+
}
105+
return sb.String()
106+
}
107+
108+
// stubMCPHandler returns a minimal, no-tool MCP server used as a placeholder for
109+
// servers that aren't needed in the test but are referenced by the config hook.
110+
func stubMCPHandler() mcp.MessageHandler {
111+
stubTools := mcp.NewServerTools()
112+
return mcp.MessageHandlerFunc(func(ctx context.Context, msg mcp.Message) {
113+
switch msg.Method {
114+
case "initialize":
115+
mcp.Invoke(ctx, msg, func(ctx context.Context, _ mcp.Message, params mcp.InitializeRequest) (*mcp.InitializeResult, error) {
116+
return &mcp.InitializeResult{
117+
ProtocolVersion: params.ProtocolVersion,
118+
Capabilities: mcp.ServerCapabilities{Tools: &mcp.ToolsServerCapability{}},
119+
ServerInfo: mcp.ServerInfo{Name: "stub", Version: "0"},
120+
}, nil
121+
})
122+
case "notifications/initialized", "notifications/cancelled":
123+
case "tools/list":
124+
mcp.Invoke(ctx, msg, stubTools.List)
125+
case "tools/call":
126+
mcp.Invoke(ctx, msg, stubTools.Call)
127+
default:
128+
msg.SendError(ctx, mcp.ErrRPCMethodNotFound.WithMessage("%s", msg.Method))
129+
}
130+
})
131+
}
132+
133+
// newTestRuntime creates a Runtime and installs a toolCallRecorder on nanobot.system.
134+
// The agent has no pre-loaded instructions; it discovers the workflows skill naturally
135+
// via the getSkill tool that the config hook injects.
136+
func newTestRuntime(t *testing.T, completer types.Completer, recorder *toolCallRecorder) (context.Context, *agents.Agents) {
137+
t.Helper()
138+
139+
rt, err := runtime.NewRuntime(context.Background(), llm.Config{})
140+
if err != nil {
141+
t.Fatalf("NewRuntime failed: %v", err)
142+
}
143+
144+
// Wrap the real system server so we can record and intercept tool calls.
145+
rt.AddServer("nanobot.system", func(string) mcp.MessageHandler {
146+
return recorder.wrap(system.NewServer("", ""))
147+
})
148+
// nanobot.tasks is only registered when LoopbackURL+Store are set; stub it so
149+
// the config hook doesn't fail when it adds the server to MCPServers.
150+
rt.AddServer("nanobot.tasks", func(string) mcp.MessageHandler { return stubMCPHandler() })
151+
152+
// Load the production config (builtin nanobot agent) from the standard location.
153+
// config.Load handles a missing .nanobot/ directory gracefully.
154+
cfg, _, err := config.Load(context.Background(), ".nanobot/", true)
155+
if err != nil {
156+
t.Fatalf("config.Load failed: %v", err)
157+
}
158+
159+
agentSvc := agents.New(completer, rt.Service)
160+
ctx := rt.WithTempSession(context.Background(), cfg)
161+
return ctx, agentSvc
162+
}
163+
164+
func runAgent(ctx context.Context, svc *agents.Agents, prompt string) error {
165+
_, err := svc.Complete(ctx, types.CompletionRequest{
166+
Agent: "nanobot",
167+
Input: []types.Message{{
168+
Role: "user",
169+
Items: []types.CompletionItem{{
170+
Content: &mcp.Content{Type: "text", Text: prompt},
171+
}},
172+
}},
173+
})
174+
return err
175+
}

‎integration_test/workflow_skill_test.go‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
//go:build integration
2+
3+
package integration_test
4+
5+
import (
6+
"context"
7+
"fmt"
8+
"os"
9+
"strings"
10+
"testing"
11+
12+
"github.com/nanobot-ai/nanobot/pkg/agents"
13+
"github.com/nanobot-ai/nanobot/pkg/llm"
14+
"github.com/nanobot-ai/nanobot/pkg/llm/anthropic"
15+
"github.com/nanobot-ai/nanobot/pkg/mcp"
16+
"github.com/nanobot-ai/nanobot/pkg/types"
17+
)
18+
19+
// setupWorkflowListing creates a recorder that mocks the glob tool with a pair of
20+
// fake workflow files, and a runtime context ready for agent execution.
21+
func setupWorkflowListing(t *testing.T, completer types.Completer) (context.Context, *agents.Agents, *toolCallRecorder) {
22+
t.Helper()
23+
recorder := newRecorder(map[string]ToolHandler{
24+
"glob": func(_ mcp.CallToolRequest) *mcp.CallToolResult {
25+
return &mcp.CallToolResult{
26+
Content: []mcp.Content{{Type: "text", Text: "workflows/code-review/SKILL.md\nworkflows/deploy/SKILL.md\n"}},
27+
}
28+
},
29+
})
30+
ctx, svc := newTestRuntime(t, completer, recorder)
31+
return ctx, svc, recorder
32+
}
33+
34+
// getSkillDiagnostic returns a string describing whether the model called getSkill,
35+
// used as additional context in glob assertion failures.
36+
func getSkillDiagnostic(recorder *toolCallRecorder) string {
37+
call := recorder.find("getSkill")
38+
if call == nil {
39+
return "model did not call getSkill"
40+
}
41+
name, _ := call.Arguments["name"].(string)
42+
return fmt.Sprintf("model called getSkill(%q)", name)
43+
}
44+
45+
// assertCorrectGlobCall verifies that the recorder captured a glob call with valid
46+
// workflow-discovery parameters. Two forms are accepted:
47+
//
48+
// pattern="**/SKILL.md" path ends with "workflows"
49+
// pattern="workflows/**/SKILL.md" path does NOT contain "workflows" (already in pattern)
50+
func assertCorrectGlobCall(t *testing.T, recorder *toolCallRecorder) {
51+
t.Helper()
52+
53+
globCall := recorder.find("glob")
54+
if globCall == nil {
55+
t.Fatalf("LLM did not call glob (%s)\nall tool calls:\n%s",
56+
getSkillDiagnostic(recorder), recorder.summary())
57+
}
58+
59+
pattern, _ := globCall.Arguments["pattern"].(string)
60+
path, _ := globCall.Arguments["path"].(string)
61+
cleanPath := strings.TrimRight(path, "/")
62+
63+
patternWithPath := pattern == "**/SKILL.md" &&
64+
(cleanPath == "workflows" || strings.HasSuffix(cleanPath, "/workflows"))
65+
patternOnly := pattern == "workflows/**/SKILL.md" &&
66+
cleanPath != "workflows" && !strings.HasSuffix(cleanPath, "/workflows")
67+
68+
if !patternWithPath && !patternOnly {
69+
t.Errorf("unexpected glob call: pattern=%q path=%q\n want: pattern=\"**/SKILL.md\" with path ending in \"workflows\"\n or: pattern=\"workflows/**/SKILL.md\" with path NOT containing \"workflows\" (would double-nest)\n (%s)\nall tool calls:\n%s",
70+
pattern, path, getSkillDiagnostic(recorder), recorder.summary())
71+
}
72+
}
73+
74+
// TestWorkflowListingGlobPattern runs the agent against a real LLM and asserts that
75+
// the model calls glob with the correct pattern (**/SKILL.md) and path (workflows/).
76+
// The glob tool is intercepted and not actually executed.
77+
//
78+
// Requires ANTHROPIC_API_KEY; skipped otherwise.
79+
func TestWorkflowListingGlobPattern(t *testing.T) {
80+
apiKey := os.Getenv("ANTHROPIC_API_KEY")
81+
if apiKey == "" {
82+
t.Fatal("ANTHROPIC_API_KEY must be set when running integration tests")
83+
}
84+
85+
completer := llm.NewClient(llm.Config{
86+
DefaultModel: "claude-sonnet-4-6",
87+
Anthropic: anthropic.Config{APIKey: apiKey},
88+
})
89+
90+
prompts := []string{
91+
"list all workflows",
92+
"show my workflows",
93+
"what workflows do I have?",
94+
"show me my available workflows",
95+
"display all workflows",
96+
}
97+
98+
for _, prompt := range prompts {
99+
t.Run(prompt, func(t *testing.T) {
100+
t.Parallel()
101+
for i := range *numRuns {
102+
t.Run(fmt.Sprintf("run%d", i+1), func(t *testing.T) {
103+
t.Parallel()
104+
ctx, svc, recorder := setupWorkflowListing(t, completer)
105+
106+
err := runAgent(ctx, svc, prompt)
107+
if err != nil {
108+
t.Fatalf("Complete() failed: %v", err)
109+
}
110+
111+
assertCorrectGlobCall(t, recorder)
112+
113+
if call := recorder.find("searchArtifacts"); call != nil {
114+
t.Errorf("model should not call searchArtifacts for local listing prompts\nall tool calls:\n%s", recorder.summary())
115+
}
116+
})
117+
}
118+
})
119+
}
120+
}

‎pkg/servers/system/skills/workflows.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ There are two places workflows can exist:
1616

1717
**When the user asks about "shared workflows", "public workflows", workflows from other users, or wants to discover/find/browse available workflows they haven't installed yet, ALWAYS use `searchArtifacts` to search the remote registry.** Do not just list the local `workflows/` directory — that only shows what is already installed locally.
1818

19-
If the user asks to "list all workflows" or "show my workflows" without specifying shared/public, list the local `workflows/` directory. If they ask to "list shared workflows" or "find workflows", use `searchArtifacts`.
19+
If the user asks to "list all workflows", "show my workflows", "which workflows are installed" or similar without specifying shared/public, find workflows by using the Glob tool (never Bash with find or ls) with pattern `**/SKILL.md` in the `workflows/` directory (do NOT include `workflows/` in both the path and the pattern — that double-nests). If they ask to "list shared workflows" or "find workflows", use `searchArtifacts`.
2020

2121
## When to Use Workflows
2222

@@ -209,7 +209,7 @@ Users may ask to run a workflow at any time — not just immediately after desig
209209

210210
When the user asks you to run a workflow:
211211

212-
1. Load the workflow from `workflows/<name>/SKILL.md`. If you're unsure which workflows are available, list the `workflows/` directory.
212+
1. Load the workflow from `workflows/<name>/SKILL.md`. If you're unsure which workflows are available, use glob pattern `**/SKILL.md` with path `workflows/` to find them.
213213
2. Use TodoWrite to create a todo for each workflow step before you begin. This is your execution plan — the user will follow along.
214214
3. **Present the execution plan to the user.** After creating the todos, present a brief summary of what will be executed and ask the user to confirm before proceeding. For example: "I've planned the following steps: [list steps]. Does this look good to proceed?"
215215
4. **Wait for user approval.** Do not begin execution until the user confirms.

0 commit comments

Comments
 (0)