Skip to content

Commit 0eaad85

Browse files
authored
Merge pull request #42 from mostlydev/fix/runtime-bootstrap-and-openclaw-topology
Fix claw runtime bootstrap and OpenClaw config generation
2 parents 5ae635d + 2511cb5 commit 0eaad85

File tree

10 files changed

+584
-24
lines changed

10 files changed

+584
-24
lines changed

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ The Clawdapus Dash fleet dashboard runs on port **8082** — live service health
4545

4646
Message `@quickstart-bot` in your Discord server. The bot responds through the proxy — it has no direct API access. The dashboard updates live.
4747

48+
`claw up` resolves `${...}` placeholders inside `x-claw` metadata from your shell environment and the pod-local `.env` file before it generates runtime config. You do not need to duplicate handle IDs, guild IDs, or channel IDs into service `environment:` just to make driver config generation work.
49+
4850
See [`examples/quickstart/`](./examples/quickstart/) for the full walkthrough, Telegram/Slack alternatives, and migration from existing OpenClaw.
4951

5052
Or scaffold from scratch:

TESTING.md

Lines changed: 103 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -28,14 +28,111 @@ go test -tags e2e -v ./...
2828

2929
---
3030

31-
## Spike Test (live Discord + Docker required)
31+
## Spike Tests (live Discord + Docker required)
3232

33-
The spike test (`TestSpikeComposeUp`) is the primary end-to-end validation instrument
34-
for the trading-desk example. It builds images, runs `claw up`, verifies all
35-
generated artifacts, starts containers, and confirms live Discord activity.
33+
Spike tests are the live end-to-end validation layer. They require real credentials,
34+
Docker, and a real Discord server. They are not CI tests.
3635

37-
**It is not a CI test.** It requires real credentials and a real Discord server.
38-
Run it when implementing or validating new driver behavior end-to-end.
36+
There are currently two spike paths:
37+
38+
- `TestSpikeRollCall`: the broad driver-parity validation path. Boots all 6 driver
39+
types plus `cllama` passthrough and `clawdash`, sends a Discord roll call, and
40+
verifies runtime-specific responses.
41+
- `TestSpikeComposeUp`: the deeper trading-desk validation path. Focuses on artifact
42+
generation, startup wiring, and Discord activity for the richer multi-service example.
43+
44+
Run a spike test when implementing or validating driver/runtime behavior end to end.
45+
46+
### Rollcall Driver Parity Spike
47+
48+
The rollcall spike (`TestSpikeRollCall`) is the best single validation path for
49+
cross-driver support. It uses [`examples/rollcall/`](./examples/rollcall/) and
50+
exercises:
51+
52+
- `openclaw`
53+
- `nullclaw`
54+
- `microclaw`
55+
- `nanoclaw`
56+
- `nanobot`
57+
- `picoclaw`
58+
- `cllama` passthrough
59+
- `clawdash`
60+
61+
### What it validates
62+
63+
- Base images build for all 6 driver families
64+
- Agent images build from their `Clawfile`s
65+
- `claw up` succeeds on the rollcall pod
66+
- All agent containers converge to healthy/running state
67+
- A Discord trigger message causes each runtime to post an AI-generated
68+
self-identification response
69+
- `cllama` exposes cost data after traffic flows through the proxy
70+
71+
### Prerequisites
72+
73+
- Docker running
74+
- Go toolchain
75+
- A Discord server with:
76+
- One bot application token with permission to read and post in the target channel
77+
- A text channel for the roll call
78+
- An incoming webhook URL for posting the non-bot trigger message
79+
- At least one LLM provider key:
80+
- `OPENROUTER_API_KEY` or
81+
- `ANTHROPIC_API_KEY`
82+
83+
### Setup
84+
85+
```bash
86+
cd examples/rollcall
87+
cp .env.example .env
88+
# Edit .env with real values
89+
```
90+
91+
Required `.env` values:
92+
93+
| Variable | What it is |
94+
|----------|------------|
95+
| `DISCORD_BOT_TOKEN` | Bot token used by all rollcall services |
96+
| `DISCORD_BOT_ID` | Discord application/user ID for that bot |
97+
| `DISCORD_GUILD_ID` | Discord server (guild) ID |
98+
| `ROLLCALL_CHANNEL_ID` | Channel ID used for the roll call |
99+
| `DISCORD_WEBHOOK_URL` | Incoming webhook URL used to post the trigger message |
100+
| `OPENROUTER_API_KEY` | Optional, used by OpenRouter-backed passthrough services |
101+
| `ANTHROPIC_API_KEY` | Optional, used by Anthropic-backed passthrough services |
102+
103+
### Running
104+
105+
```bash
106+
go test -tags spike -v -run TestSpikeRollCall ./cmd/claw/...
107+
```
108+
109+
Expected duration: 3-10 minutes depending on Docker cache warmth, image build time,
110+
Discord gateway connection, and LLM latency.
111+
112+
### Output
113+
114+
The test logs:
115+
116+
- Image builds and compose startup progress
117+
- Health convergence for each rollcall container
118+
- Matching Discord responses for each runtime name
119+
- Recent container logs on teardown or failure
120+
- `cllama` health/cost endpoint checks
121+
122+
### Cleanup
123+
124+
Containers are torn down automatically on success, failure, or Ctrl-C.
125+
126+
If a run is killed hard, clean up manually:
127+
128+
```bash
129+
docker compose -p rollcall down --volumes --remove-orphans
130+
```
131+
132+
## Trading-Desk Spike
133+
134+
The trading-desk spike (`TestSpikeComposeUp`) remains the deeper artifact and
135+
workflow validation instrument for the richer `examples/trading-desk/` example.
39136

40137
### What it validates
41138

cmd/claw/compose_up.go

Lines changed: 161 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
package main
22

33
import (
4+
"bufio"
45
"encoding/json"
56
"fmt"
67
"os"
78
"os/exec"
89
"path/filepath"
10+
"regexp"
911
"sort"
1012
"strconv"
1113
"strings"
@@ -23,6 +25,8 @@ import (
2325

2426
var composeUpDetach bool
2527

28+
var envVarPattern = regexp.MustCompile(`\$\{([A-Z_][A-Z0-9_]*)\}`)
29+
2630
var (
2731
extractServiceSkillFromImage = runtime.ExtractServiceSkill
2832
writeRuntimeFile = os.WriteFile
@@ -31,6 +35,8 @@ var (
3135
generateClawDockerfile = build.Generate
3236
buildGeneratedImage = build.BuildFromGenerated
3337
dockerBuildTaggedImage = dockerBuildTaggedImageDefault
38+
findClawdapusRepoRoot = findRepoRoot
39+
runInfraDockerCommand = runInfraDockerCommandDefault
3440
)
3541

3642
var composeUpCmd = &cobra.Command{
@@ -69,9 +75,12 @@ func runComposeUp(podFile string) error {
6975
if err != nil {
7076
return fmt.Errorf("resolve pod directory: %w", err)
7177
}
78+
if err := resolveRuntimePlaceholders(podDir, p); err != nil {
79+
return fmt.Errorf("resolve x-claw runtime placeholders: %w", err)
80+
}
7281
runtimeDir := filepath.Join(podDir, ".claw-runtime")
73-
if err := os.MkdirAll(runtimeDir, 0700); err != nil {
74-
return fmt.Errorf("create runtime dir: %w", err)
82+
if err := resetRuntimeDir(runtimeDir); err != nil {
83+
return fmt.Errorf("reset runtime dir: %w", err)
7584
}
7685

7786
results := make(map[string]*driver.MaterializeResult)
@@ -525,6 +534,139 @@ func runComposeUp(podFile string) error {
525534
return nil
526535
}
527536

537+
func resetRuntimeDir(path string) error {
538+
if err := os.RemoveAll(path); err != nil {
539+
return err
540+
}
541+
return os.MkdirAll(path, 0o700)
542+
}
543+
544+
func resolveRuntimePlaceholders(podDir string, p *pod.Pod) error {
545+
env, err := loadRuntimeEnv(podDir)
546+
if err != nil {
547+
return err
548+
}
549+
expand := func(value string) string {
550+
return envVarPattern.ReplaceAllStringFunc(value, func(match string) string {
551+
key := match[2 : len(match)-1]
552+
if v, ok := env[key]; ok {
553+
return v
554+
}
555+
return match
556+
})
557+
}
558+
559+
for _, svc := range p.Services {
560+
if svc == nil || svc.Claw == nil {
561+
continue
562+
}
563+
svc.Claw.Agent = expand(svc.Claw.Agent)
564+
svc.Claw.Persona = expand(svc.Claw.Persona)
565+
for i, value := range svc.Claw.Cllama {
566+
svc.Claw.Cllama[i] = expand(value)
567+
}
568+
for key, value := range svc.Claw.CllamaEnv {
569+
svc.Claw.CllamaEnv[key] = expand(value)
570+
}
571+
for i, value := range svc.Claw.Skills {
572+
svc.Claw.Skills[i] = expand(value)
573+
}
574+
for i := range svc.Claw.Invoke {
575+
svc.Claw.Invoke[i].Schedule = expand(svc.Claw.Invoke[i].Schedule)
576+
svc.Claw.Invoke[i].Message = expand(svc.Claw.Invoke[i].Message)
577+
svc.Claw.Invoke[i].Name = expand(svc.Claw.Invoke[i].Name)
578+
svc.Claw.Invoke[i].To = expand(svc.Claw.Invoke[i].To)
579+
}
580+
for _, handle := range svc.Claw.Handles {
581+
if handle == nil {
582+
continue
583+
}
584+
handle.ID = expand(handle.ID)
585+
handle.Username = expand(handle.Username)
586+
for gi := range handle.Guilds {
587+
handle.Guilds[gi].ID = expand(handle.Guilds[gi].ID)
588+
handle.Guilds[gi].Name = expand(handle.Guilds[gi].Name)
589+
for ci := range handle.Guilds[gi].Channels {
590+
handle.Guilds[gi].Channels[ci].ID = expand(handle.Guilds[gi].Channels[ci].ID)
591+
handle.Guilds[gi].Channels[ci].Name = expand(handle.Guilds[gi].Channels[ci].Name)
592+
}
593+
}
594+
}
595+
for i := range svc.Claw.Surfaces {
596+
svc.Claw.Surfaces[i].Target = expand(svc.Claw.Surfaces[i].Target)
597+
svc.Claw.Surfaces[i].AccessMode = expand(svc.Claw.Surfaces[i].AccessMode)
598+
if cc := svc.Claw.Surfaces[i].ChannelConfig; cc != nil {
599+
cc.DM.Policy = expand(cc.DM.Policy)
600+
for j, value := range cc.DM.AllowFrom {
601+
cc.DM.AllowFrom[j] = expand(value)
602+
}
603+
expandedGuilds := make(map[string]driver.ChannelGuildConfig, len(cc.Guilds))
604+
for guildID, guildCfg := range cc.Guilds {
605+
guildCfg.Policy = expand(guildCfg.Policy)
606+
users := make([]string, len(guildCfg.Users))
607+
for j, value := range guildCfg.Users {
608+
users[j] = expand(value)
609+
}
610+
guildCfg.Users = users
611+
expandedGuilds[expand(guildID)] = guildCfg
612+
}
613+
cc.Guilds = expandedGuilds
614+
}
615+
}
616+
}
617+
return nil
618+
}
619+
620+
func loadRuntimeEnv(podDir string) (map[string]string, error) {
621+
env := make(map[string]string)
622+
dotEnvPath := filepath.Join(podDir, ".env")
623+
if fileEnv, err := readDotEnvFile(dotEnvPath); err == nil {
624+
for key, value := range fileEnv {
625+
env[key] = value
626+
}
627+
} else if !os.IsNotExist(err) {
628+
return nil, err
629+
}
630+
for _, entry := range os.Environ() {
631+
eq := strings.IndexByte(entry, '=')
632+
if eq < 0 {
633+
continue
634+
}
635+
env[entry[:eq]] = entry[eq+1:]
636+
}
637+
return env, nil
638+
}
639+
640+
func readDotEnvFile(path string) (map[string]string, error) {
641+
f, err := os.Open(path)
642+
if err != nil {
643+
return nil, err
644+
}
645+
defer f.Close()
646+
647+
out := make(map[string]string)
648+
scanner := bufio.NewScanner(f)
649+
for scanner.Scan() {
650+
line := strings.TrimSpace(scanner.Text())
651+
if line == "" || strings.HasPrefix(line, "#") {
652+
continue
653+
}
654+
line = strings.TrimPrefix(line, "export ")
655+
eq := strings.IndexByte(line, '=')
656+
if eq < 0 {
657+
continue
658+
}
659+
key := strings.TrimSpace(line[:eq])
660+
value := strings.TrimSpace(line[eq+1:])
661+
value = strings.Trim(value, `"'`)
662+
out[key] = value
663+
}
664+
if err := scanner.Err(); err != nil {
665+
return nil, err
666+
}
667+
return out, nil
668+
}
669+
528670
func mergeResolvedSkills(imageSkills, podSkills []driver.ResolvedSkill) []driver.ResolvedSkill {
529671
merged := make([]driver.ResolvedSkill, 0, len(imageSkills)+len(podSkills))
530672
byName := make(map[string]int, len(imageSkills))
@@ -1304,23 +1446,25 @@ func ensureInfraImages(cllamaEnabled bool, proxies []pod.CllamaProxyConfig, dash
13041446
}
13051447

13061448
// ensureImage builds a Docker image if it doesn't exist locally.
1307-
// It tries: local build from repo source, then git URL build, then errors.
1449+
// It tries: local image, docker pull, local build from repo source, git URL build,
1450+
// then errors with explicit manual-build guidance.
13081451
func ensureImage(imageRef, name, dockerfilePath, contextDir string) error {
1309-
if build.ImageExistsLocally(imageRef) {
1452+
if imageExistsLocally(imageRef) {
13101453
return nil
13111454
}
13121455

13131456
fmt.Printf("[claw] building %s image (first time only)\n", name)
13141457

1315-
repoRoot, found := findRepoRoot()
1458+
if err := runInfraDockerCommand("pull", imageRef); err == nil {
1459+
return nil
1460+
}
1461+
1462+
repoRoot, found := findClawdapusRepoRoot()
13161463
if found {
13171464
df := filepath.Join(repoRoot, dockerfilePath)
13181465
ctx := filepath.Join(repoRoot, contextDir)
13191466
if _, err := os.Stat(df); err == nil {
1320-
cmd := exec.Command("docker", "build", "-t", imageRef, "-f", df, ctx)
1321-
cmd.Stdout = os.Stdout
1322-
cmd.Stderr = os.Stderr
1323-
if err := cmd.Run(); err != nil {
1467+
if err := runInfraDockerCommand("build", "-t", imageRef, "-f", df, ctx); err != nil {
13241468
return fmt.Errorf("build %s image from local source: %w", name, err)
13251469
}
13261470
return nil
@@ -1329,15 +1473,19 @@ func ensureImage(imageRef, name, dockerfilePath, contextDir string) error {
13291473

13301474
// Fallback: build from git URL.
13311475
gitURL := fmt.Sprintf("https://github.com/mostlydev/clawdapus.git#master:%s", contextDir)
1332-
cmd := exec.Command("docker", "build", "-t", imageRef, gitURL)
1333-
cmd.Stdout = os.Stdout
1334-
cmd.Stderr = os.Stderr
1335-
if err := cmd.Run(); err != nil {
1476+
if err := runInfraDockerCommand("build", "-t", imageRef, gitURL); err != nil {
13361477
return fmt.Errorf("could not build %s image; run 'docker build -t %s -f %s %s' from the repo root", name, imageRef, dockerfilePath, contextDir)
13371478
}
13381479
return nil
13391480
}
13401481

1482+
func runInfraDockerCommandDefault(args ...string) error {
1483+
cmd := exec.Command("docker", args...)
1484+
cmd.Stdout = os.Stdout
1485+
cmd.Stderr = os.Stderr
1486+
return cmd.Run()
1487+
}
1488+
13411489
// findRepoRoot walks up from cwd looking for go.mod with the clawdapus module.
13421490
func findRepoRoot() (string, bool) {
13431491
dir, err := os.Getwd()

0 commit comments

Comments
 (0)