Skip to content

Commit 428348c

Browse files
authored
Fix repo-memory subdirectory support and metrics collector paths (#7690)
1 parent 802a21d commit 428348c

6 files changed

Lines changed: 255 additions & 66 deletions

File tree

.github/workflows/metrics-collector.lock.yml

Lines changed: 8 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/metrics-collector.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ tools:
1515
toolsets: [default]
1616
repo-memory:
1717
branch-name: memory/meta-orchestrators
18-
file-glob: "metrics/**/*"
18+
file-glob: "metrics/**"
1919
timeout-minutes: 15
2020
---
2121

@@ -38,7 +38,7 @@ As an infrastructure agent, you collect and persist performance data that enable
3838
- **Repository**: ${{ github.repository }}
3939
- **Collection Date**: $(date +%Y-%m-%d)
4040
- **Collection Time**: $(date +%H:%M:%S) UTC
41-
- **Storage Path**: `/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/`
41+
- **Storage Path**: `/tmp/gh-aw/repo-memory-default/memory/default/metrics/`
4242

4343
## Metrics Collection Process
4444

@@ -131,15 +131,15 @@ Create a JSON object following this schema:
131131
### 3. Store Metrics in Repo Memory
132132

133133
**Daily Storage**:
134-
- Write metrics to: `/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/daily/YYYY-MM-DD.json`
134+
- Write metrics to: `/tmp/gh-aw/repo-memory-default/memory/default/metrics/daily/YYYY-MM-DD.json`
135135
- Use today's date for the filename (e.g., `2024-12-24.json`)
136136

137137
**Latest Snapshot**:
138-
- Copy current metrics to: `/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/latest.json`
138+
- Copy current metrics to: `/tmp/gh-aw/repo-memory-default/memory/default/metrics/latest.json`
139139
- This provides quick access to most recent data without date calculations
140140

141141
**Create Directory Structure**:
142-
- Ensure the directory exists: `mkdir -p /tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/daily/`
142+
- Ensure the directory exists: `mkdir -p /tmp/gh-aw/repo-memory-default/memory/default/metrics/daily/`
143143

144144
### 4. Cleanup Old Data
145145

@@ -150,7 +150,7 @@ Create a JSON object following this schema:
150150

151151
**Cleanup Command**:
152152
```bash
153-
find /tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/daily/ -name "*.json" -mtime +30 -delete
153+
find /tmp/gh-aw/repo-memory-default/memory/default/metrics/daily/ -name "*.json" -mtime +30 -delete
154154
```
155155

156156
### 5. Calculate Ecosystem Aggregates
@@ -222,7 +222,7 @@ At the end of collection:
222222
- Active workflows: 85
223223
- Total safe outputs: 45
224224
- Overall success rate: 89.2%
225-
- Storage: /tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/daily/2024-12-24.json
225+
- Storage: /tmp/gh-aw/repo-memory-default/memory/default/metrics/daily/2024-12-24.json
226226
227227
⏱️ Collection took: 45 seconds
228228
```

.github/workflows/workflow-health-manager.lock.yml

Lines changed: 5 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/workflow-health-manager.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ As a meta-orchestrator for workflow health, you oversee the operational health o
6969
- Flag workflows with compilation warnings
7070

7171
**Monitor workflow execution:**
72-
- Load shared metrics from: `/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/latest.json`
72+
- Load shared metrics from: `/tmp/gh-aw/repo-memory-default/memory/default/metrics/latest.json`
7373
- Use workflow_runs data for each workflow:
7474
- Total runs, successful runs, failed runs
7575
- Success rate (already calculated)
@@ -129,7 +129,7 @@ As a meta-orchestrator for workflow health, you oversee the operational health o
129129

130130
**Quality metrics:**
131131
- Use historical metrics for trend analysis:
132-
- Load daily metrics from: `/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/daily/`
132+
- Load daily metrics from: `/tmp/gh-aw/repo-memory-default/memory/default/metrics/daily/`
133133
- Calculate 7-day and 30-day success rate trends
134134
- Identify workflows with declining quality
135135
- Calculate workflow reliability score (0-100):
@@ -166,20 +166,20 @@ Execute these phases each run:
166166

167167
## Shared Memory Integration
168168

169-
**Access shared repo memory at `/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/`**
169+
**Access shared repo memory at `/tmp/gh-aw/repo-memory-default/memory/default/`**
170170

171171
This workflow shares memory with other meta-orchestrators (Campaign Manager and Agent Performance Analyzer) to coordinate insights and avoid duplicate work.
172172

173173
**Shared Metrics Infrastructure:**
174174

175175
The Metrics Collector workflow runs daily and stores performance metrics in a structured JSON format:
176176

177-
1. **Latest Metrics**: `/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/latest.json`
177+
1. **Latest Metrics**: `/tmp/gh-aw/repo-memory-default/memory/default/metrics/latest.json`
178178
- Most recent workflow run statistics
179179
- Success rates, failure counts for all workflows
180180
- Use to identify failing workflows without querying GitHub API repeatedly
181181

182-
2. **Historical Metrics**: `/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/metrics/daily/YYYY-MM-DD.json`
182+
2. **Historical Metrics**: `/tmp/gh-aw/repo-memory-default/memory/default/metrics/daily/YYYY-MM-DD.json`
183183
- Daily metrics for the last 30 days
184184
- Track workflow health trends over time
185185
- Identify recent regressions by comparing current vs. historical success rates

actions/setup/js/push_repo_memory.cjs

Lines changed: 71 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ const { execSync } = require("child_process");
1414
* BRANCH_NAME: Branch name to push to
1515
* MAX_FILE_SIZE: Maximum file size in bytes
1616
* MAX_FILE_COUNT: Maximum number of files per commit
17-
* FILE_GLOB_FILTER: Optional space-separated list of file patterns (e.g., "*.md *.txt")
17+
* FILE_GLOB_FILTER: Optional space-separated list of file patterns (e.g., "*.md metrics/ ** /*")
18+
* Supports * (matches any chars except /) and ** (matches any chars including /)
1819
* GH_TOKEN: GitHub token for authentication
1920
* GITHUB_RUN_ID: Workflow run ID for commit messages
2021
*/
@@ -86,47 +87,70 @@ async function main() {
8687
fs.mkdirSync(destMemoryPath, { recursive: true });
8788
core.info(`Destination directory: ${destMemoryPath}`);
8889

89-
// Read files from artifact directory and validate before copying
90+
// Recursively scan and collect files from artifact directory
9091
let filesToCopy = [];
91-
try {
92-
const files = fs.readdirSync(sourceMemoryPath, { withFileTypes: true });
93-
94-
for (const file of files) {
95-
if (!file.isFile()) {
96-
continue; // Skip directories
97-
}
98-
99-
const fileName = file.name;
100-
const sourceFilePath = path.join(sourceMemoryPath, fileName);
101-
const stats = fs.statSync(sourceFilePath);
102-
103-
// Validate file name patterns if filter is set
104-
if (fileGlobFilter) {
105-
const patterns = fileGlobFilter.split(/\s+/).map(pattern => {
106-
// Escape backslashes first to prevent escaping issues, then escape dots and convert asterisks
107-
const regexPattern = pattern.replace(/\\/g, "\\\\").replace(/\./g, "\\.").replace(/\*/g, "[^/]*");
108-
return new RegExp(`^${regexPattern}$`);
109-
});
110-
111-
if (!patterns.some(pattern => pattern.test(fileName))) {
112-
core.error(`File does not match allowed patterns: ${fileName}`);
113-
core.error(`Allowed patterns: ${fileGlobFilter}`);
114-
core.setFailed("File pattern validation failed");
115-
return;
92+
93+
/**
94+
* Recursively scan directory and collect files
95+
* @param {string} dirPath - Directory to scan
96+
* @param {string} relativePath - Relative path from sourceMemoryPath (for nested files)
97+
*/
98+
function scanDirectory(dirPath, relativePath = "") {
99+
const entries = fs.readdirSync(dirPath, { withFileTypes: true });
100+
101+
for (const entry of entries) {
102+
const fullPath = path.join(dirPath, entry.name);
103+
const relativeFilePath = relativePath ? path.join(relativePath, entry.name) : entry.name;
104+
105+
if (entry.isDirectory()) {
106+
// Recursively scan subdirectory
107+
scanDirectory(fullPath, relativeFilePath);
108+
} else if (entry.isFile()) {
109+
const stats = fs.statSync(fullPath);
110+
111+
// Validate file name patterns if filter is set
112+
if (fileGlobFilter) {
113+
const patterns = fileGlobFilter.split(/\s+/).map(pattern => {
114+
// Convert glob pattern to regex that supports directory wildcards
115+
// ** matches any path segment (including /)
116+
// * matches any characters except /
117+
let regexPattern = pattern
118+
.replace(/\\/g, "\\\\") // Escape backslashes
119+
.replace(/\./g, "\\.") // Escape dots
120+
.replace(/\*\*/g, "<!DOUBLESTAR>") // Temporarily replace **
121+
.replace(/\*/g, "[^/]*") // Single * matches non-slash chars
122+
.replace(/<!DOUBLESTAR>/g, ".*"); // ** matches everything including /
123+
return new RegExp(`^${regexPattern}$`);
124+
});
125+
126+
if (!patterns.some(pattern => pattern.test(relativeFilePath))) {
127+
core.error(`File does not match allowed patterns: ${relativeFilePath}`);
128+
core.error(`Allowed patterns: ${fileGlobFilter}`);
129+
core.setFailed("File pattern validation failed");
130+
throw new Error("File pattern validation failed");
131+
}
116132
}
133+
134+
// Validate file size
135+
if (stats.size > maxFileSize) {
136+
core.error(`File exceeds size limit: ${relativeFilePath} (${stats.size} bytes > ${maxFileSize} bytes)`);
137+
core.setFailed("File size validation failed");
138+
throw new Error("File size validation failed");
139+
}
140+
141+
filesToCopy.push({
142+
relativePath: relativeFilePath,
143+
source: fullPath,
144+
size: stats.size
145+
});
117146
}
118-
119-
// Validate file size
120-
if (stats.size > maxFileSize) {
121-
core.error(`File exceeds size limit: ${fileName} (${stats.size} bytes > ${maxFileSize} bytes)`);
122-
core.setFailed("File size validation failed");
123-
return;
124-
}
125-
126-
filesToCopy.push({ name: fileName, source: sourceFilePath, size: stats.size });
127147
}
148+
}
149+
150+
try {
151+
scanDirectory(sourceMemoryPath);
128152
} catch (error) {
129-
core.setFailed(`Failed to read artifact directory: ${error instanceof Error ? error.message : String(error)}`);
153+
core.setFailed(`Failed to scan artifact directory: ${error instanceof Error ? error.message : String(error)}`);
130154
return;
131155
}
132156

@@ -143,14 +167,20 @@ async function main() {
143167

144168
core.info(`Copying ${filesToCopy.length} validated file(s)...`);
145169

146-
// Copy files to destination
170+
// Copy files to destination (preserving directory structure)
147171
for (const file of filesToCopy) {
148-
const destFilePath = path.join(destMemoryPath, file.name);
172+
const destFilePath = path.join(destMemoryPath, file.relativePath);
173+
const destDir = path.dirname(destFilePath);
174+
149175
try {
176+
// Ensure destination directory exists
177+
fs.mkdirSync(destDir, { recursive: true });
178+
179+
// Copy file
150180
fs.copyFileSync(file.source, destFilePath);
151-
core.info(`Copied: ${file.name} (${file.size} bytes)`);
181+
core.info(`Copied: ${file.relativePath} (${file.size} bytes)`);
152182
} catch (error) {
153-
core.setFailed(`Failed to copy file ${file.name}: ${error instanceof Error ? error.message : String(error)}`);
183+
core.setFailed(`Failed to copy file ${file.relativePath}: ${error instanceof Error ? error.message : String(error)}`);
154184
return;
155185
}
156186
}

0 commit comments

Comments
 (0)