Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,288 changes: 1,288 additions & 0 deletions docs/verification/pr315-integration-test.log

Large diffs are not rendered by default.

23 changes: 23 additions & 0 deletions docs/verification/pr315-review.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# PR #315 review verification

Known and reasonably foreseeable failure paths, recorded before repair tests and implementation:

1. The prune boundary accepts a timezone offset but compares its original text with UTC `checkedAt` values; a row newer than the actual instant can be counted or deleted. Existing rows also mix seconds and millisecond precision, so a canonical boundary alone cannot fix lexical ordering. Preview and deletion must compare actual instants.
2. Calendar-invalid dates (for example February 30) and timezone-less local datetimes are syntactically accepted but lack the intended absolute boundary. Reject them; date-only input means UTC midnight.
3. A concurrent snapshot insert between the preview count and the deletion can cross the 50% threshold without `force`. Recount and delete under one immediate SQLite transaction. A fully deterministic process race is unsuitable as a normal regression test; code inspection and real SQLite behavior checks cover the transaction boundary.
4. Keep existing contracts: unknown projects error; dry run never deletes; over-50% requires `force`; exactly 50% is allowed; other projects remain untouched; CLI/MCP route through the guarded service.

The CLI scenario uses a real SQLite database and process. Commands, environment, inputs, results, and retained artifacts follow after execution.

## Reproduction and result

- Environment: macOS arm64, Node v26.7.0; dependencies installed with `npm ci --ignore-scripts --no-audit --no-fund`. No live service credentials are required.
- Input: a temporary SQLite project with snapshots at `2026-08-31T23:00:00.000Z` and `2026-09-01T00:00:00Z`; dry-run boundaries `2026-09-01T00:00:00+08:00` and `2026-09-01T00:00:00.500Z`; invalid `2026-02-30T00:00:00Z` and timezone-less `2026-09-01T00:00:00`.
- Before repair: `npm run build -w @appilot-labs/appilot-headless && npm run build -w @appilot-labs/appilot-headless-cli && npx tsx packages/headless-cli/tests/cli.test.ts` failed because the offset boundary matched one row instead of zero. `npx tsx packages/headless/tests/snapshots-prune-guard.test.ts` failed because February 30 was accepted.
- After repair: both commands passed, including unknown project, dry run, exactly 50%, over 50% with/without force, mixed timestamp precision, and no deletion on rejected inputs. `npm run typecheck` and `git diff --check` passed.
- The CLI output above is the end-to-end run; this report preserves the command, environment, inputs, and results. The race is closed by using the existing `BEGIN IMMEDIATE` transaction wrapper for the count, threshold decision, and delete. A deterministic cross-process insertion at precisely the former gap would require a test-only hook and is not claimed as observed.

## Integration with current master

- The repair and PR #315 merged cleanly with the current master that includes #311–#314.
- `npm run typecheck` passed. `npm test > docs/verification/pr315-integration-test.log 2>&1` exited 0; the committed log captures scheduler process, CLI, MCP, and repository test results from the combined tree.
13 changes: 10 additions & 3 deletions packages/headless-cli/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ function usage(): never {
' appilot-headless projects release-cache <name> # 发布页缓存(M4)',
' appilot-headless snapshots latest <project> [--product <id>]',
' appilot-headless snapshots history <project> [--product <id>] [--keyword <kw>] [--limit <n>]',
' appilot-headless snapshots prune <project> [--before <iso>] # 默认清 90 天前',
' appilot-headless snapshots prune <project> [--before <iso>] [--dry-run] [--force] # 默认清 90 天前;删除量超存量 50% 需 --force',
' appilot-headless tasks list [--source dsh|electron|cli]',
' appilot-headless tasks rank-progress [project] [--product <id>]',
' appilot-headless lease status # 当前租约主(多壳调度验证)',
Expand Down Expand Up @@ -233,8 +233,15 @@ export async function main(argv: string[]): Promise<void> {
if (!project) return usage();
// 默认清理 90 天前的快照(与 core RANK_SNAPSHOT_WINDOW_MS 一致)
const before = flags.get('before') || new Date(Date.now() - 90 * 24 * 60 * 60 * 1000).toISOString();
const removed = svc.snapshots.prune(project, before);
process.stdout.write(JSON.stringify({ project, beforeIso: before, removed }, null, 2) + '\n');
// 审计 H7:dry-run 预览 / force 显式确认(服务层同时做 ISO 校验、
// 未知项目报错与 50% 删除量拦截)。
const res = svc.snapshots.prune(project, before, {
dryRun: flags.has('dry-run'),
force: flags.has('force'),
});
process.stdout.write(
JSON.stringify({ project, beforeIso: before, dryRun: flags.has('dry-run'), ...res }, null, 2) + '\n',
);
return;
}
return usage();
Expand Down
29 changes: 27 additions & 2 deletions packages/headless-cli/tests/cli.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -115,19 +115,44 @@ async function main(): Promise<void> {
assert.equal(JSON.parse(histKw.stdout).count, 1);
console.log('✓ snapshots history(降序/productId/limit)');

// snapshots prune:清 90 天前(seed 里 2026-08 距今不足 90 天 → 保留?用显式 before 清全部)
// snapshots prune:dry-run 预览(审计 H7)+ --before 窗口(1/2 = 50% 恰好无需 force)
const seedStore = openStore(dbPath);
seedStore.projects.save({ id: 'old-proj-id', name: 'old-proj', path: '/old-proj', githubUrl: null, platform: null, languages: [], lastResolvedAt: new Date().toISOString(), artworkUrl: null, updatedAt: new Date().toISOString() });
seedStore.snapshots.add([
{ projectName: 'old-proj', productId: null, keyword: 'k', language: 'en', storefront: 'us', rank: 9, totalResults: 10, checkedAt: '2020-01-01T00:00:00Z' },
{ projectName: 'old-proj', productId: null, keyword: 'k', language: 'en', storefront: 'us', rank: 8, totalResults: 10, checkedAt: '2026-08-15T00:00:00Z' },
]);
seedStore.close();
const pruneDry = await run(['snapshots', 'prune', 'old-proj', '--before', '2025-01-01T00:00:00Z', '--dry-run'], dbPath);
const pruneDryJson = JSON.parse(pruneDry.stdout);
assert.equal(pruneDryJson.dryRun, true, 'dry-run 标志回显');
assert.equal(pruneDryJson.matched, 1, 'dry-run 预览命中 1 条');
assert.equal(pruneDryJson.removed, 0, 'dry-run 不删除');
const prune = await run(['snapshots', 'prune', 'old-proj', '--before', '2025-01-01T00:00:00Z'], dbPath);
const pruneJson = JSON.parse(prune.stdout);
assert.equal(pruneJson.removed, 1, '只清早于 before 的行');
assert.equal(pruneJson.beforeIso, '2025-01-01T00:00:00Z');
console.log('✓ snapshots prune(--before 窗口)');
console.log('✓ snapshots prune(--dry-run 预览 + --before 窗口)');

const offsetStore = openStore(dbPath);
offsetStore.projects.save({ id: 'offset-proj-id', name: 'offset-proj', path: '/offset-proj', githubUrl: null, platform: null, languages: [], lastResolvedAt: new Date().toISOString(), artworkUrl: null, updatedAt: new Date().toISOString() });
offsetStore.snapshots.add([
{ projectName: 'offset-proj', productId: null, keyword: 'k', language: 'en', storefront: 'us', rank: 9, totalResults: 10, checkedAt: '2026-08-31T23:00:00.000Z' },
{ projectName: 'offset-proj', productId: null, keyword: 'k', language: 'en', storefront: 'us', rank: 8, totalResults: 10, checkedAt: '2026-09-01T00:00:00Z' },
]);
offsetStore.close();
const offsetPreview = await run(['snapshots', 'prune', 'offset-proj', '--before', '2026-09-01T00:00:00+08:00', '--dry-run'], dbPath);
assert.equal(offsetPreview.code, 0, offsetPreview.stderr);
assert.equal(JSON.parse(offsetPreview.stdout).matched, 0, 'UTC+08 边界对应前一天 16:00Z,不应命中 23:00Z');
const mixedPrecision = await run(['snapshots', 'prune', 'offset-proj', '--before', '2026-09-01T00:00:00.500Z', '--dry-run'], dbPath);
assert.equal(JSON.parse(mixedPrecision.stdout).matched, 2, '秒精度旧快照应早于同秒的 500 毫秒边界');
for (const invalid of ['2026-02-30T00:00:00Z', '2026-09-01T00:00:00']) {
const rejected = await run(['snapshots', 'prune', 'offset-proj', '--before', invalid, '--force'], dbPath);
assert.notEqual(rejected.code, 0, `无效或无时区边界 ${invalid} 应拒绝`);
}
const offsetHistory = await run(['snapshots', 'history', 'offset-proj'], dbPath);
assert.equal(JSON.parse(offsetHistory.stdout).count, 2, '被拒绝的清理未删除任何快照');
console.log('✓ snapshots prune(时区归一、无效日期拒绝)');

// tasks list --source:seed 不同来源行 → 过滤正确
const tStore = openStore(dbPath);
Expand Down
38 changes: 35 additions & 3 deletions packages/headless/src/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,23 @@
import type { AppilotStore } from './store.js';
import type { ProjectRow, RankSnapshotRow, TaskRow, ProjectMetaRow, ProductRecordRow, ReleaseCacheRow } from './schema.js';

/** A prune boundary is an instant, never a locale-dependent or lexical string. */
function normalizePruneBefore(value: string): string {
const input = String(value).trim();
const match = /^(\d{4})-(\d{2})-(\d{2})(?:T(\d{2}):(\d{2})(?::(\d{2})(?:\.(\d{1,3}))?)?(Z|[+-]\d{2}:?\d{2}))?$/.exec(input);
const invalid = () => new Error(`before 必须是有效的 ISO 8601 时间(如 2026-01-01T00:00:00Z),收到:"${input.slice(0, 40)}"`);
if (!match) throw invalid();
const [, year, month, day, hour, minute, second] = match;
const y = Number(year);
const m = Number(month);
const d = Number(day);
if (y < 1 || m < 1 || m > 12 || d < 1 || d > new Date(Date.UTC(y, m, 0)).getUTCDate()) throw invalid();
if (hour && (Number(hour) > 23 || Number(minute) > 59 || Number(second || 0) > 59)) throw invalid();
const instant = new Date(input);
if (!Number.isFinite(instant.getTime())) throw invalid();
return instant.toISOString();
}

export interface HeadlessService {
projects: {
/** 注册/更新(stamp updatedAt)。 */
Expand All @@ -25,8 +42,16 @@ export interface HeadlessService {
projectName: string,
opts?: { productId?: string | null; keyword?: string; limit?: number },
): RankSnapshotRow[];
/** 清理某项目早于 beforeIso 的旧快照;返回删除行数。 */
prune(projectName: string, beforeIso: string): number;
/**
* 清理某项目早于 beforeIso 的旧快照;返回 { matched, removed, total }。
* 破坏性命令防护:before 必须是有确定时区的有效 ISO 8601 时间;未知项目报错;
* dryRun 只预览;单次删除超过存量 50% 需显式 force:true。
*/
prune(
projectName: string,
beforeIso: string,
opts?: { dryRun?: boolean; force?: boolean },
): { matched: number; removed: number; total: number };
};
tasks: {
list(): TaskRow[];
Expand Down Expand Up @@ -88,7 +113,14 @@ export function createHeadlessService(store: AppilotStore): HeadlessService {
latest: (projectName, productId) =>
store.snapshots.latestByKey(projectName, productId ?? undefined),
recent: (projectName, opts) => store.snapshots.recent(projectName, opts),
prune: (projectName, beforeIso) => store.snapshots.pruneOlderThan(projectName, beforeIso),
prune: (projectName, beforeIso, opts = {}) => {
const cutoff = normalizePruneBefore(beforeIso);
const result = store.snapshots.pruneChecked(projectName, cutoff, opts);
if (!result) {
throw new Error(`项目不存在:${projectName}`);
}
return result;
},
},
tasks: {
list: () => store.tasks.all(),
Expand Down
49 changes: 48 additions & 1 deletion packages/headless/src/store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,20 @@ export interface AppilotStore {
): RankSnapshotRow[];
/** 清理某项目早于 checkedAt 的旧快照(保留最近 N 天)。 */
pruneOlderThan(projectName: string, beforeIso: string): number;
/**
* prune 影响面预览(不删除):项目快照总数 + 将被删除的条数。
* 项目不存在返回 null(审计 H7:未知项目必须显式报错,不能静默 0)。
*/
prunePreview(
projectName: string,
beforeIso: string,
): { total: number; matched: number } | null;
/** Atomically count, guard, and delete under BEGIN IMMEDIATE. */
pruneChecked(
projectName: string,
beforeIso: string,
opts?: { dryRun?: boolean; force?: boolean },
): { total: number; matched: number; removed: number } | null;
/** 全库清理早于 checkedAt 的旧快照(数据管理/保留策略用)。返回删除行数。 */
pruneAllOlderThan(beforeIso: string): number;
/**
Expand Down Expand Up @@ -237,6 +251,18 @@ export function openStore(dbPath: string): AppilotStore {
return identity;
}

function snapshotPrunePreview(projectName: string, beforeIso: string): { total: number; matched: number; projectId: string } | null {
const identity = projectIdentity(projectName);
if (!identity) return null;
const total = Number(
(db.prepare('SELECT COUNT(*) AS n FROM rank_snapshots WHERE projectId = ?').get(identity.id) as any).n,
);
const matched = Number(
(db.prepare('SELECT COUNT(*) AS n FROM rank_snapshots WHERE projectId = ? AND julianday(checkedAt) < julianday(?)').get(identity.id, beforeIso) as any).n,
);
return { total, matched, projectId: identity.id };
}

return {
path: dbPath,

Expand Down Expand Up @@ -519,10 +545,31 @@ export function openStore(dbPath: string): AppilotStore {
const identity = projectIdentity(projectName);
if (!identity) return 0;
const res = db
.prepare('DELETE FROM rank_snapshots WHERE projectId = ? AND checkedAt < ?')
.prepare('DELETE FROM rank_snapshots WHERE projectId = ? AND julianday(checkedAt) < julianday(?)')
.run(identity.id, beforeIso);
return Number(res.changes);
},
prunePreview(projectName, beforeIso) {
const preview = snapshotPrunePreview(projectName, beforeIso);
return preview ? { total: preview.total, matched: preview.matched } : null;
},
pruneChecked(projectName, beforeIso, opts = {}) {
return tx(() => {
const preview = snapshotPrunePreview(projectName, beforeIso);
if (!preview) return null;
if (opts.dryRun) return { total: preview.total, matched: preview.matched, removed: 0 };
if (preview.total > 0 && preview.matched * 2 > preview.total && !opts.force) {
throw new Error(
`将删除 ${preview.matched}/${preview.total} 条快照(超过存量的 50%)。` +
'如确认无误请传 force:true;建议先用 dryRun 预览影响面。',
);
}
const removed = Number(db.prepare(
'DELETE FROM rank_snapshots WHERE projectId = ? AND julianday(checkedAt) < julianday(?)',
).run(preview.projectId, beforeIso).changes);
return { total: preview.total, matched: preview.matched, removed };
});
},
/** 全库清理早于 checkedAt 的旧快照(数据管理/保留策略用)。返回删除行数。 */
pruneAllOlderThan(beforeIso) {
const res = db
Expand Down
90 changes: 90 additions & 0 deletions packages/headless/tests/snapshots-prune-guard.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
/**
* snapshots.prune 破坏性命令防护测试(审计 2026-09-26 H7):
* checkedAt 是 TEXT、prune 按字典序比较——旧实现传 "z" 等非 ISO 字符串会
* 删光项目全部快照并静默返回成功;未知项目静默 removed=0。
* 修复后:ISO 强校验、未知项目报错、dryRun 预览、>50% 删除需 force。
*/
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import assert from 'node:assert';
import { openStore, createHeadlessService } from '../src/index.js';

function mk(projectName: string, keyword: string, rank: number | null, checkedAt: string) {
return { projectName, productId: null, keyword, language: 'en', storefront: 'us', rank, totalResults: 100, checkedAt };
}

async function main(): Promise<void> {
const dbPath = join(mkdtempSync(join(tmpdir(), 'prune-guard-')), 'appilot.db');
const store = openStore(dbPath);
store.projects.save({ id: 'proj-id', name: 'proj', path: '/proj', githubUrl: null, platform: null, languages: [], lastResolvedAt: '2026-08-01T00:00:00Z', artworkUrl: null, updatedAt: '2026-08-01T00:00:00Z' });
const svc = createHeadlessService(store);
// 4 条旧快照(2026-08)+ 1 条新快照(2026-09)→ before=2026-09-01 命中 4/5(>50%)
svc.snapshots.record([
mk('proj', 'app', 5, '2026-08-01T00:00:00Z'),
mk('proj', 'app', 4, '2026-08-02T00:00:00Z'),
mk('proj', 'app', 3, '2026-08-03T00:00:00Z'),
mk('proj', 'other', 9, '2026-08-04T00:00:00Z'),
mk('proj', 'app', 1, '2026-09-01T00:00:00Z'),
]);

// 1. 非 ISO before 一律拒绝(旧实现 "z" 会删光全部)
for (const bad of ['z', '9999', '', 'not-a-date', '2026-02-30T00:00:00Z', '2026-09-01T00:00:00']) {
assert.throws(
() => svc.snapshots.prune('proj', bad),
/ISO 8601/,
`非法 before "${bad}" 应被拒绝`,
);
}
assert.equal(svc.snapshots.recent('proj').length, 5, '拒绝路径不删数据');
console.log('✓ 非 ISO before 强校验(字典序陷阱封堵)');

// 2. 未知项目 → 明确报错(旧实现静默 removed=0)
assert.throws(
() => svc.snapshots.prune('no-such-project', '2026-01-01T00:00:00Z'),
/项目不存在/,
'未知项目应报错而非静默 0',
);
console.log('✓ 未知项目显式报错');

// 3. dryRun:只预览不删除
const dry = svc.snapshots.prune('proj', '2026-09-01T00:00:00Z', { dryRun: true });
assert.deepEqual(dry, { matched: 4, removed: 0, total: 5 }, 'dryRun 返回影响面且 removed=0');
assert.equal(svc.snapshots.recent('proj').length, 5, 'dryRun 不删数据');
console.log('✓ dryRun 预览影响面');

// 4. >50% 删除量:无 force 拒绝;force 放行
assert.throws(
() => svc.snapshots.prune('proj', '2026-09-01T00:00:00Z'),
/50%.*force/s,
'删除 4/5(>50%)无 force 应被拦截',
);
const forced = svc.snapshots.prune('proj', '2026-09-01T00:00:00Z', { force: true });
assert.deepEqual(forced, { matched: 4, removed: 4, total: 5 }, 'force 放行并删除 4 条');
assert.equal(svc.snapshots.recent('proj').length, 1, '只剩新快照');
console.log('✓ >50% 删除量需 force');

// 5. 小批量删除(≤50%):无 force 直接执行
svc.snapshots.record([mk('proj', 'app', 2, '2026-08-05T00:00:00Z')]);
const small = svc.snapshots.prune('proj', '2026-08-06T00:00:00Z');
assert.equal(small.removed, 1, '小批量删除直接执行');
assert.equal(svc.snapshots.recent('proj').length, 1, '删除后仅剩 1 条');
console.log('✓ ≤50% 小批量正常执行');

// 6. ISO 变体兼容:日期 only / 带毫秒 / 带时区偏移
for (const ok of ['2026-12-31', '2026-12-31T00:00:00.123Z', '2026-12-31T00:00:00+08:00']) {
assert.doesNotThrow(
() => svc.snapshots.prune('proj', ok, { dryRun: true }),
`ISO 变体应放行:${ok}`,
);
}
console.log('✓ ISO 8601 变体兼容');

store.close();
console.log('snapshots.prune 防护测试全部通过 ✓');
}

main().catch((err) => {
console.error('prune 防护测试失败:', err);
process.exit(1);
});
Loading
Loading