Skip to content

Commit 16eb113

Browse files
feat(media-use): usage visibility — shared telemetry identity, miss log, resolve --stats (#2113)
* feat(media-use): usage visibility — shared telemetry identity, miss log, resolve --stats - U6: join the CLI/studio telemetry identity — read the shared install id from ~/.hyperframes/config.json (seed if absent) instead of a media-use-only ~/.media/anon-id, and $identify to the HeyGen account (email/username) once per run on sign-in. One PostHog person across surfaces; pseudonymous before sign-in, account-linked after. Event properties stay coarse (no intent/paths). - U1: one-time first-run disclosure to stderr + Privacy section in SKILL.md; honors DO_NOT_TRACK / HYPERFRAMES_NO_TELEMETRY. - U2: persist resolve misses to ~/.media/misses.jsonl (local → intent kept; the media_use_resolve_miss telemetry event stays intent-free). - U3: `resolve --stats` (+ --days) — local usage report over .media/ + ~/.media (volume by type, source/provider/via split, hit-rate, top missed intents, global-cache size/reuse); human + --json. - U4: reproducible PostHog dashboard definition (references/telemetry-dashboard.md). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(media-use): address #2113 review — shared notice state, legacy id migration, stats robustness - Notice-shown state now lives in the shared ~/.hyperframes/config.json (config.telemetryNoticeShown, the CLI's own field) instead of a media-use-only ~/.media marker — so shared-identity users see the first-run notice once per person, not once per tool. - Migrate a pre-existing ~/.media/anon-id into the shared config on upgrade, so media-use-only users keep their PostHog persona instead of resetting. - buildStats: --days only windows on a positive finite value (negative/NaN → all time, not an empty report); dropped the top-level catch that masked a real error as an all-zero "no usage" report (sub-reads are individually guarded). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent cdb8d73 commit 16eb113

11 files changed

Lines changed: 944 additions & 39 deletions

File tree

skills-manifest.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,8 @@
4646
"files": 10
4747
},
4848
"media-use": {
49-
"hash": "14ea99a7f4d750cd",
50-
"files": 116
49+
"hash": "6c4aa8649e1eaf99",
50+
"files": 121
5151
},
5252
"motion-graphics": {
5353
"hash": "96ed2f7d8051b009",

skills/media-use/SKILL.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,8 @@ node <SKILL_DIR>/scripts/resolve.mjs --type lut --intent "teal orange blockbuste
135135
| `--provider` | Force one generator (e.g. `codex`, `mflux`, `kokoro`, `heygen`) |
136136
| `--adopt` | Bulk-import existing assets/ into manifest |
137137
| `--doctor` | Check local CLI dependencies; no manifest changes |
138+
| `--stats` | Print local usage stats from `.media/` and `~/.media`; no manifest changes |
139+
| `--days N` | Limit `--stats` to timestamped records/misses from the last N days |
138140
| `--json` | Output JSON instead of one-line result |
139141

140142
## Reuse before you resolve
@@ -316,11 +318,24 @@ Assets are cached automatically on resolve. Every resolved/ingested asset is aut
316318

317319
For a _semantically_ similar (not identical) need in another project, the exact-match floor won't fire — use [Reuse before you resolve](#reuse-before-you-resolve): `--candidates` lists the global assets, and `--reuse <sha>` imports the one you pick. This is how a track resolved in one project gets reused in the next when the wording differs.
318320

321+
## Usage stats
322+
323+
Use `resolve --stats` for a local, shareable report over the current project's `.media/` manifest, the global `~/.media/` cache, and local resolve misses. Human output is compact; add `--json` for a single machine-readable object, and `--days N` to window timestamped records.
324+
325+
```bash
326+
node <SKILL_DIR>/scripts/resolve.mjs --stats --project . --days 7
327+
# media-use stats
328+
# total resolves: 12
329+
# misses: 2
330+
# hit rate: 86%
331+
```
332+
319333
## Files
320334

321335
- `.media/manifest.jsonl`: machine SSOT, one JSON record per line
322336
- `.media/index.md`: agent-readable table (id, type, dur, dims, path, description)
323337
- `~/.media/`: global cross-project reuse cache (content-addressed, SHA-256)
338+
- `~/.media/misses.jsonl`: local-only resolve misses, including intent text for `--stats`
324339

325340
## Audio engine: voiceover, music, SFX, captions, transcription
326341

@@ -389,3 +404,12 @@ resolve never waits on or fails from telemetry).
389404

390405
Opt out with `DO_NOT_TRACK=1` or `HYPERFRAMES_NO_TELEMETRY=1` (also off in CI and
391406
dev). Same public PostHog project key and opt-outs as the `hyperframes` CLI.
407+
408+
## Privacy
409+
410+
media-use uses the same shared install id as the `hyperframes` CLI/studio
411+
(`~/.hyperframes/config.json`). When you are signed in to HeyGen, usage is
412+
linked to your account email, or username when email is unavailable, matching
413+
the CLI behavior. The events stay coarse: media type, source, provider, and
414+
small counts only; intent text and paths stay local. Disable telemetry with
415+
`HYPERFRAMES_NO_TELEMETRY=1` or `DO_NOT_TRACK=1`.
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# media-use usage dashboard (PostHog)
2+
3+
Reproducible definition of the media-use usage dashboard. The dashboard answers
4+
"how much is media-use used, for what, is reuse working, and what can't it
5+
satisfy" from the telemetry `scripts/lib/telemetry.mjs` already emits. Build it
6+
in PostHog project **Hyperframes (356858)**; this doc is the source of truth so
7+
it can be recreated. Local complement: `resolve --stats` (same questions, from
8+
`.media/` + `~/.media`, no PostHog access needed).
9+
10+
## Identity (see `scripts/lib/telemetry.mjs`)
11+
12+
Events attribute to the **same PostHog person as the hyperframes CLI and studio**
13+
— the shared install id in `~/.hyperframes/config.json` (`anonymousId`), stitched
14+
to the HeyGen account (`$identify`, `distinct_id` = email/username) on sign-in.
15+
Not fully anonymous by design; pseudonymous before sign-in, account-linked after.
16+
`$ip:null`. Opt-out: `HYPERFRAMES_NO_TELEMETRY=1` / `DO_NOT_TRACK=1` (also CI, dev).
17+
18+
## Event catalog (verified present in-project)
19+
20+
Every event carries `surface: "media-use"`. Event **properties are coarse**
21+
never intent text, file names, or paths.
22+
23+
| Event | Fires on | Key properties |
24+
| ---------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------- |
25+
| `media_use_resolve` | a resolve that produced/returned an asset | `type`, `source`, `provider`, `via`, `local_only`, `provider_override` |
26+
| `media_use_resolve_miss` | a resolve that found nothing | `type`, `local_only`, `provider_override` (no intent) |
27+
| `media_use_candidates` | `--candidates` / `--dry-run` listing | `type`, counts |
28+
| `media_use_doctor_run` | `--doctor` | `ok`, `checks_failed`, `failed[]` |
29+
| `media_use_compare` | `grade-compare` / `compare` | `command`, `cells`, `truncated`, `total`, `render_ready_timed_out` |
30+
| `media_use_transcribe` · `media_use_duck` · `media_use_transcript_cut` | audio-engine ops | op-specific |
31+
32+
## Dashboard tiles
33+
34+
1. **Invocation volume**`query-trends`, count of `media_use_resolve` over time (daily). "How much."
35+
2. **By media type**`media_use_resolve` broken down by `type` (bgm/sfx/image/icon/logo/voice/grade/lut). "For what."
36+
3. **Resolve hit-rate** — trends formula: `A / (A + B)` where A = `media_use_resolve`, B = `media_use_resolve_miss`. "Is the catalog covering needs."
37+
4. **Provider mix**`media_use_resolve` broken down by `provider`; a second tile by `via` (`url` / `params-fallback` / `params`) to catch CDN→params LUT downgrades.
38+
5. **Top misses**`media_use_resolve_miss` broken down by `type` (the tuning signal — pair with local `resolve --stats`, which also shows the missed _intents_ that telemetry deliberately omits).
39+
6. **Doctor health**`media_use_doctor_run` broken down by `failed[]` (which dependency check fails most) + `checks_failed` distribution.
40+
7. **Compare cost**`media_use_compare` by `command`, plus `truncated` / `render_ready_timed_out` rates (observe before lifting the 16-cell cap).
41+
8. **Adoption (optional)** — if the `first_run` property ships (plan U5), segment `media_use_resolve` first-run vs repeat.
42+
43+
## Recreate via the PostHog MCP
44+
45+
For each tile: `read-data-schema` to confirm the event/property, then a
46+
`query-*` tool (`query-trends` for 1–7), then `insight-create`, then
47+
`dashboard-create` collecting the insights. Keep names prefixed `media-use:` so
48+
the dashboard is greppable. Cross-surface note: because identity is shared with
49+
CLI/studio, you can also break these down by the same person across `cli_command*`
50+
and `studio:*` events.
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
import { appendFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
2+
import { homedir } from "node:os";
3+
import { join } from "node:path";
4+
5+
const MISSES_FILE = "misses.jsonl";
6+
7+
function missesPath() {
8+
return join(homedir(), ".media", MISSES_FILE);
9+
}
10+
11+
export function recordMiss({ type, intent, provider_override, local_only }) {
12+
try {
13+
const dir = join(homedir(), ".media");
14+
mkdirSync(dir, { recursive: true });
15+
appendFileSync(
16+
join(dir, MISSES_FILE),
17+
JSON.stringify({
18+
ts: new Date().toISOString(),
19+
type,
20+
intent,
21+
provider_override: !!provider_override,
22+
local_only: !!local_only,
23+
}) + "\n",
24+
);
25+
} catch {
26+
// local miss logging is best-effort; never surface into resolve
27+
}
28+
}
29+
30+
export function readMisses() {
31+
const p = missesPath();
32+
try {
33+
if (!existsSync(p)) return [];
34+
const raw = readFileSync(p, "utf8");
35+
const records = [];
36+
for (const line of raw.split(/\r?\n/)) {
37+
const trimmed = line.trim();
38+
if (!trimmed) continue;
39+
try {
40+
records.push(JSON.parse(trimmed));
41+
} catch {
42+
// skip malformed local lines, don't crash stats
43+
}
44+
}
45+
return records;
46+
} catch {
47+
return [];
48+
}
49+
}
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
import { strict as assert } from "node:assert";
2+
import { test } from "node:test";
3+
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
4+
import { tmpdir } from "node:os";
5+
import { join } from "node:path";
6+
import { readMisses, recordMiss } from "./misses.mjs";
7+
8+
function sandbox() {
9+
const root = mkdtempSync(join(tmpdir(), "mu-misses-"));
10+
const home = join(root, "home");
11+
mkdirSync(home, { recursive: true });
12+
process.env.HOME = home;
13+
return { root, home };
14+
}
15+
16+
function restoreEnv(saved) {
17+
for (const k of Object.keys(process.env)) if (!(k in saved)) delete process.env[k];
18+
Object.assign(process.env, saved);
19+
}
20+
21+
test("recordMiss appends a well-formed local miss", () => {
22+
const savedEnv = { ...process.env };
23+
const { root, home } = sandbox();
24+
try {
25+
recordMiss({ type: "bgm", intent: "moody synth pulse", provider_override: true });
26+
const misses = readMisses();
27+
assert.equal(misses.length, 1);
28+
assert.equal(misses[0].type, "bgm");
29+
assert.equal(misses[0].intent, "moody synth pulse");
30+
assert.equal(misses[0].provider_override, true);
31+
assert.equal(misses[0].local_only, false);
32+
assert.ok(!Number.isNaN(Date.parse(misses[0].ts)));
33+
34+
const raw = readFileSync(join(home, ".media/misses.jsonl"), "utf8");
35+
assert.match(raw, /moody synth pulse/);
36+
} finally {
37+
restoreEnv(savedEnv);
38+
rmSync(root, { recursive: true, force: true });
39+
}
40+
});
41+
42+
test("recordMiss swallows filesystem failures", () => {
43+
const savedEnv = { ...process.env };
44+
const { root, home } = sandbox();
45+
try {
46+
writeFileSync(join(home, ".media"), "not a directory");
47+
assert.doesNotThrow(() =>
48+
recordMiss({ type: "image", intent: "unwritable", local_only: true }),
49+
);
50+
assert.equal(existsSync(join(home, ".media/misses.jsonl")), false);
51+
} finally {
52+
restoreEnv(savedEnv);
53+
rmSync(root, { recursive: true, force: true });
54+
}
55+
});
56+
57+
test("readMisses skips corrupt lines", () => {
58+
const savedEnv = { ...process.env };
59+
const { root, home } = sandbox();
60+
try {
61+
mkdirSync(join(home, ".media"), { recursive: true });
62+
writeFileSync(
63+
join(home, ".media/misses.jsonl"),
64+
[
65+
JSON.stringify({ ts: "2026-07-09T00:00:00.000Z", type: "bgm", intent: "one" }),
66+
"{not json",
67+
JSON.stringify({ ts: "2026-07-09T00:00:01.000Z", type: "sfx", intent: "two" }),
68+
].join("\n"),
69+
);
70+
assert.deepEqual(
71+
readMisses().map((miss) => miss.intent),
72+
["one", "two"],
73+
);
74+
} finally {
75+
restoreEnv(savedEnv);
76+
rmSync(root, { recursive: true, force: true });
77+
}
78+
});
Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
import { statSync } from "node:fs";
2+
import { readGlobalManifest } from "./cache.mjs";
3+
import { readManifest } from "./manifest.mjs";
4+
import { readMisses } from "./misses.mjs";
5+
6+
const TOP_MISSES = 5;
7+
8+
function emptyReport() {
9+
return {
10+
total_resolves: 0,
11+
by_type: {},
12+
by_source: {},
13+
by_provider: {},
14+
by_via: {},
15+
misses: 0,
16+
hit_rate: null,
17+
top_missed_intents: {},
18+
global_cache_assets: 0,
19+
global_cache_disk_bytes: 0,
20+
cross_project_reuse: 0,
21+
};
22+
}
23+
24+
function increment(map, key) {
25+
if (!key) return;
26+
map[key] = (map[key] || 0) + 1;
27+
}
28+
29+
function timestampOf(record) {
30+
return record?.ts || record?.timestamp || record?.created_at || record?.createdAt || null;
31+
}
32+
33+
function inWindow(record, cutoff) {
34+
if (!cutoff) return true;
35+
const ts = timestampOf(record);
36+
// Older manifest records may not carry a timestamp; keep them in the report
37+
// because --days can only window records/misses that carry a ts/timestamp.
38+
if (!ts) return true;
39+
const time = Date.parse(ts);
40+
return Number.isNaN(time) ? true : time >= cutoff;
41+
}
42+
43+
function sourceOf(record) {
44+
return record?._source || record?.source || record?.provenance?.source || "unknown";
45+
}
46+
47+
function normalizeIntent(intent) {
48+
return String(intent ?? "")
49+
.trim()
50+
.toLowerCase()
51+
.replace(/\s+/g, " ");
52+
}
53+
54+
function topMissedIntents(misses) {
55+
const grouped = {};
56+
for (const miss of misses) {
57+
const type = miss?.type || "unknown";
58+
const intent = normalizeIntent(miss?.intent);
59+
if (!intent) continue;
60+
grouped[type] ||= {};
61+
grouped[type][intent] = (grouped[type][intent] || 0) + 1;
62+
}
63+
const out = {};
64+
for (const [type, intents] of Object.entries(grouped)) {
65+
out[type] = Object.entries(intents)
66+
.map(([intent, count]) => ({ intent, count }))
67+
.sort((a, b) => b.count - a.count || a.intent.localeCompare(b.intent))
68+
.slice(0, TOP_MISSES);
69+
}
70+
return out;
71+
}
72+
73+
function diskBytes(records) {
74+
let total = 0;
75+
for (const record of records) {
76+
const p = record?.cached_path || record?.path;
77+
if (!p) continue;
78+
try {
79+
total += statSync(p).size;
80+
} catch {
81+
// cache entries can outlive files; stats skips missing files
82+
}
83+
}
84+
return total;
85+
}
86+
87+
export function buildStats({ projectDir, days, now = Date.now() } = {}) {
88+
// Only a positive finite --days windows the report; null / NaN / <= 0 mean
89+
// "all time" rather than silently excluding everything (a negative cutoff
90+
// would land in the future and drop every record). The reads below are each
91+
// best-effort (they return [] / skip on IO errors), so there is no top-level
92+
// catch masking a real logic bug as an all-zero "no usage" report.
93+
const n = Number(days);
94+
const cutoff = Number.isFinite(n) && n > 0 ? Number(now) - n * 24 * 60 * 60 * 1000 : null;
95+
const records = (projectDir ? readManifest(projectDir) : []).filter((r) => inWindow(r, cutoff));
96+
const misses = readMisses().filter((miss) => inWindow(miss, cutoff));
97+
const globalRecords = readGlobalManifest();
98+
const report = emptyReport();
99+
100+
report.total_resolves = records.length;
101+
report.misses = misses.length;
102+
for (const record of records) {
103+
increment(report.by_type, record?.type || "unknown");
104+
increment(report.by_source, sourceOf(record));
105+
increment(report.by_provider, record?.provenance?.provider);
106+
increment(report.by_via, record?.provenance?.via);
107+
}
108+
109+
const attempts = report.total_resolves + report.misses;
110+
report.hit_rate = attempts === 0 ? null : report.total_resolves / attempts;
111+
report.top_missed_intents = topMissedIntents(misses);
112+
report.global_cache_assets = globalRecords.length;
113+
report.global_cache_disk_bytes = diskBytes(globalRecords);
114+
report.cross_project_reuse = globalRecords.filter((r) => r?.provenance?.reused_by).length;
115+
116+
return report;
117+
}

0 commit comments

Comments
 (0)