Working notes for building the system. The design doc is the brief; this is what to actually do, in what order, and what will bite.
The prompts worth generating are the ones real potential customers of the client are asking, where the client is a legitimate answer.
Two-part test every prompt must pass:
- Someone actually asks it — grounded in impressions, ticket volume, issue counts. Not invented.
- The client belongs in the answer — grounded in verified capabilities.
Fail (1) and you're tracking a query nobody types. Fail (2) and you're tracking one they deserve to lose. The capsule enforces both: search_intent + pain_points prove someone asks it, capabilities proves the client belongs.
Ignore the top-of-funnel / bottom-of-funnel framing. It's the wrong axis. Grounding is the axis.
Second axis: ICP. Most clients sell to several distinct buyer segments who ask about the same product in almost no shared vocabulary. Mastra's individual agent builder asks "how do I stop my agent looping"; its enterprise platform lead asks "which framework gives per-agent trace isolation and audit logging." Same capability, opposite register, and neither prompt surfaces the other's results.
Persona is a generation axis, not a metadata field. Generate across archetype × ICP, and enforce a per-ICP floor. Twenty prompts that all sound like one buyer is a set-level failure no per-prompt check can see.
Connectors → normalized signals → Company Capsule → synthesis → validation → prompt records that feed the existing eval service.
This replaces the multi-hour pipeline, so the bar is parity, not speed. A fast set that's merely earlier than the slow one is not a replacement. Build accordingly: the validation cascade and the benchmark are not polish, they're the claim.
Slack is out of scope. Deferred deliberately, reasoning in the design doc §8.
clients (id, name, domain, sources_json, created_at)
raw_signals (
id, client_id, source, kind, text, ref, weight,
persona, -- nullable FK to personas.slug
fetched_at
)
personas (
id, client_id, slug, label, evidence_weight, vocabulary_json, source
)
capsules (
id, client_id, version, xml, coverage_json, directives_json, built_at
)The persona column on raw_signals is the ICP change, and it is the one thing that's expensive to retrofit. Add it before writing adapters.
Most signals are unattributed — a GitHub issue has no known segment, so persona is NULL. Signals from call transcripts do carry a segment, because you segmented by deal size before extracting. That attribution is what lets the capsule say "enterprise buyers raise this objection" rather than "someone raised this objection."
personas.vocabulary_json holds the terms that segment actually uses (["audit trail", "RBAC", "data residency"]). Without it, persona differentiation exists only as a label and the model writes the same prompt four times with the field swapped.
raw_signals is the cache layer. Adapters write here. The capsule builder reads only from here, never from an API. This lets you rebuild capsules without re-hitting anything, which you'll do constantly while tuning.
capsules is versioned, never overwritten. Gives you a diff when output quality shifts.
build_capsule()must run to completion with the network disabled.
If it can't, something is fetching where it should be reading from cache. Enforce this from day one — it's the single check that keeps the cache boundary from eroding as the code grows.
You almost certainly do not need vector embeddings for client data. Nothing here does semantic retrieval — you run filtered queries and hand the whole capsule to the model. The only embedding use is the diversity pass over ~40 candidate prompts, and that's transient. "Embed vs. query" reduces to cache vs. fetch.
| Source | Strategy | Cadence | Why |
|---|---|---|---|
| Google Search Console | Cache | Nightly | Slow, paginated, stable day-to-day |
| Ahrefs | Cache | Weekly | Metered, changes slowly |
| GitHub issues | Cache | Daily | Rate-limited, high volume |
| Sales call transcripts | Cache | Weekly | Large, slow, stable |
| Onboarding recordings | Cache | On upload | First-party to us |
| llms.txt / homepage | Fetch | At build | Tiny, must be fresh |
| Docs tree | Fetch | At build | Small, deterministic location |
| Pricing / case studies | Fetch | At build | Small |
Rule: cache anything paginated, metered, or rate-limited. Fetch anything that's one small request.
Third category — derive and discard: for PII-bearing or high-volume sources, extract at ingest and persist only the derived string plus a ref (source, id, ts). Never store raw bodies. One decision covers storage cost, privacy, and context limits.
Every capsule slot has a tech and a non-tech source that plays the identical role. One adapter interface, no vertical branching.
| # | Slot | Tech | Non-tech | Auth |
|---|---|---|---|---|
| 1 | identity |
llms.txt, homepage | Homepage | None |
| 2 | capabilities |
Docs tree, mint.json |
Product catalog, category nav | None |
| 3 | pain_points |
GitHub issue titles | Public reviews (Reddit, Trustpilot) | None |
| 4 | buyer_personas |
Pricing tiers + gates | Product-line / price-point tiers | None |
| 5 | search_intent |
Google Search Console | Google Search Console | Credentialed |
Plus Ahrefs → competitive_landscape, credentialed, same for both.
Implication for the build: 1–4 run with zero credentials, so the whole pipeline is demoable end-to-end on day one. Write the vertical detection once and let it pick the adapter:
def pick_adapters(domain):
has_docs = probe(f"https://{domain}/docs") or probe(f"https://{domain}/mint.json")
has_repo = github_org_for(domain)
return {
"capabilities": DocsTreeAdapter if has_docs else CatalogNavAdapter,
"pain_points": GitHubIssuesAdapter if has_repo else PublicReviewAdapter,
"personas": PricingTierAdapter, # both verticals
"identity": HomepageAdapter,
"search_intent": GSCFixtureAdapter, # swap to GSCAdapter with creds
}Same normalized output either way, so nothing downstream changes.
This is where signal quality is won or lost. The specific query matters more than the integration.
Search Analytics API, dimension=query, 90-day window.
- Phrasing pull — question stems (
^(how|why|what|best|vs)) sorted by impressions. Gives authentic cadence. - Gap pull —
impressions > threshold AND position > 10 AND ctr < baseline.
The gap pull is the highest-value extraction in the system. These are queries where potential buyers are actively looking and the client is not being found. It's the thesis restated as a database filter, from first-party data.
Implementation note: the CTR baseline has to be picked per client. Site-wide average CTR by position is a reasonable start.
Titles-only bypasses PII in bodies and keeps payloads small enough for one fast-model extraction call.
- GitHub — issues API,
labels=bug,sort=comments&direction=desc. Comment count is a usable pain-intensity proxy: a 40-comment issue is a real symptom, a 0-comment one is noise. - Intercom — conversations search, 90 days,
source.subjectonly. Weight by frequency of similar subjects. - Linear — GraphQL, filter by label, titles only.
This is the only honest source for the Implicit archetype. Marketing copy describes solutions; it never describes frustration.
Run in order. Each refines the previous. Ships with 1–3; 4–6 are additive.
1. Pricing tiers + feature gates (build this first — highest value, zero auth)
def personas_from_pricing(html):
tiers = parse_tiers(html) # name, price, feature list
base = set(tiers[0].features)
out = []
for t in tiers:
gated = set(t.features) - base # what THIS tier unlocks
out.append(Persona(
slug=slugify(t.name),
label=t.name,
vocabulary=extract_terms(gated), # "SSO","RBAC","audit log","SLA"
evidence_weight=3,
gated_capabilities=list(gated), # → capability→persona map
))
return outThe gated set is the whole trick: it gives segment and vocabulary and the capability mapping in one parse. Nothing else does all three.
2. GSC behavioral split (free — rows are already in raw_signals)
SELECT
CASE
WHEN landing_page ~ '/(enterprise|security|compliance|pricing)' THEN 'enterprise'
WHEN landing_page ~ '/(docs|quickstart|guides|api)' THEN 'practitioner'
END AS segment,
query, impressions, position
FROM raw_signals
WHERE client_id = $1 AND source = 'gsc' AND segment IS NOT NULL;Second cut on vocabulary: procurement register (soc 2|sso|on-prem|for teams) vs practitioner (how do i|error|example). This is a GROUP BY, not an integration.
3. Docs IA + analytics — top-level section split reveals audiences; pageview mix reveals which is real. /security traffic means enterprise evaluation is actually happening.
4. Job postings — enterprise AEs and compliance leads mean a real enterprise motion. Only method that catches segments the client is investing in before data accumulates.
5. Call data — highest fidelity. Sales transcripts (Gong, Chorus), last two quarters, segment by deal size before extracting or ICPs blend into one average buyer. Per segment: role, objections (→ pain_points, weighted high), vocabulary. Onboarding recordings are first-party to us — no client credential, no approval, clients name their segments unprompted in the first ten minutes.
6. Client confirmation — render derived personas at generation time, let them edit. Beats every method above and catches expansion segments that have no data anywhere.
Metadata join where it exists (Intercom plan tier, GitHub org) → real attribution, free, use it. Text classification → ~70% accurate and fails in the direction that hurts.
Structural reason: tickets are filed by users, ICPs are defined by buyers. The platform lead evaluating an enterprise purchase never files a ticket; five engineers on their team file in individual-builder register. Same person for self-serve, different people for enterprise — exactly the split you're capturing.
Prevents four prompts about one feature with the field swapped.
def map_capabilities(capabilities, personas):
gated = {c: p.slug for p in personas for c in p.gated_capabilities}
for cap in capabilities:
cap.personas = ([gated[cap.text]] if cap.text in gated
else match_by_vocabulary(cap, personas) or ALL)Pricing gates supply most of it. Fall back to vocabulary overlap, then to "relevant to all."
Flat 5/5/5/5 per persona is wrong — segments search differently.
ARCHETYPE_MIX = {
"practitioner": {"Implicit": .35, "Command": .35, "Explicit": .20, "Multi-Turn": .10},
"enterprise": {"Implicit": .10, "Command": .15, "Explicit": .40, "Multi-Turn": .35},
}Builders are mid-task with a symptom. Enterprise buyers are comparing vendors and adding constraints. Starting values — tune against §7.
- Ahrefs — competing domains endpoint + keywords matching
vs|alternative|instead of. - Docs — sitemap or
mint.jsonH1/H2 tree. If docs analytics available, rank by pageview so Command prompts hit features people actually use, not whatever sits first in the nav. - Pricing + case studies — tiers name segments, cases name job titles. This is your
personassource. - llms.txt / homepage — root fetch, fall back to H1 + meta.
<company_capsule>
<identity> <!-- name, category, value prop, region, locale -->
<official_capabilities>
<user_search_intent>
<community_pain_points>
<competitive_landscape>
<buyer_personas>
</company_capsule><coverage> is tracked separately and never injected. It drives quotas only.
def build_capsule(client_id, total=20):
signals = fetch_signals(client_id) # from raw_signals only
personas = fetch_personas(client_id)
slots = {}
for slot, kind in SLOT_MAP.items():
rows = [s for s in signals if s.kind == kind]
rows.sort(key=lambda r: -r.weight)
slots[slot] = rows[:CAPS[slot]]
# pain points carry their segment through, so the model can see
# which ICP raised what
for row in slots["community_pain_points"]:
row.attrs["persona"] = row.persona # may be None
coverage = {k: len(v) for k, v in slots.items()}
directives = build_directives(personas, coverage, total)
return render_xml(slots, personas), coverage, directives
def build_directives(personas, coverage, total):
quotas = archetype_quotas(coverage, total) # existing degradation logic
# floors distribute by evidence, not evenly
if not personas:
floors = {} # generic single persona, flag in UI
else:
w = sum(p.evidence_weight for p in personas)
floors = {
p.slug: max(2, round(total * 0.7 * p.evidence_weight / w))
for p in personas
}
return {"quotas": quotas, "floors": floors,
"candidates_per_archetype": 10}Why 0.7 and max(2, ...): floors are a lower bound, not a partition. Summing them to the full total would over-constrain the generator and leave no slack for the best candidates to win on merit. Reserving ~30% unfloored lets quality decide the remainder. The max(2, ...) stops a thinly-evidenced ICP from rounding to zero and disappearing entirely — which is the exact failure the axis exists to prevent. Both numbers are starting values; tune them.
Per-slot caps (~12 capabilities, ~8 pain points, ~6 personas) solve the size-variance problem: a large client truncates to top-N by weight, a small one gets what it has, neither starves another slot.
Weight is set by the adapter and normalized 1–10 so cross-source sorting is meaningful: GitHub uses comment count, GSC uses impressions, Intercom uses subject frequency, sales calls use deal count per segment.
Persona floors derive from weight. An ICP backed by 30 sales calls and a pricing tier gets a higher floor than one inferred from a single case study. Compute floors in the builder and emit them in <generation_directives> alongside archetype quotas.
Not within one company — between companies. Expect these and don't read them as design failure:
- Slot sizes vary enormously. Hundreds of docs pages vs. six. Per-slot caps handle it.
- "Capability" means different things. SDK method vs. shoe model. Resist the pull toward per-industry schemas — one schema + per-source adapters is what stays maintainable.
- Empty slots are the common case, not the edge case. Build for incompleteness from the start.
- Weight and provenance need a home. Decide XML attribute vs. flattened text before writing adapters. Retrofitting is painful.
- Most signals have no persona. NULL is the common case, not an error. Don't build logic that assumes attribution exists — pain points from GitHub are still valid, just unattributed.
- ICP counts vary wildly. A single-segment client and a five-segment client both have to work. Floors computed from weight handle both; a hardcoded split doesn't.
Deterministic slot-filling, not agentic retrieval. The generator receives a complete capsule and gets no tools. A model given the option to look something up will decide it has enough. Pre-filling every available slot is the mechanism that satisfies "does not miss context when it could have had it."
Note this answers two separate requirements, and say so in the writeup:
- References context correctly → truth lock +
source_evidence - Doesn't miss context → deterministic slot-filling
Prefix must be byte-identical across clients or the cache never hits — which is exactly why counts and quotas can't live in it.
[ STATIC PREFIX — cacheable ] master prompt · contract · archetypes · golden examples
[ DYNAMIC SUFFIX — per run ] <company_capsule> · <generation_directives>
Quotas travel as data:
<generation_directives>
<candidates_per_archetype>10</candidates_per_archetype>
<archetype name="Explicit" quota="7"/>
<archetype name="Implicit" quota="2"/>
</generation_directives>A hardcoded "5 per archetype" contradicts the truth lock the moment a connector is missing, and the model resolves that conflict by fabricating.
- Schema check (deterministic) — valid JSON, every field non-empty.
- Brand-leak regex (deterministic) — Implicit prompt containing brand, product names, or category jargon → auto-fail.
- Quality judge (small model) — binary pass/fail on constraint specificity. Use structured output or prefill;
logit_biasisn't portable across providers. - Diversity pass (embeddings) — drop anything above cosine threshold to an already-selected prompt.
- Persona coverage (deterministic, set-level) — assert every ICP hits its floor; backfill from that ICP's candidate pool if short.
Stages 2 and 4 decide whether output looks impressive or looks generated. Both are cheap. Both are the first things skipped under time pressure. Don't skip them.
- Stage 2 catches the most likely and most visible failure: Implicit prompts naming the brand. The model drifts back to this constantly.
- Stage 4 catches the second most likely: twenty individually-valid near-paraphrases. A per-prompt binary judge structurally cannot detect this.
- Stage 5 catches the opposite of stage 4: prompts diverse in wording but all aimed at one buyer segment. Both are set-level failures, invisible from any single prompt.
Provenance: every prompt carries source_evidence — the verbatim capsule string it derives from. Turns the truth lock from an instruction into a substring match.
def enforce_persona_floors(selected, candidates, floors):
for slug, floor in floors.items():
have = [p for p in selected if p["Personas"] == slug]
if len(have) >= floor:
continue
pool = [c for c in candidates
if c["Personas"] == slug and c not in selected]
pool.sort(key=lambda c: -c["_judge_score"])
need = floor - len(have)
if len(pool) < need:
log.warning("persona %s short: %d/%d", slug, len(have) + len(pool), floor)
selected += pool[:need] # backfill, still diversity-checked
return selectedLog the shortfall rather than silently accepting it. An ICP that can't reach its floor means the capsule lacks evidence for that segment — a data problem to fix upstream, not an output to quietly ship. Surfacing it is how the persona axis stays honest instead of decorative.
Backfilled prompts still pass through the diversity check; the floor is a lower bound on coverage, not a licence to ship near-duplicates.
Required fields, from existing production records:
Topic · Prompt · Tags · Regions · Language · Platforms · Personas · Analysis Types
Checks before handoff:
- Does the service reject unknown keys?
Archetypeandsource_evidenceare internal. Keep them for validation, strip before handoff unless the service tolerates extras. Cheap to verify, annoying to discover late. - Match
Platformsto what's actually tracked. Production records saychatgpt, perplexity, google-ai-overviews. The master prompt currently addsclaude. Fix unless Claude is tracked. - Fill
TagsandPersonas. Production ships them empty. Yours won't — visible improvement, no downside.Personasmust match a persona name from the directives block, not free text, or the coverage check can't verify anything. - Hard floor of 10 prompts. The brief says "at least ten." Degradation reallocates per archetype but doesn't guarantee a total. Add a floor so archetypes with grounded slots absorb the reallocation.
Build the capsule before the synthesis call. The temptation is to start with the model call because output appears immediately — but then you're tuning prompt quality against fake context and you'll tune the wrong things. Get one real capsule from real sources first, even half-empty. What it actually looks like will change your synthesis decisions.
- Adapters + normalization →
raw_signals. Largest chunk. Include thepersonacolumn from the start — retrofitting attribution after adapters exist means re-running every extraction. - Capsule builder + coverage + directives. Verify the no-network invariant.
- Synthesis + schema check. End-to-end prompts out. Demoable.
- Full validation cascade. Stages 2 and 4 are what make output visibly good.
- Coverage → quota reallocation and persona floors. Demo by toggling sources off; toggle personas off separately to show the axis is independent.
- Golden capsule snapshots for regression.
Four of six slots run live without client credentials:
| Slot | Status |
|---|---|
identity |
Live — public HTTP |
capabilities |
Live — public HTTP |
personas |
Live — public HTTP |
pain_points |
Live — public GitHub repo API |
search_intent |
Fixture — gsc_sample.json |
competitive_landscape |
Fixture — ahrefs_sample.json |
Fixtures must carry real field shapes (query, impressions, clicks, ctr, position) so the gap-pull filter actually executes rather than being described.
Stated up front it reads as scoping. Discovered by the reviewer it reads as a gap.
Vercel — public repo, docs, pricing. Three adapters live, two fixture. Full capsule, all archetypes.
Nike — no repo, no docs site, no tracker. pain_points falls through to public forum search. Coverage shows the gap, quotas reallocate, Implicit ships reduced rather than fabricating.
The Nike case is the better demo. It exercises degradation under real conditions rather than a toggle, and proves the system isn't tuned to developer tooling.
Say these; don't let them be discovered.
- Public-source parsing is brittle. llms.txt adoption is low, homepages are JS-rendered, pricing pages share no structure. Largest chunk of unglamorous work; expect flaky rather than solved.
- Thresholds are starting values, not derived. Cosine cutoff, candidate counts, symptom counts. Say so rather than defending an invented constant.
- Real GSC volume untested. The filter runs correctly on fixtures. Whether it surfaces good queries at real volume and CTR distribution is unknown.
- Ahrefs competitor quality untested. Hand-picked fixtures flatter the system; real output is noisier.
- Benchmark not run. The existing pipeline is both the incumbent and the labeled eval set — measure topic-coverage overlap, survival rate, and novel-prompt rate against it. This is the gating item for the replacement claim, and it needs internal access. Until it runs, say "designed to replace, pending validation." Migration is parallel-run: ship the fast set immediately, retire the slow pipeline per-client once thresholds clear, so a shortfall costs nothing.
- Working system, end to end
- The actual generated prompts, with
source_evidenceon each — the brief asks for prompts, not just a generator. This is the part that gets read closely. - Design doc
- Access limits stated up front
- Both demo companies runnable