Skip to content

Latest commit

 

History

History
466 lines (319 loc) · 24.1 KB

build-guide.md

File metadata and controls

466 lines (319 loc) · 24.1 KB

Build Guide

Working notes for building the system. The design doc is the brief; this is what to actually do, in what order, and what will bite.


The thesis — one line

The prompts worth generating are the ones real potential customers of the client are asking, where the client is a legitimate answer.

Two-part test every prompt must pass:

  1. Someone actually asks it — grounded in impressions, ticket volume, issue counts. Not invented.
  2. The client belongs in the answer — grounded in verified capabilities.

Fail (1) and you're tracking a query nobody types. Fail (2) and you're tracking one they deserve to lose. The capsule enforces both: search_intent + pain_points prove someone asks it, capabilities proves the client belongs.

Ignore the top-of-funnel / bottom-of-funnel framing. It's the wrong axis. Grounding is the axis.

Second axis: ICP. Most clients sell to several distinct buyer segments who ask about the same product in almost no shared vocabulary. Mastra's individual agent builder asks "how do I stop my agent looping"; its enterprise platform lead asks "which framework gives per-agent trace isolation and audit logging." Same capability, opposite register, and neither prompt surfaces the other's results.

Persona is a generation axis, not a metadata field. Generate across archetype × ICP, and enforce a per-ICP floor. Twenty prompts that all sound like one buyer is a set-level failure no per-prompt check can see.


What you're building

Connectors → normalized signals → Company Capsule → synthesis → validation → prompt records that feed the existing eval service.

This replaces the multi-hour pipeline, so the bar is parity, not speed. A fast set that's merely earlier than the slow one is not a replacement. Build accordingly: the validation cascade and the benchmark are not polish, they're the claim.

Slack is out of scope. Deferred deliberately, reasoning in the design doc §8.


Storage — three tables

clients (id, name, domain, sources_json, created_at)

raw_signals (
  id, client_id, source, kind, text, ref, weight,
  persona,            -- nullable FK to personas.slug
  fetched_at
)

personas (
  id, client_id, slug, label, evidence_weight, vocabulary_json, source
)

capsules (
  id, client_id, version, xml, coverage_json, directives_json, built_at
)

The persona column on raw_signals is the ICP change, and it is the one thing that's expensive to retrofit. Add it before writing adapters.

Most signals are unattributed — a GitHub issue has no known segment, so persona is NULL. Signals from call transcripts do carry a segment, because you segmented by deal size before extracting. That attribution is what lets the capsule say "enterprise buyers raise this objection" rather than "someone raised this objection."

personas.vocabulary_json holds the terms that segment actually uses (["audit trail", "RBAC", "data residency"]). Without it, persona differentiation exists only as a label and the model writes the same prompt four times with the field swapped.

raw_signals is the cache layer. Adapters write here. The capsule builder reads only from here, never from an API. This lets you rebuild capsules without re-hitting anything, which you'll do constantly while tuning.

capsules is versioned, never overwritten. Gives you a diff when output quality shifts.

The invariant that keeps it honest

build_capsule() must run to completion with the network disabled.

If it can't, something is fetching where it should be reading from cache. Enforce this from day one — it's the single check that keeps the cache boundary from eroding as the code grows.


Cache vs. fetch

You almost certainly do not need vector embeddings for client data. Nothing here does semantic retrieval — you run filtered queries and hand the whole capsule to the model. The only embedding use is the diversity pass over ~40 candidate prompts, and that's transient. "Embed vs. query" reduces to cache vs. fetch.

Source Strategy Cadence Why
Google Search Console Cache Nightly Slow, paginated, stable day-to-day
Ahrefs Cache Weekly Metered, changes slowly
GitHub issues Cache Daily Rate-limited, high volume
Sales call transcripts Cache Weekly Large, slow, stable
Onboarding recordings Cache On upload First-party to us
llms.txt / homepage Fetch At build Tiny, must be fresh
Docs tree Fetch At build Small, deterministic location
Pricing / case studies Fetch At build Small

Rule: cache anything paginated, metered, or rate-limited. Fetch anything that's one small request.

Third category — derive and discard: for PII-bearing or high-volume sources, extract at ingest and persist only the derived string plus a ref (source, id, ts). Never store raw bodies. One decision covers storage cost, privacy, and context limits.


The five guaranteed sources

Every capsule slot has a tech and a non-tech source that plays the identical role. One adapter interface, no vertical branching.

# Slot Tech Non-tech Auth
1 identity llms.txt, homepage Homepage None
2 capabilities Docs tree, mint.json Product catalog, category nav None
3 pain_points GitHub issue titles Public reviews (Reddit, Trustpilot) None
4 buyer_personas Pricing tiers + gates Product-line / price-point tiers None
5 search_intent Google Search Console Google Search Console Credentialed

Plus Ahrefs → competitive_landscape, credentialed, same for both.

Implication for the build: 1–4 run with zero credentials, so the whole pipeline is demoable end-to-end on day one. Write the vertical detection once and let it pick the adapter:

def pick_adapters(domain):
    has_docs    = probe(f"https://{domain}/docs") or probe(f"https://{domain}/mint.json")
    has_repo    = github_org_for(domain)
    return {
      "capabilities": DocsTreeAdapter if has_docs else CatalogNavAdapter,
      "pain_points":  GitHubIssuesAdapter if has_repo else PublicReviewAdapter,
      "personas":     PricingTierAdapter,      # both verticals
      "identity":     HomepageAdapter,
      "search_intent": GSCFixtureAdapter,      # swap to GSCAdapter with creds
    }

Same normalized output either way, so nothing downstream changes.

Extraction, per source

This is where signal quality is won or lost. The specific query matters more than the integration.

Google Search Console — two pulls, not one

Search Analytics API, dimension=query, 90-day window.

  1. Phrasing pull — question stems (^(how|why|what|best|vs)) sorted by impressions. Gives authentic cadence.
  2. Gap pull — impressions > threshold AND position > 10 AND ctr < baseline.

The gap pull is the highest-value extraction in the system. These are queries where potential buyers are actively looking and the client is not being found. It's the thesis restated as a database filter, from first-party data.

Implementation note: the CTR baseline has to be picked per client. Site-wide average CTR by position is a reasonable start.

Support / issue titles — titles only

Titles-only bypasses PII in bodies and keeps payloads small enough for one fast-model extraction call.

  • GitHub — issues API, labels=bug, sort=comments&direction=desc. Comment count is a usable pain-intensity proxy: a 40-comment issue is a real symptom, a 0-comment one is noise.
  • Intercom — conversations search, 90 days, source.subject only. Weight by frequency of similar subjects.
  • Linear — GraphQL, filter by label, titles only.

This is the only honest source for the Implicit archetype. Marketing copy describes solutions; it never describes frustration.

ICP identification — a chain, not one source

Run in order. Each refines the previous. Ships with 1–3; 4–6 are additive.

1. Pricing tiers + feature gates (build this first — highest value, zero auth)

def personas_from_pricing(html):
    tiers = parse_tiers(html)              # name, price, feature list
    base  = set(tiers[0].features)
    out = []
    for t in tiers:
        gated = set(t.features) - base     # what THIS tier unlocks
        out.append(Persona(
            slug=slugify(t.name),
            label=t.name,
            vocabulary=extract_terms(gated),   # "SSO","RBAC","audit log","SLA"
            evidence_weight=3,
            gated_capabilities=list(gated),    # → capability→persona map
        ))
    return out

The gated set is the whole trick: it gives segment and vocabulary and the capability mapping in one parse. Nothing else does all three.

2. GSC behavioral split (free — rows are already in raw_signals)

SELECT
  CASE
    WHEN landing_page ~ '/(enterprise|security|compliance|pricing)' THEN 'enterprise'
    WHEN landing_page ~ '/(docs|quickstart|guides|api)'             THEN 'practitioner'
  END AS segment,
  query, impressions, position
FROM raw_signals
WHERE client_id = $1 AND source = 'gsc' AND segment IS NOT NULL;

Second cut on vocabulary: procurement register (soc 2|sso|on-prem|for teams) vs practitioner (how do i|error|example). This is a GROUP BY, not an integration.

3. Docs IA + analytics — top-level section split reveals audiences; pageview mix reveals which is real. /security traffic means enterprise evaluation is actually happening.

4. Job postings — enterprise AEs and compliance leads mean a real enterprise motion. Only method that catches segments the client is investing in before data accumulates.

5. Call data — highest fidelity. Sales transcripts (Gong, Chorus), last two quarters, segment by deal size before extracting or ICPs blend into one average buyer. Per segment: role, objections (→ pain_points, weighted high), vocabulary. Onboarding recordings are first-party to us — no client credential, no approval, clients name their segments unprompted in the first ten minutes.

6. Client confirmation — render derived personas at generation time, let them edit. Beats every method above and catches expansion segments that have no data anywhere.

Do NOT infer ICP from ticket text

Metadata join where it exists (Intercom plan tier, GitHub org) → real attribution, free, use it. Text classification → ~70% accurate and fails in the direction that hurts.

Structural reason: tickets are filed by users, ICPs are defined by buyers. The platform lead evaluating an enterprise purchase never files a ticket; five engineers on their team file in individual-builder register. Same person for self-serve, different people for enterprise — exactly the split you're capturing.

Capability → persona mapping

Prevents four prompts about one feature with the field swapped.

def map_capabilities(capabilities, personas):
    gated = {c: p.slug for p in personas for c in p.gated_capabilities}
    for cap in capabilities:
        cap.personas = ([gated[cap.text]] if cap.text in gated
                        else match_by_vocabulary(cap, personas) or ALL)

Pricing gates supply most of it. Fall back to vocabulary overlap, then to "relevant to all."

Per-ICP archetype weighting

Flat 5/5/5/5 per persona is wrong — segments search differently.

ARCHETYPE_MIX = {
  "practitioner": {"Implicit": .35, "Command": .35, "Explicit": .20, "Multi-Turn": .10},
  "enterprise":   {"Implicit": .10, "Command": .15, "Explicit": .40, "Multi-Turn": .35},
}

Builders are mid-task with a symptom. Enterprise buyers are comparing vendors and adding constraints. Starting values — tune against §7.

Others

  • Ahrefs — competing domains endpoint + keywords matching vs|alternative|instead of.
  • Docs — sitemap or mint.json H1/H2 tree. If docs analytics available, rank by pageview so Command prompts hit features people actually use, not whatever sits first in the nav.
  • Pricing + case studies — tiers name segments, cases name job titles. This is your personas source.
  • llms.txt / homepage — root fetch, fall back to H1 + meta.

The capsule

<company_capsule>
  <identity>              <!-- name, category, value prop, region, locale -->
  <official_capabilities>
  <user_search_intent>
  <community_pain_points>
  <competitive_landscape>
  <buyer_personas>
</company_capsule>

<coverage> is tracked separately and never injected. It drives quotas only.

Builder

def build_capsule(client_id, total=20):
    signals  = fetch_signals(client_id)       # from raw_signals only
    personas = fetch_personas(client_id)

    slots = {}
    for slot, kind in SLOT_MAP.items():
        rows = [s for s in signals if s.kind == kind]
        rows.sort(key=lambda r: -r.weight)
        slots[slot] = rows[:CAPS[slot]]

    # pain points carry their segment through, so the model can see
    # which ICP raised what
    for row in slots["community_pain_points"]:
        row.attrs["persona"] = row.persona     # may be None

    coverage   = {k: len(v) for k, v in slots.items()}
    directives = build_directives(personas, coverage, total)
    return render_xml(slots, personas), coverage, directives


def build_directives(personas, coverage, total):
    quotas = archetype_quotas(coverage, total)   # existing degradation logic

    # floors distribute by evidence, not evenly
    if not personas:
        floors = {}                              # generic single persona, flag in UI
    else:
        w = sum(p.evidence_weight for p in personas)
        floors = {
            p.slug: max(2, round(total * 0.7 * p.evidence_weight / w))
            for p in personas
        }
    return {"quotas": quotas, "floors": floors,
            "candidates_per_archetype": 10}

Why 0.7 and max(2, ...): floors are a lower bound, not a partition. Summing them to the full total would over-constrain the generator and leave no slack for the best candidates to win on merit. Reserving ~30% unfloored lets quality decide the remainder. The max(2, ...) stops a thinly-evidenced ICP from rounding to zero and disappearing entirely — which is the exact failure the axis exists to prevent. Both numbers are starting values; tune them.

Per-slot caps (~12 capabilities, ~8 pain points, ~6 personas) solve the size-variance problem: a large client truncates to top-N by weight, a small one gets what it has, neither starves another slot.

Weight is set by the adapter and normalized 1–10 so cross-source sorting is meaningful: GitHub uses comment count, GSC uses impressions, Intercom uses subject frequency, sales calls use deal count per segment.

Persona floors derive from weight. An ICP backed by 30 sales calls and a pricing tier gets a higher floor than one inferred from a single case study. Compute floors in the builder and emit them in <generation_directives> alongside archetype quotas.

Where the capsule will fight you

Not within one company — between companies. Expect these and don't read them as design failure:

  • Slot sizes vary enormously. Hundreds of docs pages vs. six. Per-slot caps handle it.
  • "Capability" means different things. SDK method vs. shoe model. Resist the pull toward per-industry schemas — one schema + per-source adapters is what stays maintainable.
  • Empty slots are the common case, not the edge case. Build for incompleteness from the start.
  • Weight and provenance need a home. Decide XML attribute vs. flattened text before writing adapters. Retrofitting is painful.
  • Most signals have no persona. NULL is the common case, not an error. Don't build logic that assumes attribution exists — pain points from GitHub are still valid, just unattributed.
  • ICP counts vary wildly. A single-segment client and a five-segment client both have to work. Floors computed from weight handle both; a hardcoded split doesn't.

Synthesis

Deterministic slot-filling, not agentic retrieval. The generator receives a complete capsule and gets no tools. A model given the option to look something up will decide it has enough. Pre-filling every available slot is the mechanism that satisfies "does not miss context when it could have had it."

Note this answers two separate requirements, and say so in the writeup:

  • References context correctly → truth lock + source_evidence
  • Doesn't miss context → deterministic slot-filling

Injection order

Prefix must be byte-identical across clients or the cache never hits — which is exactly why counts and quotas can't live in it.

[ STATIC PREFIX — cacheable ]  master prompt · contract · archetypes · golden examples
[ DYNAMIC SUFFIX — per run  ]  <company_capsule> · <generation_directives>

Quotas travel as data:

<generation_directives>
  <candidates_per_archetype>10</candidates_per_archetype>
  <archetype name="Explicit" quota="7"/>
  <archetype name="Implicit" quota="2"/>
</generation_directives>

A hardcoded "5 per archetype" contradicts the truth lock the moment a connector is missing, and the model resolves that conflict by fabricating.


Validation cascade — cheapest first

  1. Schema check (deterministic) — valid JSON, every field non-empty.
  2. Brand-leak regex (deterministic) — Implicit prompt containing brand, product names, or category jargon → auto-fail.
  3. Quality judge (small model) — binary pass/fail on constraint specificity. Use structured output or prefill; logit_bias isn't portable across providers.
  4. Diversity pass (embeddings) — drop anything above cosine threshold to an already-selected prompt.
  5. Persona coverage (deterministic, set-level) — assert every ICP hits its floor; backfill from that ICP's candidate pool if short.

Stages 2 and 4 decide whether output looks impressive or looks generated. Both are cheap. Both are the first things skipped under time pressure. Don't skip them.

  • Stage 2 catches the most likely and most visible failure: Implicit prompts naming the brand. The model drifts back to this constantly.
  • Stage 4 catches the second most likely: twenty individually-valid near-paraphrases. A per-prompt binary judge structurally cannot detect this.
  • Stage 5 catches the opposite of stage 4: prompts diverse in wording but all aimed at one buyer segment. Both are set-level failures, invisible from any single prompt.

Provenance: every prompt carries source_evidence — the verbatim capsule string it derives from. Turns the truth lock from an instruction into a substring match.


Stage 5 in code

def enforce_persona_floors(selected, candidates, floors):
    for slug, floor in floors.items():
        have = [p for p in selected if p["Personas"] == slug]
        if len(have) >= floor:
            continue
        pool = [c for c in candidates
                if c["Personas"] == slug and c not in selected]
        pool.sort(key=lambda c: -c["_judge_score"])
        need = floor - len(have)
        if len(pool) < need:
            log.warning("persona %s short: %d/%d", slug, len(have) + len(pool), floor)
        selected += pool[:need]           # backfill, still diversity-checked
    return selected

Log the shortfall rather than silently accepting it. An ICP that can't reach its floor means the capsule lacks evidence for that segment — a data problem to fix upstream, not an output to quietly ship. Surfacing it is how the persona axis stays honest instead of decorative.

Backfilled prompts still pass through the diversity check; the floor is a lower bound on coverage, not a licence to ship near-duplicates.


Output contract — match the eval service exactly

Required fields, from existing production records:

Topic · Prompt · Tags · Regions · Language · Platforms · Personas · Analysis Types

Checks before handoff:

  • Does the service reject unknown keys? Archetype and source_evidence are internal. Keep them for validation, strip before handoff unless the service tolerates extras. Cheap to verify, annoying to discover late.
  • Match Platforms to what's actually tracked. Production records say chatgpt, perplexity, google-ai-overviews. The master prompt currently adds claude. Fix unless Claude is tracked.
  • Fill Tags and Personas. Production ships them empty. Yours won't — visible improvement, no downside. Personas must match a persona name from the directives block, not free text, or the coverage check can't verify anything.
  • Hard floor of 10 prompts. The brief says "at least ten." Degradation reallocates per archetype but doesn't guarantee a total. Add a floor so archetypes with grounded slots absorb the reallocation.

Build order

Build the capsule before the synthesis call. The temptation is to start with the model call because output appears immediately — but then you're tuning prompt quality against fake context and you'll tune the wrong things. Get one real capsule from real sources first, even half-empty. What it actually looks like will change your synthesis decisions.

  1. Adapters + normalization → raw_signals. Largest chunk. Include the persona column from the start — retrofitting attribution after adapters exist means re-running every extraction.
  2. Capsule builder + coverage + directives. Verify the no-network invariant.
  3. Synthesis + schema check. End-to-end prompts out. Demoable.
  4. Full validation cascade. Stages 2 and 4 are what make output visibly good.
  5. Coverage → quota reallocation and persona floors. Demo by toggling sources off; toggle personas off separately to show the axis is independent.
  6. Golden capsule snapshots for regression.

Access limits — state these up front

Four of six slots run live without client credentials:

Slot Status
identity Live — public HTTP
capabilities Live — public HTTP
personas Live — public HTTP
pain_points Live — public GitHub repo API
search_intent Fixture — gsc_sample.json
competitive_landscape Fixture — ahrefs_sample.json

Fixtures must carry real field shapes (query, impressions, clicks, ctr, position) so the gap-pull filter actually executes rather than being described.

Stated up front it reads as scoping. Discovered by the reviewer it reads as a gap.


Demo set

Vercel — public repo, docs, pricing. Three adapters live, two fixture. Full capsule, all archetypes.

Nike — no repo, no docs site, no tracker. pain_points falls through to public forum search. Coverage shows the gap, quotas reallocate, Implicit ships reduced rather than fabricating.

The Nike case is the better demo. It exercises degradation under real conditions rather than a toggle, and proves the system isn't tuned to developer tooling.


Honest limits

Say these; don't let them be discovered.

  • Public-source parsing is brittle. llms.txt adoption is low, homepages are JS-rendered, pricing pages share no structure. Largest chunk of unglamorous work; expect flaky rather than solved.
  • Thresholds are starting values, not derived. Cosine cutoff, candidate counts, symptom counts. Say so rather than defending an invented constant.
  • Real GSC volume untested. The filter runs correctly on fixtures. Whether it surfaces good queries at real volume and CTR distribution is unknown.
  • Ahrefs competitor quality untested. Hand-picked fixtures flatter the system; real output is noisier.
  • Benchmark not run. The existing pipeline is both the incumbent and the labeled eval set — measure topic-coverage overlap, survival rate, and novel-prompt rate against it. This is the gating item for the replacement claim, and it needs internal access. Until it runs, say "designed to replace, pending validation." Migration is parallel-run: ship the fast set immediately, retire the slow pipeline per-client once thresholds clear, so a shortfall costs nothing.

Deliverable checklist

  • Working system, end to end
  • The actual generated prompts, with source_evidence on each — the brief asks for prompts, not just a generator. This is the part that gets read closely.
  • Design doc
  • Access limits stated up front
  • Both demo companies runnable