Skip to content

Latest commit

 

History

History
391 lines (316 loc) · 13.6 KB

implementation-plan.md

File metadata and controls

391 lines (316 loc) · 13.6 KB

Technical Implementation Plan

Build target: a system that takes a client domain and returns 10–20 grounded, ICP-spread prompts in under 30 seconds.

Stack assumption: Python 3.11+, Postgres, async throughout. Swap freely — the structure holds.


Module layout

promptgen/
  adapters/
    base.py            # Adapter ABC + Signal dataclass
    homepage.py        # identity        — live
    docs_tree.py       # capabilities    — live (tech)
    catalog_nav.py     # capabilities    — live (non-tech)
    github_issues.py   # pain_points     — live (tech)
    public_reviews.py  # pain_points     — live (non-tech)
    pricing.py         # personas        — live (both)
    gsc.py             # search_intent   — fixture, real interface
    ahrefs.py          # competitive     — fixture, real interface
    registry.py        # vertical detection → adapter selection
  capsule/
    builder.py         # signals → capsule XML + coverage
    directives.py      # coverage → archetype quotas + ICP floors
    render.py          # XML serialization
  synthesis/
    prompt.py          # static prefix loader
    generate.py        # parallel archetype calls, streaming
  validation/
    schema.py          # stage 1
    brand_leak.py      # stage 2
    judge.py           # stage 3
    setlevel.py        # stage 4 — diversity + ICP coverage
  store/
    models.py          # SQLAlchemy
    repo.py            # queries
  cli.py               # promptgen run <domain>
fixtures/
  gsc/<domain>.json
  ahrefs/<domain>.json
tests/
  golden/              # snapshot capsules

Core contracts

Define these first. Everything else depends on them, and changing them later is expensive.

# adapters/base.py
@dataclass
class Signal:
    text: str                      # the extracted string, verbatim
    source: str                    # "github" | "gsc" | "pricing" | ...
    kind: Literal["identity", "capability", "query",
                  "pain", "competitor", "persona"]
    ref: str                       # "issue:8891" — provenance
    weight: int                    # 1–10, normalized by the adapter
    persona: str | None = None     # ICP slug, when attributable
    meta: dict = field(default_factory=dict)

class Adapter(ABC):
    slot: str
    @abstractmethod
    async def fetch(self, client: Client) -> list[Signal]: ...
# capsule/builder.py
@dataclass
class Persona:
    slug: str
    label: str
    vocabulary: list[str]          # ["SSO", "RBAC", "audit log"]
    gated_capabilities: list[str]
    price_tier: Literal["free", "low", "mid", "enterprise"]
    evidence_weight: int

@dataclass
class Capsule:
    xml: str
    coverage: dict[str, int]
    personas: list[Persona]
    version: int

Every generated prompt carries the ref of the Signal it came from. That's what makes grounding checkable by string match instead of trust.


Schema

CREATE TABLE clients (
  id BIGSERIAL PRIMARY KEY,
  domain TEXT UNIQUE NOT NULL,
  name TEXT,
  vertical TEXT,                        -- 'tech' | 'consumer'
  sources JSONB DEFAULT '{}',
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE raw_signals (
  id BIGSERIAL PRIMARY KEY,
  client_id BIGINT REFERENCES clients(id),
  source TEXT NOT NULL,
  kind TEXT NOT NULL,
  text TEXT NOT NULL,
  ref TEXT,
  weight INT DEFAULT 5,
  persona TEXT,                         -- nullable; NULL is the common case
  meta JSONB DEFAULT '{}',
  fetched_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX ON raw_signals (client_id, kind, weight DESC);

CREATE TABLE personas (
  id BIGSERIAL PRIMARY KEY,
  client_id BIGINT REFERENCES clients(id),
  slug TEXT NOT NULL,
  label TEXT NOT NULL,
  vocabulary JSONB DEFAULT '[]',
  gated_capabilities JSONB DEFAULT '[]',
  price_tier TEXT,
  evidence_weight INT DEFAULT 1,
  source TEXT,
  UNIQUE (client_id, slug)
);

CREATE TABLE capsules (
  id BIGSERIAL PRIMARY KEY,
  client_id BIGINT REFERENCES clients(id),
  version INT NOT NULL,
  xml TEXT NOT NULL,
  coverage JSONB,
  directives JSONB,
  built_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE generated_prompts (
  id BIGSERIAL PRIMARY KEY,
  capsule_id BIGINT REFERENCES capsules(id),
  payload JSONB NOT NULL,               -- the eval-service record
  archetype TEXT,
  persona TEXT,
  source_ref TEXT,                      -- Signal.ref it derived from
  judge_score REAL,
  shipped BOOLEAN DEFAULT false
);

raw_signals.persona must exist in the first migration. Adding it later means re-running every extraction to backfill attribution.

The invariant: build_capsule() runs to completion with the network disabled. Enforce it in tests. If it fails, something is fetching where it should be reading cache.


Pipeline

domain
  → detect vertical            (probe /docs, /mint.json, GitHub org)
  → select adapters            registry.pick(vertical, available_creds)
  → fetch concurrently         asyncio.gather, per-adapter timeout
  → normalize → raw_signals
  → derive personas            pricing gates → GSC landing split → docs IA
  → build capsule + coverage
  → build directives           quotas × ICP floors
  → generate                   4 parallel calls, one per archetype
  → validate                   streaming stages 1–3, then 4
  → emit                       eval-service records

Vertical detection

async def detect(domain: str) -> str:
    docs, repo = await asyncio.gather(
        probe_any(f"https://{domain}/docs",
                  f"https://{domain}/mint.json",
                  f"https://docs.{domain}"),
        github_org_for(domain),
    )
    return "tech" if (docs or repo) else "consumer"

Adapter selection follows from it; nothing downstream branches on vertical.

Persona derivation

async def derive_personas(client, signals) -> list[Persona]:
    personas = personas_from_pricing(await fetch_pricing(client.domain))
    if len(personas) < 2:
        personas += personas_from_gsc_landing(signals)     # /enterprise vs /docs
    if len(personas) < 2:
        personas += personas_from_docs_ia(signals)
    if not personas:
        personas = [generic_persona(client)]               # FLAG in output
    return dedupe(personas)

personas_from_pricing is the important one:

def personas_from_pricing(html) -> list[Persona]:
    tiers = parse_tiers(html)               # name, price, feature list
    base  = set(tiers[0].features)
    out = []
    for t in tiers:
        gated = set(t.features) - base      # what THIS tier unlocks
        out.append(Persona(
            slug=slugify(t.name),
            label=t.name,
            vocabulary=extract_terms(gated),        # SSO, RBAC, audit log
            gated_capabilities=sorted(gated),
            price_tier=classify_price(t.price),     # → archetype weighting
            evidence_weight=3,
        ))
    return out

One parse yields segment, vocabulary, capability mapping, and price tier. Price tier feeds ARCHETYPE_MIX — higher price means more comparison behavior, so more Explicit and Multi-Turn.

Directives

ARCHETYPE_MIX = {
  "free":       {"Implicit": .35, "Command": .35, "Explicit": .20, "Multi-Turn": .10},
  "low":        {"Implicit": .30, "Command": .30, "Explicit": .25, "Multi-Turn": .15},
  "mid":        {"Implicit": .20, "Command": .25, "Explicit": .30, "Multi-Turn": .25},
  "enterprise": {"Implicit": .10, "Command": .15, "Explicit": .40, "Multi-Turn": .35},
}

def build_directives(personas, coverage, total=20):
    quotas = archetype_quotas(coverage, total)      # degradation logic
    if not personas:
        return {"quotas": quotas, "floors": {}, "candidates_per_archetype": 10}
    w = sum(p.evidence_weight for p in personas)
    floors = {p.slug: max(2, round(total * 0.7 * p.evidence_weight / w))
              for p in personas}
    return {"quotas": quotas, "floors": floors,
            # emitted as skew="Explicit,Multi-Turn" on each <persona> —
            # the two highest-weighted archetypes for that price tier
            "skew": {p.slug: top2(ARCHETYPE_MIX[p.price_tier]) for p in personas},
            "candidates_per_archetype": 10}

0.7 leaves ~30% unfloored so quality decides the remainder — floors are a lower bound, not a partition. max(2, ...) stops a thin ICP rounding to zero. Both are starting values.

skew is a hint the model may ignore. Check the first run: if the distribution comes back flat, stop hinting and request explicit per-cell counts (persona × archetype) instead of trusting it.

Synthesis — parallel, streaming

Synthesis dominates latency (8–20s for one call). Split it.

async def generate(capsule, directives):
    tasks = [call_model(STATIC_PREFIX, capsule.xml, directives, arch)
             for arch in directives["quotas"]]
    async for done in asyncio.as_completed(tasks):
        for cand in parse_json(await done):
            yield cand

Four concurrent calls: wall time is the slowest (~6s), not the sum. Same static prefix on each, so prefix caching still applies.

Validation — greedy, streaming

Global diversity would block until all 40 candidates land. Greedy ships as they arrive.

async def run(capsule, directives):
    shipped, vecs, leftovers = [], [], []
    async for c in generate(capsule, directives):
        if not schema_ok(c) or brand_leak(c, capsule): 
            continue
        if not await judge(c):
            continue
        v = await embed(c["Prompt"])
        if any(cosine(v, s) > 0.85 for s in vecs):
            leftovers.append(c); continue
        shipped.append(c); vecs.append(v)
        yield c                                    # renders immediately
    for c in enforce_floors(shipped, leftovers, directives["floors"]):
        yield c

Greedy gives a marginally worse final set than global selection. Worth it: first prompt at ~3s instead of a spinner until 20s.

UNBRANDED = {"Command", "Implicit", "Multi-Turn"}

def brand_leak(c, capsule):
    if c["Archetype"] not in UNBRANDED:
        return False                     # Explicit names the brand by design
    terms = capsule.brand_terms          # client name + proprietary product names
    return any(re.search(rf"\b{re.escape(t)}\b", c["Prompt"], re.I) for t in terms)

Regex, never a model. This is the most visible failure mode and the cheapest to prevent.

Three archetypes, not one. Only Explicit is allowed to name the client, because it's the only one carrying Sentiment. Command and Multi-Turn measure Visibility, and an engine told to use a product will use it.

brand_terms excludes two things that a naive build would wrongly include:

  • Category nouns ("hosting platform", "agent framework"). Implicit prompts need these — they're what force the engine to return products instead of a tutorial.
  • Open-source projects the client maintains that run anywhere (Next.js for Vercel). Real users name their own stack, and stripping it makes prompts sound artificial.

So brand_terms = client name + aliases + proprietary product names. Nothing else.


Output

Match the eval service exactly. Verify before building whether it rejects unknown keys.

FIELDS = ["Topic","Prompt","Tags","Regions","Language",
          "Platforms","Personas","Analysis Types"]

def to_record(c, capsule):
    r = {k: c[k] for k in FIELDS}
    r["Platforms"] = "chatgpt, perplexity, google-ai-overviews"   # confirm Claude
    r["Personas"]  = c["Personas"]        # must match a persona slug
    return r                              # source_ref kept internally

Hard floor: never emit fewer than 10.


Latency budget

Stage Target
Vertical detect + parallel fetch 1.5s
Capsule + personas + directives 0.2s
First archetype call returns 3s
First prompt rendered ≤3.5s
Ten prompts ≤12s
Full validated set ≤25s

Warm capsule skips the fetch entirely. Cold start with no credentials still hits phase 1 on public sources alone.


Build sequence

Day 1 — foundation Schema and migrations, including raw_signals.persona. Signal and Adapter contracts. Homepage, docs tree, and pricing adapters. CLI that dumps normalized signals for a domain. Done when: promptgen signals vercel.com prints real rows.

Day 2 — capsule Builder with per-slot caps, coverage, XML render. personas_from_pricing. Golden snapshot test. Network-disabled invariant test. Done when: promptgen capsule mastra.ai prints valid XML with ≥2 personas.

Day 3 — synthesis Static prefix from master-prompt.md, directives builder, parallel archetype calls, schema check. Done when: 20 prompts out, end to end.

Day 4 — validation Brand-leak regex, judge, greedy diversity, ICP floors with backfill and shortfall logging. Done when: zero brand leaks in Implicit and every ICP hits its floor.

Day 5 — breadth + demo GitHub issues and public reviews adapters, GSC/Ahrefs fixtures, vertical detection. Run Vercel and Nike. Export the prompt sets with provenance. Done when: both demos run from a cold CLI invocation.


Test checklist

  • build_capsule() passes with network disabled
  • Golden capsule snapshots for both demo clients
  • Zero brand leaks across 100 generated Implicit prompts
  • Every persona meets its floor, or the shortfall is logged
  • Empty pain_points degrades to fewer Implicit rather than fabricating
  • Output never drops below 10 prompts
  • Every prompt's source_ref resolves to a real raw_signals row

Verify before writing code

  1. Does the eval service reject unknown JSON keys?
  2. Is Claude a tracked platform, or only the three in production records?
  3. Does the demo client's pricing page render server-side, or is it JS-only?