har -> corpus (#54)

Author: nedtwigg

.claude/agents/corpus-fixer.md

@@ -0,0 +1,39 @@
+---
+name: corpus-fixer
+description: Use this agent when you need to fix or improve the detection logic for a specific Gitcasso corpus by testing changes in the corpus:view development environment. Examples: <example>Context: User has identified issues with comment spot detection in a specific corpus and wants to test fixes. user: 'The comment detection is missing some spots in corpus ABC123, can you help fix the enhancer logic?' assistant: 'I'll use the corpus-fixer agent to investigate and fix the detection issues in that corpus.' <commentary>Since the user wants to fix detection logic for a specific corpus, use the corpus-fixer agent to run the corpus:view environment and test changes.</commentary></example> <example>Context: User wants to validate that recent changes to an enhancer are working correctly. user: 'I made some changes to the GitHub enhancer, can you test it against corpus XYZ789?' assistant: 'Let me use the corpus-fixer agent to test your enhancer changes against that specific corpus.' <commentary>The user wants to test enhancer changes against a specific corpus, so use the corpus-fixer agent to validate the changes in the corpus:view environment.</commentary></example>
+model: inherit
+---
+
+You are an expert Gitcasso corpus debugging specialist with deep knowledge of browser extension development. You operate exclusively within the `browser-extension` directory and specialize in using the corpus:view development environment to diagnose and fix detection logic issues.
+
+Your primary workflow:
+
+1. **Environment Setup**: Always start by reading the documentation at the top of the `corpus-view.ts` file to understand the dev environment.
+
+2. **Launch Development Environment**: Execute `pnpm corpus:view` to bring up the corpus:view development environment. Ensure the environment starts successfully before proceeding.
+
+3. **Browser Navigation**: Use the Playwright MCP to interact with the development environment. Navigate to the specific Gitcasso corpus that needs investigation or fixing.
+
+4. **Code Synchronization**: Always click the button with id `gitcasso-rebuild-btn` to ensure you're testing against the latest code changes. Wait for the rebuild to complete before analyzing results.
+
+5. **Detection Analysis**: Examine the detected spots in the `gitcasso-comment-spots` element. Analyze what spots are being detected, what might be missing, and identify patterns in the detection logic that need improvement.
+
+6. **Enhancer Modification**: Based on your analysis, make targeted changes to the specific enhancer's detection logic. Focus on:
+   - Improving selector accuracy
+   - Handling edge cases in the DOM structure
+   - Optimizing detection algorithms for the specific site pattern
+   - Ensuring compatibility with dynamic content loading
+
+7. **Iterative Testing**: After making changes, rebuild and test again to validate improvements. Continue this cycle until the detection logic works correctly for the target corpus.
+
+8. **Documentation**: Clearly explain what issues you found, what changes you made, and why those changes improve the detection logic.
+
+Key principles:
+- Always work incrementally - make small, targeted changes and test frequently
+- Focus on the specific corpus mentioned by the user unless told otherwise
+- Pay attention to browser console errors and network issues that might affect detection
+- Consider how your changes might impact other sites or corpus entries
+- Be methodical in your debugging approach - document what you try and what results you observe
+- Understand that corpus can be either HAR files (for initial page loads) or HTML snapshots (for post-interaction states)
+
+You have expertise in CSS selectors, DOM manipulation, JavaScript debugging, and understanding how different websites structure their comment systems. Use this knowledge to create robust, reliable detection logic that works across various edge cases.

.claude/agents/har-fixer.md

@@ -1 +1,5 @@
-Run `git status` to see the changes in the working copy. Complete whatever tasks are necessary to complete this change. Make sure that `pnpm -r precommit` succeeds. Don't fix `precommit` just be reverting the changes, the goal is to complete the change.
+- run `git status` to see the changes in the working copy
+- **figure out the intent of the change**
+- complete whatever tasks are necessary to complete this change
+- make sure that `pnpm precommit` succeeds when you are done
+- don't fix `precommit` by reverting the changes, it is more important to complete the initial change

.claude/commands/finish-wc.md

@@ -69,17 +69,42 @@ When the `textarea` gets removed from the page, the `TextareaRegistry` is notifi
 ## Testing
 
 - `npm run playground` gives you a test environment where you can tinker with the popup with various test data, supports hot reload
-- `npm run har:view` gives you recordings of various web pages which you can see with and without enhancement by the browser extension
-
-### Recording new HAR files
-
-- the har recordings live in `tests/har`, they are complete recordings of the network requests of a single page load
-- you can add or change URLs in `tests/har-index.ts`
-- `npx playwright codegen https://github.com/login --save-storage=playwright/.auth/gh.json` will store new auth tokens
-  - login manually, then close the browser
-  - ***these cookies are very sensitive! we only run this script using a test account that has no permissions or memberships to anything, recommend you do the same!***
-- `pnpm run har:record` this records new snapshots using those auth tokens (it needs args, run it with no args for docs)
-  - DO NOT COMMIT AND PUSH NEW OR CHANGED `har` files!
-  - we try to sanitize these (see `har-record.ts` for details) but there may be important PII in them
-  - if you need new HAR files for something, let us know and we will generate them ourselves using a dummy account
-  - IF YOUR PR CHANGES OR ADDS HAR FILES WE WILL CLOSE IT. Ask for HAR files and we'll be happy to generate clean ones you can test against.
+- `npm run corpus:view` gives you recordings of various web pages which you can see with and without enhancement by the browser extension
+
+### Test Corpus
+
+We maintain a corpus of test pages in two formats for testing the browser extension:
+
+#### HAR Corpus (Automated)
+
+- For testing initial page loads and network requests
+- HAR recordings live in `tests/corpus/har/`, complete recordings of the network requests of a single page load
+- You can add or change URLs in `tests/corpus/_corpus-index.ts`
+- **Recording new HAR files:**
+  - `npx playwright codegen https://github.com/login --save-storage=playwright/.auth/gh.json` will store new auth tokens
+    - login manually, then close the browser
+    - ***these cookies are very sensitive! we only run this script using a test account that has no permissions or memberships to anything, recommend you do the same!***
+  - `pnpm run corpus:record:har` records new HAR files using those auth tokens (it needs args, run it with no args for docs)
+    - DO NOT COMMIT AND PUSH NEW OR CHANGED HAR files!
+    - we try to sanitize these (see `har-record.ts` for details) but there may be important PII in them
+    - if you need new HAR files for something, let us know and we will generate them ourselves using a dummy account
+    - IF YOUR PR CHANGES OR ADDS HAR FILES WE WILL CLOSE IT. Ask for HAR files and we'll be happy to generate clean ones you can test against.
+
+#### HTML Corpus (Manual)
+
+- For testing post-interaction states (e.g., expanded textareas, modal dialogs, dynamic content)
+- HTML snapshots live in `tests/corpus/html/`, manually captured using SingleFile browser extension
+- All assets are inlined in a single HTML file by SingleFile
+- **Creating new HTML corpus files:**
+  1. Navigate to the desired page state (click buttons, expand textareas, etc.)
+  2. Use SingleFile browser extension to save the complete page
+  3. Save the `.html` file to `tests/corpus/html/` with a descriptive name
+  4. Add an entry to `tests/corpus/_corpus-index.ts` with `type: 'html'` and a description of the captured state
+
+#### Viewing Corpus Files
+
+- Run `pnpm run corpus:view` to start the test server at http://localhost:3001
+- Select any corpus file to view in two modes:
+  - **Clean**: Original page without extension
+  - **Gitcasso**: Page with extension injected for testing
+- Both HAR and HTML corpus types are supported

browser-extension/README.md

@@ -62,8 +62,8 @@
     "test": "vitest run",
     "playground": "vite --config vite.playground.config.ts",
     "playground:build": "vite build --config vite.playground.config.ts",
-    "har:record": "tsx tests/har-record.ts",
-    "har:view": "tsx tests/har-view.ts"
+    "corpus:record:har": "tsx tests/har-record.ts",
+    "corpus:view": "tsx tests/corpus-view.ts"
   },
   "type": "module",
   "version": "0.0.1"

browser-extension/package.json

@@ -99,9 +99,7 @@ function enhanceMaybe(textarea: HTMLTextAreaElement) {
     return
   }
 
-  logger.debug('activating textarea {}', textarea)
   injectStyles()
-
   try {
     const location = detectLocation()
     logger.debug('[gitcasso] Calling tryToEnhance with location:', location)

browser-extension/src/entrypoints/content.ts

@@ -9,6 +9,6 @@ export type LogLevel = (typeof LOG_LEVELS)[number]
 export const CONFIG = {
   ADDED_OVERTYPE_CLASS: 'gitcasso-overtype',
   EXTENSION_NAME: 'gitcasso', // decorates logs
-  LOG_LEVEL: 'INFO' satisfies LogLevel,
+  LOG_LEVEL: 'DEBUG' satisfies LogLevel,
   MODE: 'PROD' satisfies ModeType,
 } as const

browser-extension/src/lib/config.ts

@@ -30,21 +30,21 @@ vi.mock('overtype', () => {
 })
 
 import { describe as baseDescribe, test as baseTest, expect } from 'vitest'
-import type { PAGES } from './har/_har-index'
-import { cleanupDOM, setupHarDOM } from './har-fixture-utils'
+import type { CORPUS } from './corpus/_corpus-index'
+import { cleanupDOM, setupDOM } from './corpus-utils'
 
 export const describe = baseDescribe
 
 // Re-export expect from vitest
 export { expect }
 
-// Fluent interface for HAR-based tests
-export function usingHar(harKey: keyof typeof PAGES) {
+// Fluent interface for any corpus type (HAR or HTML)
+export function forCorpus(corpusKey: keyof typeof CORPUS) {
   return {
     it: (name: string, fn: () => void | Promise<void>) => {
-      return baseTest(`${harKey}:${name}`, async () => {
-        // Setup HAR DOM before test
-        await setupHarDOM(harKey)
+      return baseTest(`${String(corpusKey)}:${name}`, async () => {
+        // Setup DOM for any corpus type (delegates to HAR or HTML based on type)
+        await setupDOM(corpusKey)
 
         try {
           return await fn()

browser-extension/tests/har-fixture.ts renamed to browser-extension/tests/corpus-fixture.ts

@@ -3,7 +3,7 @@ import path from 'node:path'
 import { fileURLToPath } from 'node:url'
 import type { Har as HarFile } from 'har-format'
 import { parseHTML } from 'linkedom'
-import { PAGES } from './har/_har-index'
+import { CORPUS } from './corpus/_corpus-index'
 
 const __dirname = path.dirname(fileURLToPath(import.meta.url))
 
@@ -32,21 +32,51 @@ export interface TestDOMContext {
 let currentDOMInstance: any = null
 let originalGlobals: Partial<TestDOMGlobals> = {}
 
-export async function loadHtmlFromHar(key: keyof typeof PAGES): Promise<string> {
-  const url = PAGES[key]
-  const harPath = path.join(__dirname, 'har', `${key}.har`)
+export async function setupDOM(key: keyof typeof CORPUS): Promise<TestDOMGlobals> {
+  const entry = CORPUS[key]
+  if (!entry) {
+    throw new Error(`Invalid corpus key: ${String(key)}`)
+  }
+
+  let html: string
+  if (entry.type === 'har') {
+    html = await loadRootHtmlStringFromHar(key)
+  } else if (entry.type === 'html') {
+    html = await loadHtmlStringFromHtml(key)
+  } else {
+    throw new Error(`Unsupported corpus type: ${entry.type}`)
+  }
+  const domGlobals = createDOMFromString(html, entry.url)
+  setupDOMFromHar(domGlobals)
+  return domGlobals
+}
+
+async function loadRootHtmlStringFromHar(key: keyof typeof CORPUS): Promise<string> {
+  const entry = CORPUS[key]
+  if (!entry || entry.type !== 'har') {
+    throw new Error(`Invalid HAR corpus key: ${String(key)}`)
+  }
+  const url = entry.url
+  const harPath = path.join(__dirname, 'corpus', `${String(key)}.har`)
   const harContent = await fs.readFile(harPath, 'utf-8')
   const harData: HarFile = JSON.parse(harContent)
   const mainEntry = harData.log.entries.find((entry) => entry.request.url === url)
-
   if (!mainEntry) {
     throw new Error(`No entry found for URL: ${url} in HAR file: ${harPath}`)
   }
+  return mainEntry.response.content.text!
+}
 
-  return mainEntry.response.content.text || ''
+async function loadHtmlStringFromHtml(key: keyof typeof CORPUS): Promise<string> {
+  const entry = CORPUS[key]
+  if (!entry || entry.type !== 'html') {
+    throw new Error(`Invalid HTML corpus key: ${String(key)}`)
+  }
+  const htmlPath = path.join(__dirname, 'corpus', `${String(key)}.html`)
+  return await fs.readFile(htmlPath, 'utf-8')
 }
 
-export function createDOMFromHar(html: string, url: string): TestDOMGlobals {
+function createDOMFromString(html: string, url: string): TestDOMGlobals {
   const dom = parseHTML(html)
 
   return {
@@ -68,7 +98,7 @@ export function createDOMFromHar(html: string, url: string): TestDOMGlobals {
   }
 }
 
-export function setupDOMFromHar(domGlobals: TestDOMGlobals): void {
+function setupDOMFromHar(domGlobals: TestDOMGlobals): void {
   // Store original globals for cleanup
   originalGlobals = {
     Document: (globalThis as any).Document,
@@ -100,11 +130,3 @@ export function cleanupDOM(): void {
     originalGlobals = {}
   }
 }
-
-export async function setupHarDOM(key: keyof typeof PAGES): Promise<TestDOMGlobals> {
-  const html = await loadHtmlFromHar(key)
-  const url = PAGES[key]
-  const domGlobals = createDOMFromHar(html, url)
-  setupDOMFromHar(domGlobals)
-  return domGlobals
-}