Skip to content

feat(cli): comprehensive indexer logging#136

Merged
prosdev merged 2 commits into
mainfrom
feat/scanner-logging
Dec 10, 2025
Merged

feat(cli): comprehensive indexer logging#136
prosdev merged 2 commits into
mainfrom
feat/scanner-logging

Conversation

@prosdev

@prosdev prosdev commented Dec 10, 2025

Copy link
Copy Markdown
Contributor

Summary

Add comprehensive logging to all indexers (code scanner, git history, GitHub issues) for better visibility during indexing operations.

Changes

Code Scanner & Indexer

  • Add logger option to ScanOptions and IndexOptions
  • Log per-language file counts during discovery
  • Log scan progress per language (start/complete)
  • Log embedding progress every 10 batches
  • Spinner shows: Embedding 4480/49151 documents (9%)

Git Indexer

  • Add logger option to GitIndexOptions
  • Log commit extraction count
  • Log embedding progress every 10 batches
  • dev git index --verbose for debug output
  • Spinner shows: Embedding 320/500 commits (64%)

GitHub Indexer

  • Add logger and onProgress to GitHubIndexOptions
  • Log document fetch count and type breakdown
  • dev gh index --verbose for debug output
  • Spinner shows: Embedding 50/150 GitHub docs

Go-Specific Exclusions

  • *.pb.go (protobuf), *.gen.go (generators)
  • mock_*.go, mocks/, testdata/

Example Output

[scanner] Starting repository scan {repoRoot: "."}
[scanner] Found 4338 go files {language: "go", files: 4338}
⠧ Embedding 4480/49151 documents (9%)

[git-indexer] Extracted commits {commits: 500}
⠧ Embedding 320/500 commits (64%)

[github-indexer] Fetched GitHub documents {documents: 150}
[github-indexer] Document breakdown {issues: 100, prs: 50}
⠧ Embedding 50/150 GitHub docs

Testing

  • 1552 tests passing
  • Tested on large Go codebase (~4k Go files, 49k documents)

Logging:
- Add logger option to ScanOptions and IndexOptions
- Log per-language file counts during discovery
- Log scan progress per language
- Log embedding progress every 10 batches
- Pass logger from CLI (--verbose enables debug level)

Go Exclusions (default):
- *.pb.go (protobuf)
- *.gen.go, *_gen.go (code generators)
- *.pb.gw.go (gRPC gateway)
- mock_*.go (mockgen)
- mocks/ (mock directories)
- testdata/ (test fixtures)

Spinner UX:
- Show 'Embedding X/Y documents' during storing phase
- Added totalDocuments to IndexProgress for accurate counts

Tests updated for new type imports.
- GitIndexer: add logger option with progress logging
- GitHubIndexer: add logger and onProgress callback
- CLI git/gh commands: add --verbose flag
- Spinner shows progress: 'Embedding X/Y commits/docs (N%)'

Example output:
[git-indexer] Starting git commit extraction {limit: 500}
[git-indexer] Extracted commits {commits: 500}
[git-indexer] Embedded 320/500 commits {batch: 10}
[github-indexer] Fetched GitHub documents {documents: 150}
[github-indexer] Document breakdown {issues: 100, prs: 50}
@prosdev prosdev merged commit 74b721b into main Dec 10, 2025
1 check passed
@prosdev prosdev deleted the feat/scanner-logging branch December 10, 2025 21:32
@prosdev prosdev mentioned this pull request Dec 10, 2025
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant