Skip to content

Embedding dedup hash uses only first 60 chars - updated content goes stale silently #2

Description

@jiya2401
  • embed_and_store() computes the dedup key as sha256(f"{url}|{i}|{chunk[:60]}") - only the first 60 characters of each chunk are hashed, not the full text.
  • If a scraped blog post is edited but the first 60 characters of a chunk stay the same, the new content is wrongly treated as a duplicate and skipped.
  • Result: the stale embedding stays in ChromaDB indefinitely, and your RAG pipeline analyzes outdated competitor data without you knowing.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions