Proposal: keep multimodal support as a sibling plugin to memory-lancedb-pro

[gexinggexing]

Following up from issue #272, I think the cleanest upstream path is to keep multimodal support as a sibling plugin first instead of turning memory-lancedb-pro into an all-in-one runtime.

Why this split makes sense:

• the current core plugin is still centered on text memory lifecycle and recall
• multimodal ingest introduces different file, chunking, metadata, and storage concerns
• a sibling plugin keeps migration reversible and reduces regression risk for existing users

Suggested split

Plugin A: memory-lancedb-pro

• text memory lifecycle hooks
• smart extraction from conversations
• recall injection
• scoring / rerank / decay / scope isolation
• CLI and maintenance commands

Plugin B: memory-multimodal-ingest

• image / audio / video / PDF ingest
• MIME detection and file normalization
• media chunking
• embedding generation via Gemini Embedding 2
• writing media records into LanceDB
• optional blob sidecar storage

Storage recommendation

• start with separate tables / paths for text and multimodal embeddings
• unify retrieval later only if cross-modal ranking needs a single scoring pipeline

Retrieval recommendation

• keep the core memory runtime as the broker
• query text memory and multimodal memory in parallel
• normalize scores and rerank before context injection

Local validation I already completed:

• Gemini Embedding 2 accepted image / audio / video / PDF inputs
• a standalone memory-multimodal-ingest plugin MVP worked end-to-end with LanceDB-backed ingest and text-query search over stored media rows

Prototype repo:

• https://github.com/gexinggexing/memory-multimodal-ingest

After diffing the current upstream master, the smaller core changes I originally planned to PR now appear to already be present there, so the main remaining question is the multimodal architecture boundary rather than the Gemini Embedding 2 core compatibility work.

If this direction sounds reasonable, I can turn the MVP into a cleaner upstream-facing proposal with explicit integration points and migration stages.

[gexinggexing]

Thanks, that makes sense.

I agree that linking an early-stage external prototype in the main README was too early. I have stopped pushing that route and will keep the multimodal work under discussion first.

I am also tightening the prototype repo wording so it is clearly positioned as:

• an experimental sibling-plugin prototype
• independent from the core memory-lancedb-pro runtime
• not an official upstream companion path at this stage
• something that should wait for consensus here before any further upstream docs placement

So from here I will use #275 as the main place to converge on the boundary:

• what stays in memory-lancedb-pro
• what belongs in a sibling multimodal ingest plugin
• what the eventual retrieval broker / integration surface should look like

Once there is clearer consensus in #275, I can come back with a more appropriate follow-up proposal.

[gexinggexing]

Small update on my side: I have now tightened the prototype repo wording to match this discussion more closely.

Specifically, the repo now states that it is:

• experimental
• independent from the core memory-lancedb-pro runtime
• not an official upstream companion path at this stage
• something that should wait for consensus in #275 before any further upstream docs placement

Latest repo update: gexinggexing/memory-multimodal-ingest@4bb74dd

So I will keep using this discussion as the main place to converge on the boundary before proposing anything upstream again.