Vocabulary Serialization & Persistence

[lg-epitech]

Description

Implement the ability to export and reload the vocabulary state to/from a file. Since BPE is a computationally expensive algorithm, persisting the vocabulary state will significantly speed up development workflows by avoiding redundant recomputation.

This also lays the groundwork for a potential future feature: incremental BPE merges using git history.

Requirements

• Make the vocabulary data structure serializable/deserializable using Serde, the Rust standard for serialization
• Implement a way to export the current vocabulary state to a file
• Implement a way to load a vocabulary state back from a file

Notes

• Choose an appropriate file format (e.g. JSON for readability, or binary like MessagePack/Bincode for performance - your call, open to discussion)
• Keep the future incremental BPE use case in mind when designing the persistence format, as we may want to layer git history on top of this later

Deliverables

• Vocabulary struct implements Serialize / Deserialize via Serde
• Export vocabulary state to file
• Load vocabulary state from file
• Brief note on chosen format and rationale

[linear-code]

DET-9