Check the possibility of Persistent MLC index

[arjunsuresh]

Currently the MLC indexes are recreated when we start an MLC command. It'll be good to do a persistent index to speed up the execution of multiple MLC commands in sequence.

[Sid9993]

I would like to provide some suggestions for addressing this issue:

Problem Statement:
If we keep the MLC indexes(index_script.json) persistent then it should also handle the situations when we make any kind of changes in the script's tags, paths etc. it should also get reflected in the already generated index_script.json.

Suggested Solutions

1. Using Persistent MLC Index with Timestamp Validation:

Store a last-modified timestamp for each index in the index_script.json.
Before using the persistent index, validate if the stored timestamps match the actual file system timestamps.
If there's a mismatch, regenerate the index. If not, use the already generated index.
In this way, I believe that we could avoid regenerating every index and instead, we could modify only the changed index.

2. Incremental Index Updates:

Instead of recreating the entire index, detect changes (additions, deletions, or modifications) in the directory by tracking the file changes from the system and only update the affected parts of the index_script.json.
This minimizes processing overhead while maintaining consistency.

[arjunsuresh]

@Sid9993 These are good suggestions. I believe the suggested solution 1 is the way to go. Would you like to give a try?

[Sid9993]

@Sid9993 These are good suggestions. I believe the suggested solution 1 is the way to go. Would you like to give a try?

Yeah, ofc. I would like to give it a try.