Neo object IDs

[asobolev]

The main idea is to have IOs to always return objects with the same identifiers when while reading the same file.

This would make possible to also extend IO libraries (or any Neo backend) with new operations like update, delete etc. E.g. it would make possible to save Neo objects to the file or DB without duplicating the whole structure every time.

Technically this could be achieved by creating UUID for every Neo object. But this question is open. There could be different more or less complicated strategies.

[toddrjen]

I am unclear how this would help with "update, delete etc.". What, specifically, would IO classes do with the ID information if they had it?

What do you mean be "same file"? Same file name? Same file path? Same file contents?

Under what situations, if any, should the ID change? I see a lot of corner cases related to this.

If you open a file, modify it, then save it to a new path, would it have the same ID or a different ID? If it has the same ID, what happens if you open the original and modify it, too? They would have the same ID but incompatible contents.

If you save the same file using multiple output classes, then open them again, would they have the same ID or different IDs? What if some of the IO classes lose data, like Matlab IO?

[rproepp]

These IDs would be useful for more complex storage backends like databases and HDF5. Then you could, for example, load a spike train, change some of its content and then call something like io.update(my_spike_train) to save the changes. Such IOs would effectively implement features of object relational mappers.

While truly global IDs would be nice to have, I don't think that's possible for Neo. IDs would be optional (for most existing IOs it makes little sense to support them) and valid while using the same backend (database or file contents, no matter what their respective address/path is). They should not change, except maybe when switching backends.

This does not solve the question of modified copies. I don't have a solution for that, I think they would simply become mutually incompatible and should not be used together. However, this would only affect features that do not exist in current IOs anyway.

[toddrjen]

So I see three main ways to implement this:

1. Path-based: the ID would be based on the absolute file path and the path of the object within the resulting neo object hierarchy at the time of initial import. The simplest way to do this would probably to make strings then run it through hashlib to create a unique hash. It would also be harder to do after import time since you can't guarantee there will still be a path stored at all, not to mention a long enough path to avoid collisions.
2. Content-based: there would be a hash made of the object contents at import time using hashlib. This means objects with identical contents have identical hashes, which may or may not be a good thing.
3. Time-based: the precise import time of the objects would be calculated, and it would either be used as the ID directly or be used as the input to a hash. This has the disadvantage that it is platform-dependent: not all platforms provide sufficient precision to be able to differentiate multiple objects imported in the same operation (like the children of a block or segment). It also can't be run on objects after import.
4. ID-based: each python object is given a unique ID, which is its memory location. This can be obtained with the id function. So at import time, the ID of the object is obtained and stored. Of course this would only be done the first time, since the memory address is not persistent across sessions. The problem with this approach is that there is a largeer chance of collisions compared to the previous methods, since the same memory address can be used at different points in the same session or different sessions, and doesn't produce a very long ID. You can hash together the ID of the object and its contents, but this will not work well when all attributes are None (since all the None values will have the same memory address on sane python setups). This may be a rare enough situation that it isn't worth worrying about, however.
5. Some hybrid approach. Combine more than one of these methods, and feed them all into hashlib. For example, if you hash the contents of an object, and the memory address of the object, and the memory address of each of the object contents, you will get something that is essentially guaranteed to be unique but does not depend on particular attributes being defined. The disadvantage here is that the hash will take longer to computer

[rproepp]

What about GUIDs? Pretty standard and practically no chance of collision, but takes 16 bytes. They would be generated when an IO supporting identifiers first stores an object. The current NeoHdf5IO actually uses them already for linking the non-tree parts of the hierarchy.

[toddrjen]

@rproepp Good idea. I wasn't aware of the uuid module but it seems like it would be the simplest solution. If you use uuid4, it should be random so you could probably just slice or hash the bytes down to whatever size you want and still should avoid collisions to whatever certainty you need.

[samuelgarcia]

Lets close this for now. Feel free to repopen something similar.