shelve dataset loading is throwing an error

[bibbygoodwin]

Many thanks for making your code publicly available!

An issue arises for me on line 373 of the ddpg agent code. The line uses shelve to open an alignment dataset (pickle file) as:

dail/agents/ddpg.py

Line 373 in 72a7089

hybrid_dataset = shelve.open(self.load_dataset_dir, writeback=True)

Unfortunately, for me this results in

dbm.error: db type is dbm.gnu, but the module is not available

I am using Python 3.7.9, OSX, with all requirements.txt installed. Is it possible that the pickle files are in some way not cross-platform? Certainly I'm also unable to open these datasets with pickle itself (I get _pickle.UnpicklingError: invalid load key, "'\xce'").

Any advice would be greatly appreciated!

[bibbygoodwin]

Update: I thought this was an issue with me not having gdbm (which is the same library that dbm.gnu refers to) installed, but it seems I do. Having said that, I can isolate the problem a little - simply running import dbm.gnu throws an error.

Update 2: It seems that getting gdbm installs to play with conda environment installations of dbm is a known difficulty. On OSX, gdbm is usually installed with Homebrew. I found that I could use the Homebrew python3 to open these pickle files, and save them in a standard pickle format rather than one requiring shelve. This also made them about 1/3 the size of the original datasets from GDrive. The shelve-based loading lines in ddpg then just need to be replaced with the equivalent pickle command e.g.

        with open(self.load_dataset_dir, 'rb') as input_file:
            hybrid_dataset = pkl.load(input_file)