fastmultigather against rocksdb gives "No such file or directory (os error 2)"

[AnneliektH]

Trying to run fastmultigather against a rocksdb in
/group/ctbrowngrp2/scratch/annie/2023-swine-sra/sourmash/viral_taxonomy/genbank/sc100rocksdb

When running against .zip db, command works and I get results.
rocksdb is present, so is the file with paths to the sig.zip.

For rocksdb:

input:
sourmash scripts fastmultigather \
../../votu_sigs_s100.txt \
 /home/ntpierce/2023-spillover/output.genbank-viral/genbank.2023-05.viral.dna-k21-sc100.rocksdb \
-c 5 -k 21 -t 100 -s 100

output:
== This is sourmash version 4.8.6. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

=> sourmash_plugin_branchwater 0.9.1; cite Irber et al., doi: 10.1101/2022.11.02.514947

ksize: 21 / scaled: 100 / moltype: DNA / threshold bp: 100.0
gathering all sketches in '../../votu_sigs_s100.txt' against '/home/ntpierce/2023-spillover/output.genbank-viral/genbank.2023-05.viral.dna-k21-sc100.rocksdb' using 5 threads
Error: No such file or directory (os error 2)

[bluegenes]

Could you try running this from within the same directory as your rocksdb database (so you don't need to pass in a full path)? I'm curious if this is related to #206

[ctb]

it should also be possible to symlink (ln -s) the rocksdb into your current directory -

ln -s /home/ntpierce/2023-spillover/output.genbank-viral/genbank.2023-05.viral.dna-k21-sc100.rocksdb .

and then use the path genbank.2023-05.viral.dna-k21-sc100.rocksdb directly.

[AnneliektH]

Both those options work!

[tnmquann]

it should also be possible to symlink (ln -s) the rocksdb into your current directory -

ln -s /home/ntpierce/2023-spillover/output.genbank-viral/genbank.2023-05.viral.dna-k21-sc100.rocksdb .

and then use the path genbank.2023-05.viral.dna-k21-sc100.rocksdb directly.

Hi @ctb ,
In version 0.9.5, I followed your instructions, but it seems they no longer work. Currently, the only way to get things working seems to be by directly indexing the database into the processing folder (based on @bluegenes 's suggestion). Could you please help me identify the problem in this situation?
Thanks

[ctb]

@tnmquann could you let us know what commands you are running and what errors you are seeing? We haven't changed anything intentionally in this area.

I wrote a brief tutorial here:

https://hackmd.io/18cdFFciTIa6bDY2S111OQ?both

and if you can give us similar copy/paste commands to reproduce your error, that would help greatly!

Note that I tested a few things successfully -

mkdir subdir
cd subdir
cp ../fake-metag* .

# run against rocksdb in a relative path
sourmash scripts manysearch fake-metag.list ../first-10.rocksdb -o xyz.csv

# run against rocksdb in an absolute path
sourmash scripts manysearch fake-metag.list /Users/t/dev/sourmash/rocks-index/first-10.rocksdb -o xyz2.csv

# run against rocksdb with a '/' at the end
sourmash scripts manysearch fake-metag.list /Users/t/dev/sourmash/rocks-index/first-10.rocksdb/ -o xyz3.csv

and all three worked. So it does not appear to me as if the path of the rocksdb index is a problem in v0.9.5.

thanks!

[ctb]

Addendum: re-reading the initial issue, I am wondering if perhaps the paths that @AnneliektH was using in ../../votu_sigs_s100.txt were not valid in the subdirectory location she was in? The paths in the pathlist file are loaded relative to the current directory, not the directory the pathlist is in.

[ctb]

So, TODO:

• write some tests against rocksdb being created in different directories and/or used from a different directory
• improve error message so we print out what file could not be found (in top error message)

[tnmquann]

Hi @ctb,
I created issue #381 for better describe the issue. Hopefully the problem can be resolved because the approach with rocksdb is really impressive :)

[ctb]

closing in favor of #415