Question on the error: v1 and v2 hashes do not describe the same data

[a-s-tarantoga]

Hi all,
I hope this is the right place to ask. If not I would be very grateful if you point me to the right place. Also I want to Sorry, if I missed something obvious in the descriptions or tutorials.

I'm using libtorrent v2.0.9 in a C++ project where I try to exchange data in a local network between a number of computers. The data can be fairly large (~100GB to 300GB).
The way it works is that a new data-set is presented to libtorrent, which then creates a magnet link, that is exchanged between the machines and the others then use libtorrent to download the data from the first one. This works so far in most cases and especially for files up to 10GB, but when changing to large files (for instance 100 GB), I eventually will get the error "217 v1 and v2 hashes do not describe the same data". The error occurs at what seems random positions between 30% to 60% into the file on the downloading machine.

I have been reading the manual for some time and also have been trying to debug and trying different configurations for libtorrent. But nonetheless I'm still struggling to understand what's happening and I still get the error unless i turn of hash_checks or switch to large piece-sizes. Maybe you can explain to me why I get this error, and what I need to do. My focus is to get to a reliable solution.

The code I use is mainly from the excellent tutorials on libtorrent. I did some adaptions over time.

The creation of the magnet-uri is as follows(cf. https://libtorrent.org/reference-Create_Torrents.html):

    auto abs_path = std::filesystem::absolute(path);
    lt::file_storage file_storage;
    lt::create_flags_t flags {lt::create_torrent::v2_only }; // v2_hashes only
    lt::add_files(file_storage, abs_path, flags);
    lt::create_torrent torrent(file_storage, 16 * 1024 * 1024); // I varied and this works for now
    torrent.set_creator("<Me>");
    lt::set_piece_hashes(torrent, abs_path.parent_path());
    lt::entry entry_info = torrent.generate();
 
    auto torrent_name = (work_directory / path.filename()).string() + ".torrent";

    std::vector<char> buffer = torrent.generate_buf();
    std::ofstream ss(torrent_name, std::ios_base::binary);
    lt::bencode(std::ostream_iterator<char>(ss), buffer);
    ss.close();

    lt::make_magnet_uri(lt::torrent_info(torrent_info)); 

I assume that setting the flags to v2_only would only generate v2 hashes for the pieces of the given path in the set_piece_hashes call.

Later I follow the libtorrent tuorial to create a torrent_session (https://www.libtorrent.org/tutorial-ref.html):

    // ...
    auto session_params = read_file(session_file);
    auto params = ( session_params.empty() ? lt::session_params() : lt::read_session_params(session_params) );
    params.settings.set_int(
        lt::settings_pack::alert_mask, 
        lt::alert_category::error | lt::alert_category::storage 
                                  | lt::alert_category::status 
                                  | lt::alert_category::connect 
    );
    params.settings.set_bool( lt::settings_pack::enable_dht, false);  
    params.settings.set_bool( lt::settings_pack::enable_lsd, true);
    params.settings.set_bool( lt::settings_pack::enable_outgoing_tcp, true);
    params.settings.set_bool( lt::settings_pack::enable_outgoing_utp, true);
    params.settings.set_bool( lt::settings_pack::enable_incoming_tcp, true);
    params.settings.set_bool( lt::settings_pack::enable_incoming_utp, true);
    params.settings.set_bool( lt::settings_pack::allow_multiple_connections_per_ip, true);
    // I experimented with some settings:
    // params.settings.set_bool( lt::settings_pack::disable_hash_checks, true);
    // params.settings.set_int( lt::settings_pack::max_piece_count, 0xA00000);

    // skip some stuff
    lt::session(params)

Things that I figured out:

• disable_hash_checks suppresses the error. The data gets transferred fully and running checksums on the original data and the copy shows that it is the same
• switching to larger piece-sizes works for the a testfile (105GB). Which is good. However, I was wondering if this would be really the issue and if there is a rule of thumb how large the pieces should be for a given file and if there is an optimal size.
• Setting the max_piece_count does not work, I thought this maybe related

What really puzzles me is the I explicitely set the hashes to be generated to v2_only and still I get the error on v1 and v2 hashes not describing the same data.

Do you have an idea what is going on and what to do about it? Any advice is highly appreciated!

Thank you very much,
Martin

[arvidn]

sorry for not getting back to you sooner. I would need to do some investigation to have a good answer for you. But it doesn't sound right that, if you create a hybrid torrent with libtorrent, you would get this error. That sounds like a bug somewhere. This error shouldn't happen for v1-only or v2-only torrents, so why you end up with a hybrid torrent is another question.