Question about Bittorrent V2 File Hashes

[BernhardGlueck]

Hi,

First if this is the wrong venue to ask these questions, i apologize but i seem to have run out of options otherwise.

I am implementing bittorrent support in my own project, which only requires tracker and torrent file generation. (Basically allowing me to expose files as torrents )

I have almost everything working but the file hash generation.
I also cannot use libtorrents2 implementation since my files i generate torrent files for are virtualized and not stored on disk.

Due to several constraints i only support V2 of the bitorrent protocol ( no V1 or Hybrid torrents )

But i can't seem to figure out the hash generation for a given file and BEP 0052 seems very vague in regard.

Right now i am doing the following ( in regards to a single file, as in Bittorrent V2, each file forms its own hash tree )

1. Select a piece size which has to be divisible by 16 kib and is a power of two.
   I use 1 MIB ( 1024 * 1024 )
2. Split my file into pieces. The last piece is usually smaller than the piece size of course.
3. Split each piece into blocks of 16 kib
4. Calculate the SHA256 for each block
5. Check that i have a power of two number of hashes, if not add zero hash values until i get a power of two hashes
6. Take each pair of hashes, concatenate them and calculate the SHA256.
7. Repeat until i only have one hash left.

8 If my file had only once piece, i am done, the root of the block hash tree is also the pieces root.
THIS works correctly, and i get the same pieces root as libtorrentv2 calculates.

My problems start with

9. I now have one root hash for each piece in my file.

10. I assume i have to build another hash tree until i have one root hash left.
    But so far any attempt to build this has failed, my results never match liborrentv2s results.

    I tried padding the piece hash values with zero hashes, with the last hash duplicated and so on until i have a power of two.
    I have also tried just calculating the block hash tree over the whole file ....

Does anybody have an idea on what i am doing wrong ? 

[arvidn]

there is a reference implementation for creating torrents here: http://bittorrent.org/beps/bep_0052_torrent_creator.py
linked to from BEP52.

Just a few comments;

1. the piece size is not selected per file, but for the whole torrent. it primarily determines the granularity of the HAVE messages and the BITFIELD message, also determines which hashes are included in the "piece layers".

2. Note that the last piece is padded. The last 16kiB block is not padded, but the piece is padded with padding blocks to make it complete.

3. Note that the last 16 kiB block may be truncated, and you compute the hash on the truncated block.

4. The padding is the hash of an (imaginary) block of zeros that's hashed

5. the root hashes you have for each piece, do they all match the "piece layers" hashes libtorrent produce? That's effectively what you've computed at that point.

6. I think it helps to look at it as a single merkle hash tree for the whole file