Seperate source (requestor) and target (provider) data or index file locations

[TonyxSun]

This aims to solve the problem of 'frogomobile' by preventing simultaneous R/W into the same file (in the same directory, as was happening with the index files).

We can also actually send files now (before, we either rely on the optimization that the files already exist, so we don't need to initiate fetching, or have FTP commented out for actual training files. The only files we have sent before where the index files). This is pretty important if we are going to test multiple machines.

Fixed some misc. issues that were needed to get 10,000 + file transfers to happen, like dangling file pointers/sockets, excessive creation of new sockets, and added socket timeout.

So I defined the following macros:

/*
 * Defines the data location of training files and index files for the requestor.
 */
const std::string SOURCE_DATA_DIR = "CIFAR10/train";

/*
 * Defines the data location of training files being used locally by a provider.
 */
const std::string TARGET_DATA_DIR = "data/CIFAR10";

The data should be in SOURCE_DATA_DIR for the requester. The training files will be downloaded to TARGET_DATA_DIR on the provider machines.

Demo
https://www.youtube.com/watch?v=gSePWZN4YAs

TODO in a meeting: I arbitrarily set the names of the folders. We can confirm whether they make sense and also the training scripts might need to changed to read from new folders. One problem with this is that we can't send the test files for now. We need a way to specify the set of files we want and where to download them.

[jordanmao]

I think we should add ml/test_first.txt, ml/test_second.txt, and ml/test_third.txt to .gitignore