Dynamic lying factor

[make]

@lvdmaaten Have you tried to use dynamic lying factor instead of static 12.0?
Got little lower error (dropped from 1.428 to 1.394 ) by replacing static lying factor with dynamic value:

    double lying_factor = 12.0;
    double lying_decrease = 0;
    if(stop_lying_iter > 0)
        lying_decrease = (lying_factor - 1.0) / (double) stop_lying_iter;

        if(iter < stop_lying_iter) {
            if(exact) { for(int i = 0; i < N * N; i++)        P[i] /= lying_factor; }
            else      { for(int i = 0; i < row_P[N]; i++) val_P[i] /= lying_factor; }
            lying_factor -= lying_decrease;
            if(exact) { for(int i = 0; i < N * N; i++)        P[i] *= lying_factor; }
            else {      for(int i = 0; i < row_P[N]; i++) val_P[i] *= lying_factor; }
        }

Output for bhtsne with static lying factor for input of 10000 x 200 samples:

Learning embedding...
Iteration 50: error is 83.834001 (50 iterations in 6.29 seconds)
Iteration 100: error is 82.772901 (50 iterations in 5.75 seconds)
Iteration 150: error is 82.683113 (50 iterations in 5.59 seconds)
Iteration 200: error is 82.657414 (50 iterations in 6.91 seconds)
Iteration 250: error is 4.401062 (50 iterations in 5.98 seconds)
Iteration 300: error is 2.621082 (50 iterations in 6.51 seconds)
Iteration 350: error is 2.200671 (50 iterations in 5.28 seconds)
Iteration 400: error is 1.975511 (50 iterations in 5.58 seconds)
Iteration 450: error is 1.834608 (50 iterations in 6.70 seconds)
Iteration 500: error is 1.740066 (50 iterations in 5.66 seconds)
Iteration 550: error is 1.671593 (50 iterations in 7.15 seconds)
Iteration 600: error is 1.619357 (50 iterations in 7.49 seconds)
Iteration 650: error is 1.579096 (50 iterations in 8.11 seconds)
Iteration 700: error is 1.550052 (50 iterations in 6.58 seconds)
Iteration 750: error is 1.528960 (50 iterations in 6.04 seconds)
Iteration 800: error is 1.516330 (50 iterations in 7.10 seconds)
Iteration 850: error is 1.507626 (50 iterations in 5.70 seconds)
Iteration 900: error is 1.500774 (50 iterations in 7.26 seconds)
Iteration 950: error is 1.495621 (50 iterations in 6.24 seconds)
Iteration 1000: error is 1.490954 (50 iterations in 6.61 seconds)
Iteration 1050: error is 1.486083 (50 iterations in 5.91 seconds)
Iteration 1100: error is 1.480977 (50 iterations in 9.09 seconds)
Iteration 1150: error is 1.475732 (50 iterations in 12.08 seconds)
Iteration 1200: error is 1.471433 (50 iterations in 7.36 seconds)
Iteration 1250: error is 1.467418 (50 iterations in 5.78 seconds)
Iteration 1300: error is 1.463404 (50 iterations in 7.68 seconds)
Iteration 1350: error is 1.459785 (50 iterations in 5.97 seconds)
Iteration 1400: error is 1.456251 (50 iterations in 5.53 seconds)
Iteration 1450: error is 1.453166 (50 iterations in 6.60 seconds)
Iteration 1500: error is 1.450100 (50 iterations in 5.54 seconds)
Iteration 1550: error is 1.447593 (50 iterations in 7.94 seconds)
Iteration 1600: error is 1.445209 (50 iterations in 5.48 seconds)
Iteration 1650: error is 1.442866 (50 iterations in 5.72 seconds)
Iteration 1700: error is 1.440184 (50 iterations in 6.16 seconds)
Iteration 1750: error is 1.437438 (50 iterations in 5.08 seconds)
Iteration 1800: error is 1.435075 (50 iterations in 6.89 seconds)
Iteration 1850: error is 1.432948 (50 iterations in 5.19 seconds)
Iteration 1900: error is 1.431293 (50 iterations in 5.38 seconds)
Iteration 1950: error is 1.429766 (50 iterations in 6.69 seconds)
Iteration 1999: error is 1.428257 (50 iterations in 5.49 seconds)

Output for bhtsne with dynamic lying factor for input of 10000 x 200 samples:

Learning embedding...
Iteration 50: error is 65.368017 (50 iterations in 7.20 seconds)
Iteration 100: error is 46.293893 (50 iterations in 6.24 seconds)
Iteration 150: error is 29.246011 (50 iterations in 7.21 seconds)
Iteration 200: error is 13.983286 (50 iterations in 6.51 seconds)
Iteration 250: error is 2.634763 (50 iterations in 7.59 seconds)
Iteration 300: error is 2.010282 (50 iterations in 5.58 seconds)
Iteration 350: error is 1.809022 (50 iterations in 5.98 seconds)
Iteration 400: error is 1.698381 (50 iterations in 6.79 seconds)
Iteration 450: error is 1.626216 (50 iterations in 6.61 seconds)
Iteration 500: error is 1.575453 (50 iterations in 7.00 seconds)
Iteration 550: error is 1.539009 (50 iterations in 5.83 seconds)
Iteration 600: error is 1.511758 (50 iterations in 7.45 seconds)
Iteration 650: error is 1.493206 (50 iterations in 5.83 seconds)
Iteration 700: error is 1.480972 (50 iterations in 6.17 seconds)
Iteration 750: error is 1.473207 (50 iterations in 6.50 seconds)
Iteration 800: error is 1.467411 (50 iterations in 5.75 seconds)
Iteration 850: error is 1.462050 (50 iterations in 6.94 seconds)
Iteration 900: error is 1.457054 (50 iterations in 5.78 seconds)
Iteration 950: error is 1.452213 (50 iterations in 6.93 seconds)
Iteration 1000: error is 1.447649 (50 iterations in 5.43 seconds)
Iteration 1050: error is 1.443852 (50 iterations in 6.00 seconds)
Iteration 1100: error is 1.440698 (50 iterations in 6.88 seconds)
Iteration 1150: error is 1.437592 (50 iterations in 6.08 seconds)
Iteration 1200: error is 1.434110 (50 iterations in 7.13 seconds)
Iteration 1250: error is 1.430698 (50 iterations in 5.43 seconds)
Iteration 1300: error is 1.427438 (50 iterations in 6.43 seconds)
Iteration 1350: error is 1.424260 (50 iterations in 4.72 seconds)
Iteration 1400: error is 1.421095 (50 iterations in 10.57 seconds)
Iteration 1450: error is 1.418043 (50 iterations in 11.95 seconds)
Iteration 1500: error is 1.415134 (50 iterations in 5.66 seconds)
Iteration 1550: error is 1.412521 (50 iterations in 6.57 seconds)
Iteration 1600: error is 1.409757 (50 iterations in 6.42 seconds)
Iteration 1650: error is 1.407288 (50 iterations in 5.95 seconds)
Iteration 1700: error is 1.404948 (50 iterations in 6.25 seconds)
Iteration 1750: error is 1.403020 (50 iterations in 5.08 seconds)
Iteration 1800: error is 1.401276 (50 iterations in 7.00 seconds)
Iteration 1850: error is 1.399376 (50 iterations in 5.49 seconds)
Iteration 1900: error is 1.397579 (50 iterations in 5.88 seconds)
Iteration 1950: error is 1.395958 (50 iterations in 6.58 seconds)
Iteration 1999: error is 1.394586 (50 iterations in 6.41 seconds)

Any opinions on this experiment?

[lvdmaaten]

My only comment would be that there is quite some variance in the error values that you get between runs (because the optimization problem is non-convex). So to really determine what works better, you'd have to do 100s of runs with both settings and then compare their means (and variances). Moreover, you'd probably want to repeat the experiment on different datasets and with different dataset sizes.

[make]

You are right. I forgot to mention that I use externally defined initialization for Y so that error and output is same for each run.

[lvdmaaten]

Even if the Y is fixed in your experiment: who says the outcome of the experiment wouldn't be different if you had used on another initial Y? You'd have to repeat this experiments many times (for many different datasets) to confirm this is a statistically significant improvement.