add step option to AutoOptions

[fruch]

we have the -rate auto threads>=100 threads<=1000 option, and currently it's logic for
increasing the number of thread is hardcode like the following:

            runIds.add(threadCount + " threadCount");
            prevThreadCount = threadCount;
            if (threadCount < 16)
                threadCount *= 2;
            else
                threadCount *= 1.5;

we want to add a step option to the AutoOptions, so we'll be able to control better the jumps until saturation happens

    private static final class AutoOptions extends GroupedOptions
    {
        final OptionSimple auto = new OptionSimple("auto", "", null, "stop increasing threads once throughput saturates", false);
        final OptionSimple minThreads = new OptionSimple("threads>=", "[0-9]+", "4", "run at least this many clients concurrently", false);
        final OptionSimple maxThreads = new OptionSimple("threads<=", "[0-9]+", "1000", "run at most this many clients concurrently", false);

        @Override
        public List<? extends Option> options()
        {
            return Arrays.asList(minThreads, maxThreads, auto);
        }
    }

[fruch]

@roydahan

how do you prefer the step to be define ?

absolute numbers ? percentage of max threads ? multipier (like in the code but that we can control) ?

[roydahan]

Ideally we should implement the following "TODO" in the code (from 2013):

// TODO : vary thread count based on percentage improvement of previous increment, not by fixed amounts

This will allow us not configuring anything and let c-s detect the best throughput.

[roydahan]

A quick fix for the issue is suggested here:
#12

However, a better fix should be a closed feedback loop as suggested in my comment above.
It should increase/decrease threadCount based on percentage improvement of previous step.

[roydahan]

Actually, I just noticed this stop condition:

// run until we have not improved throughput significantly for previous three runs
} while (!auto || (hasAverageImprovement(results, 3, 0) && hasAverageImprovement(results, 5, settings.command.targetUncertainty)));

Which may be sufficient for our needs.

[CodeLieutenant]

Actually, I just noticed this stop condition:

// run until we have not improved throughput significantly for previous three runs } while (!auto || (hasAverageImprovement(results, 3, 0) && hasAverageImprovement(results, 5, settings.command.targetUncertainty)));

Which may be sufficient for our needs.

Do we still need #12 PR? @roydahan

[roydahan]

Yes, I'm using it with a manual docker I built and the granularity is much better, helping me to easily find the max.

I'm still considering if we need to add even lower granularity.

[fruch]

@roydahan

you still think it's needed ?

[roydahan]

Yes, I believe it would be good to have lower granularity for max finding.
It can be low priority.