[SERVE][AUTOSCALERS] Replica scaling sampling period and stability.

[JGSweets]

In autoscalers.py within serve:

skypilot/sky/serve/autoscalers.py

Lines 258 to 269 in 3f62588

	elif target_num_replicas > self.target_num_replicas:
	self.upscale_counter += 1
	self.downscale_counter = 0
	if self.upscale_counter >= self.scale_up_consecutive_periods:
	self.upscale_counter = 0
	self.target_num_replicas = target_num_replicas
	elif target_num_replicas < self.target_num_replicas:
	self.downscale_counter += 1
	self.upscale_counter = 0
	if self.downscale_counter >= self.scale_down_consecutive_periods:
	self.downscale_counter = 0
	self.target_num_replicas = target_num_replicas

When a single qps check is below or above the threshold, the downscale_counter or upscale_counter is set to 0.
This means a single jitter in qps could disrupt scaling.

I propose we allow a sampling over a period to allow scaling to occur based on a percentage of occurrences vs resetting to 0.
This could be set in the scaling policy.

---

Also, since scaling utilizes math.ceil, it errors on scaling and keeping qps below the value as a max bar vs a target.

skypilot/sky/serve/autoscalers.py

Line 192 in 3f62588

target_num_replicas = math.ceil(

---

Version & Commit info:

• sky -v: 0.7.0
• sky -c: 3f62588

[Michaelvll]

cc'ing @cblmemo

[cblmemo]

Also, since scaling utilizes math.ceil, it errors on scaling and keeping qps below the value as a max bar vs a target.

Hi @JGSweets ! Thanks for reporting this. I think the first solution makes sense! Could you elaborate on the second one?

[JGSweets]

Error may have been a poor choice in wording. I meant that it hedges towards scaling up since it rounds up with any float. This means qps/replica drops significantly if it is barely breaching the requirement. This may be ideal behavior for some.

Maybe this was chosen because if there's any qps, you want to guarantee scaling to 1?

[cblmemo]

Error may have been a poor choice in wording. I meant that it hedges towards scaling up since it rounds up with any float. This means qps/replica drops significantly if it is barely breaching the requirement. This may be ideal behavior for some.

Maybe this was chosen because if there's any qps, you want to guarantee scaling to 1?

That is one reason, but the primary reason is that we dont want any overload on the service.

[JGSweets]

That is one reason, but the primary reason is that we dont want any overload on the service.

Would also be convenient if this was a setting where we had a choice. Similar to AWS alarms and triggers.