[WIP] Scaling on AWS

[suhlrich]

This closes #113 .

TODO AWS CLI

1. Get machine ID
2. send shutdown message

turn off backend machine after 5 mins of no trial and there is nothing recording

[antoinefalisse]

@sashasimkin and @suhlrich could you take a look at this? I added the boto3 scripts and adjusted a couple of things.

• I am not sure this is what we want here. I understand that if the number of trials is under a certain threshold (like the number of on-prem workers times a number of trials per worker) then we want to remove the scale-in protection. I think this is what this will do, is that what you had in mind?
• I don't think this will ever be triggered, except if we don't use on-prem workers (and we want this to work). But the idea here is to remove the scale-in protection if the instance hasn't processed a trial for more than x minutes and if there aren't trials being recorded.

[suhlrich]

looks good!

[antoinefalisse]

Hi @sashasimkin as I mentioned in the comment, I don't think that part should ever be triggered when we have on-prem workers since there should be no case when that status is 404. However, I would like that part to work, and us to test it, when there are no no-prem workers. The intended behavior is NOT to process only 1 trial by every single worker machine. The intention is to remove the scale-in protection after a certain time minutesBeforeRemoveScaleInProtection and if there aren't trials being recorded. If there is a queue, then the justProcessed variable gets set again and again to True and the scale-in protection is not removed. Does that make sense?

[antoinefalisse]

I don't think I understand why we would need to move this there:

only to move if r.status_code == 404: block before if with_on_prem:

[sashasimkin]

can you explain what would be the reason for r.status_code == 404? I assumed that this is returned when there's no work to do.

From what I see, it seems that setting "max_on_prem_pending_trials" to 0 when running without on prem workers will do exactly what you want, if you lift the "with_on_prem" block and run the check in the main loop?

re. "The intended behavior is NOT to process only 1 trial by every single worker machine." - my bad, I didn't read the code fully, sorry.

[antoinefalisse]

can you explain what would be the reason for r.status_code == 404? I assumed that this is returned when there's no work to do.

Yes, exactly. If there are no job, then it returns 404 and start over until there is a job to process

From what I see, it seems that setting "max_on_prem_pending_trials" to 0 when running without on prem workers will do exactly what you want, if you lift the "with_on_prem" block and run the check in the main loop?

I'm not sure I follow this. Are you suggesting to move this somewhere else?

[sashasimkin]

My suggestion is to not have it at all as well as not have "if with_on_prem:" condition for unprotecting to simplify the code.

But I'm now not sure that I understand the whole logic, so I'll resolve this comment now and will try to explain the point on the call today.