feat: add grace period after sumitter finished

[machichima]

Why are these changes needed?

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[Future-Outlier]

todo: maybe a fake dashboard always return job running, but the submitter is finished.

[Future-Outlier]

use ray image with fixed version + gcs fault tolerance, maybe is a way to do this if e2e test is too hard to write.

cc @rueian @kevin85421

[Future-Outlier]

We can use LastTransitionTime from @rueian

// JobCondition describes current state of a job.
type JobCondition struct {
	// Type of job condition, Complete or Failed.
	Type JobConditionType `json:"type" protobuf:"bytes,1,opt,name=type,casttype=JobConditionType"`
	// Status of the condition, one of True, False, Unknown.
	Status corev1.ConditionStatus `json:"status" protobuf:"bytes,2,opt,name=status,casttype=k8s.io/api/core/v1.ConditionStatus"`
	// Last time the condition was checked.
	// +optional
	LastProbeTime metav1.Time `json:"lastProbeTime,omitempty" protobuf:"bytes,3,opt,name=lastProbeTime"`
	// Last time the condition transit from one status to another.
	// +optional
	LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty" protobuf:"bytes,4,opt,name=lastTransitionTime"`
	// (brief) reason for the condition's last transition.
	// +optional
	Reason string `json:"reason,omitempty" protobuf:"bytes,5,opt,name=reason"`
	// Human readable message indicating details about last transition.
	// +optional
	Message string `json:"message,omitempty" protobuf:"bytes,6,opt,name=message"`
}

[machichima]

I used CompletionTime in job.Status for getting the time when job submitter finished. Will look into how to do the testing then

https://github.com/ray-project/kuberay/pull/4091/files#diff-f10777dd022e918221782f3ea3f88c70293404f9e62ded13cbad66b68c6d19efR1094