[GCP] Activate service account for storage and controller

[Michaelvll]

Fixes #4528 and #4512

When a service account is used locally, although the service account key is uploaded, it cannot be used by the code for cloud storage. This is because gcloud auth activate-service-account is required for service account key, which is not needed for the normal user application key. We now activate the service account anyway, to make sure the service account works.

To reproduce: #4528

1. Set a ~/.sky/config.yaml

jobs:
  controller:
    resources:
      cpus: 2
      cloud: kubernetes

2. Have the service account activated for GCP locally: gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIAL
3. Run a managed job with sky jobs launch test.yaml

resources:
  cloud: kubernetes
  cpus: 2


workdir: examples

run: ls

4. The current master will fail to download the workdir from the bucket to the job cluster due to the gsutil fail to find the credential.

To reproduce: #4512
1. Set a ~/.sky/config.yaml

jobs:
  controller:
    resources:
      cpus: 2
      cloud: kubernetes

2. Have the service account activated for GCP locally: gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIAL
3. sky jobs launch -n test --cloud gcp --cpus 2 echo hi

Actually, after testing, the controller could correctly launch clusters on GCP even without the activation of service account. It seems that the only GCP-related commands that require activating service account is the storage, so we don't have to activate the service account for controller. (Note: the kubernetes cluster is started with sky local up, so there should be no default gcp credential)

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • The reproducibility above
  • With normal user account.
• All smoke tests: pytest tests/test_smoke.py
  • pytest tests/test_smoke.py --managed-jobs --gcp with a service account activated and controller on k8s.
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[Michaelvll]

/quicktest-core

[romilbhardwaj]

Awesome @Michaelvll!

[romilbhardwaj]

Does something similar need to be done for our VM provisioning pipeline on the jobs/serve controller?

For example, if the user is using service account for auth, runs a controller on k8s and wants to launch a managed job on GCP, will he run into the same issue (see #4512)?

[Michaelvll]

Good point! I changed the PR to fix the issue.

[romilbhardwaj]

Awesome, thanks for the quick turnaround @Michaelvll!

[Michaelvll]

Actually, after testing, the controller could correctly launch clusters on GCP even without the activation of service account. It seems that the only GCP-related commands that require activating service account is the storage, so we don't have to activate the service account for controller. (Note: the kubernetes cluster is started with sky local up, so there should be no default gcp credential). I reverted the changes for controller.

PTAL @romilbhardwaj

[romilbhardwaj]

Thanks for verifying @Michaelvll!