gcloud container get-credentials not authenticating service account

Kubernetes version: Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.3", GitCommit:"c6411395e09da356c608896d3d9725acab821418", GitTreeState:"clean", BuildDate:"2016-07-22T20:29:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: GKE
  • OS (e.g. from /etc/os-release): Debian 8
  • Kernel (e.g. uname -a): Linux 5ab86176e0e5 3.13.0-88-generic Add load balancing support to services. #135-Ubuntu SMP Wed Jun 8 21:10:42 UTC 2016 x86_64 GNU/Linux

What happened: After successfully authenticating with a service account on gcloud using:

gcloud auth activate-service-account $GOOGLE_AUTH_EMAIL --key-file /keyconfig.json --project $GOOGLE_PROJECT_ID

The cluster credentials were downloaded using:

gcloud container clusters get-credentials $CLUSTER_NAME

Even tho no errors are raised, when attempting to run kubectl version I get this error:

kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.3", GitCommit:"c6411395e09da356c608896d3d9725acab821418", GitTreeState:"clean", BuildDate:"2016-07-22T20:29:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

What you expected to happen: Kubectl should be configured to use the cluster properly

How to reproduce it (as minimally and precisely as possible):

  • Authenticate with a service account on gcloud
  • Get credentials from a cluster through gcloud
  • Run any kubectl command that reaches the server

Anything else do we need to know:
After looking into the configuration environment I observed the user is not properly configured on .kube/config:

kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: REDACTED
  name: gke_cluster_name
contexts:
- context:
    cluster:  gke_cluster_name
    user: gke_cluster_name
  name:  gke_cluster_name
current-context:  gke_cluster_namel
kind: Config
preferences: {}
users:
- name:  gke_cluster_name
  user:
    auth-provider:
      config: null
      name: gcp

After looking in the gcloud docs, I found this instruction:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"

After exporting the variable and running get-credentials once again, running kubectl version worked, and then running kubectl config view correctly displayed the user as authenticated

...
users:
- name: gke_cluster_name
  user:
    auth-provider:
      config:
        access-token: REDACTED
        expiry: 2016-08-15T12:30:18.220399721Z
      name: gcp

Note that this workflow was running perfectly in the past. My guess is that kubectl is no longer correctly detecting the gcloud service account that is authenticated. It's only looking for the environment variable. Either way, something seems broken :)