I’m writing about the Kubernetes API’s use of the “LIST” verb it controls access to Secrets in a cluster. I’ve seen way too may environments, tools, templates, and examples that are hoping that LIST verbs can provide a meaningful security control compared to the GET verb. I’m going through a simple demo below and a few one-liners that can help you audit this yourself.

Background

Kubernetes Roles are designed to give fine grained access to various API actions within the cluster by defining “Verbs” that you’re allowed to send to the API. GET, LIST, UPDATE, CREATE… and a bunch more.

They are simple:

• GET: Retrieve a full kubernetes object.

  ~ kubectl get secret mysecret
  

• LIST: Retrieve a list of objects available in the cluster.

  ~ kubectl get secrets
  

But for the Secrets resource, LIST is a lie. When you “list” a secret, it goes out to the Kuberntes API, pulls down all the secrets including individual secret values themselves. This is a known and documented issue but I see lots of environments that continue to rely on LIST to prevent access to secrets.

For example, if you need to obtain a list of secrets but don’t want to grant someone access to the secret itself, it’s logical to build a Role that grants LIST and not GET. There’s even a fake client-side error message in kubectl saying when you don’t have permission to GET a value even though it already has a copy of it.

I’ll demo below how this works and then give you some examples of when it often comes into play.

Demo Setup

Here’s a simple setup for a ClusterRole, ClusterRoleBinding, and ServiceAccount. In our scenario you can imagine that you manage a cluster and someone is coming in to audit or “scan kubernetes”. You’ll want to just give them read only access to the cluster which means you don’t want them able to access the secrets because they would just be able to escalate their privileges.

You can fast forward through this part with apply -f https://gist.githubusercontent.com/antitree/7a2461207259eb36f46ec250eef91ab1/raw

Here’s my ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: poc-list-bypass
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - list
  - watch

Here’s my ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: poc-list-bypass
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: poc-list-bypass
subjects:
- kind: ServiceAccount
  name: poc-list-bypass-sa
  namespace: default

My ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: poc-list-bypass-sa
  namespace: default

How about a secret to steal:

 
apiVersion: v1 
data: 
  mysecret: a3ViZXJuZXRlcyBMSVNUIHZlcmIgaXMgYSBsaWU= 
kind: Secret 
metadata: 
  name: ultra-secret-string 

We can see I have a service account setup named poc-list-bypass-sa-token-jsv9b:

~ kubectl get secrets
NAME                             TYPE                                  DATA   AGE
poc-list-bypass-sa-token-jsv9b   kubernetes.io/service-account-token   3      22s

Let’s emulate the scenario where my pod was compromised and they stole the service account token. I’ll use view_secret from Krew:

~ kubectl-view_secret poc-list-bypass-sa-token-jsv9b token
eyJhbGciOiJSUzI1NiIsImtpZCI6ImNwLTVpSDVmZ0pDaDZ1RzNEZkdSZy1qMFJkaExxcUt6ZnNZeFdBSjg5SnMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InBvYy1saXN0LWJ5cGFzcy1zYS10b2tlbi1qc3Y5YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwb2MtbGlzdC1ieXBhc3Mtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxMjNjNmUyOS03MzE2LTQ2MzgtYjEwMy0yY2U1ZDMzMTJhNmYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwb2MtbGlzdC1ieXBhc3Mtc2EifQ.L3OrhKMVatUmx0uzzHf8-AG4jyi8nfLZqc6i8R0zajOzscOovAalSeBv5GQIE6RwBiGfwkE3wUHw7nXoOvgi2P1Ly2lnhhAaaipiwUOohpgZbQ5WplEMoFDAKvvYAhtYw3Ly3thSmVWrftuMm6y7X0q-aCWLWOldJq3bxH6idv3kiUcNbT7-ql6Zebu0BrUth0KNQAt4IMTEDbGmd1YLX1QI_aTH87pXMvBqhhqz57cYqqw_vv0PX4uguLORoWaFtQtq273As2vCKMOPU4o4ooOJvDIsOQd6p11kKxqFaeV5JEoUSvwtx8KWOfRZGdG1GcpzveXE9FNjQC6oGr6QQg%            

I’ll load it into an env var so we don’t have to look at that token:

~ export TOKEN=$(kubectl-view_secret poc-list-bypass-sa-token-jsv9b token)

Demo

I’m going to make sure we’re using our stolen service account token and not accidentally using an admin account (because everything seems to want to automagically update my kubeconfig file and I hate it!). First I’ll check my normal access and then switch using the --token option to confirm.

~ kubectl config view --raw > temp_kube.config
~ export KUBECONFIG=./temp_kube.config
~ kubectl config unset users
~ kubectl get po
Please enter Username:

Now switch to the stolen creds:

~ kubectl-whoami --token=$TOKEN
system:serviceaccount:default:poc-list-bypass-sa
~ kubectl get po --token=$TOKEN
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:poc-list-bypass-sa" cannot list resource "pods" in API group "" in the namespace "default"

We can list secrets:

 
~ kubectl get secret --token=$TOKEN 
NAME                                  TYPE                                  DATA   AGE 
default-token-k9gv8                   kubernetes.io/service-account-token   3      72m 
poc-list-bypass-sa-token-jsv9b        kubernetes.io/service-account-token   3      66m 
ultra-secret-string                   Opaque                                1      22m 

We shouldln’t be able to GET a secret directly:

~ kubectl get secrets ultra-secret-string --token $TOKEN
Error from server (Forbidden): secrets "ultra-secret-string" is forbidden: User "system:serviceaccount:default:poc-list-bypass-sa" cannot get resource "secrets" in API group "" in the namespace "default"

Finally, let’s just get the secret anyways:

~ kubectl get secret --token $TOKEN -o json | jq -r '.items[] | select(.metadata.name=="ultra-secret-string")'
{
  "apiVersion": "v1",
  "data": {
    "mysecret": "a3ViZXJuZXRlcyBMSVNUIHZlcmIgaXMgYSBsaWU="
  },
  "kind": "Secret",
...  

Even better, dump the exact secret value:

~ kubectl get secret --token $TOKEN -o json | \
 jq -r '.items[] | select(.metadata.name=="ultra-secret-string")| .data["mysecret"]' | \
 base64 -d

kubernetes LIST verb is a lie%  

When Does This Matter?

I first figured this out with my then co-worker Josh Makinen when we were on a job reviewing a GKE related environment. The administrators logically granted us “View” permission into the cluster and as part of his testing, he would run kubectl-info dump > cluster.dump to have an offline version of the objects. At the same time, I was politely asking the API for permission to GET secrets with kubectl get secret some_service_account – I was getting denied but he already had a copy of the secret stored in his file. A little extra work and reading through the documentation we found that this was a documented problem with the LIST verb when it applies to secrets.

“For these reasons watch and list requests for secrets within a namespace are extremely powerful capabilities and should be avoided, since listing secrets allows the clients to inspect the values of all secrets that are in that namespace. The ability to watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.” - kubernetes.io

To quote Twitter, “is it a security issue if it’s been documented?" Yes, when people are using LIST as a security barrier, it is.

This only matters when you’re trying to give partial access to a cluster. A consultant that needs to look at your cluster or a new tool that you want to run that shouldn’t need to make any changes in the cluster.

Now that I know about it, I see how pervasive this is in Helm charts, default templates for services, and just everywhere. Even Istio deployed in GKE is configured a Role for Pilot with LIST only permissions on Secrets.

So what now? Well don’t do this.

But also, here’s a nice little jq one-liner you can use to dump all the ClusterRoles that have LIST but not GET permissions. The thought is that if you have LIST without GET, you’re attempting to restrict access to secrets but you’re going to be sorely mistaken.

~ kubectl get clusterroles -o json |\
    jq -r '.items[] | select(.rules[] |
    select((.resources | index("secrets")) 
    and (.verbs | index("list")) 
    and (.verbs | index("get") | not))) |
    .metadata.name'