Accessing Minikube ETCD Database in Kubernetes

What is etcd?

ETCD is the distributed key/value database that works in tandem with Kubernetes deployments to hold vital information about the cluster. The security implications of allowing open access to it are vast and can leave your data exposed to bad actors.

ETCD does not operate like other components of the Kubernetes cluster, in that it might not be deployed as a pod, but instead runs as a process across the control plane nodes. Therefore, it is managed completely differently than other components because of the critical nature of it. A system can be rebuilt if etcd is still intact, however a loss of it's data could mean a complete loss of critical data.

Minikube

Minikube is a test environment for Kubernetes that runs usually as a docker container on localhost. It is essentially a 1-node Kubernetes deployment and playground, and is by far the most popular option for development. Because it is a docker container, it's important to know how exactly Minikube is running on your localhost. You can grab it's IP by running:

minikube ip

This prints out the IP of the docker container, as well as the port Minikube is listening on. It is this you will connect to. Once you have minikube up and running, connect into the container via Docker exec:

docker exec -it minikube -- /bin/bash

You now have a shell into the container. You can view the various processes involves with the Kubernetes control plane (since it all runs on this node) by checking the processes and grepping for various services.

ps -aux | grep etcd
ps -aux | grep api

You can view the output and see that kube-apiserver is running with a bunch of command line arguments. A few of these arguments are the cert files required to connect. To connect to etcd, you need following:

• CA cert

• Server cert

• Server key

These are all located in the /var/lib/minikube/certs/etcd directory.

cd /var/lib/minikube/certs/etcd

Minikube runs as an Ubuntu base image and therefore requires you to use apt to install packages. The package we install is etcdctl, the etcd client that connects to the etcd backend.

sudo apt-get update -y
sudo apt-get install -y etcd-client

Now from the etcd certs directory, running the following to test connectivity"

ETCDCTL_API=3 etcdctl --cacert ca.crt --cert server.crt --key server.key --endpoints https://127.0.0.1:2379 get /registry/ --prefix --keys-only

This does a few things:

• Sets the etcd-client to the correct API version based on the current etcd version

• Sets the appropriate certs and key flags as mentioned before

• Tells the etcd-client which endpoint to connect to, in this case, our etcd cluster

• Calls a get against the /registry/ URI

• Output just the keys (remember, etcd is a K/V store)

This should output a list of various keys in URI format, and they look pretty familiar. For example, you can list all current pods by running:

ETCDCTL_API=3 etcdctl --cacert ca.crt --cert server.crt --key server.key --endpoints https://127.0.0.1:2379 get /registry/ --prefix --keys-only | grep pods/default

Clearly this data is valuable to Kubernetes, and even more to an attacker. This is the Kubernetes backbone, and it's great to actually 'see' where those yaml deployments are stored.

This ability to basically curl the etcd endpoint gives us the ability to be a bit more malicious, where we can view secrets and service-account tokens in plaintext. We cause severe data confidentiality and integrity issues here, but we can also affect the availability of the application by wreaking havok on the cluster. You can perform any number of operations using the etcd API, including writing bogus data or just deciding to completely wipe out the cluster.

Summary

This was a simple demonstration of how to connect locally to your Minikube cluster and view the 'meat' of the Kubernetes cluster. All the deployments, applies, deletes and edits all end up reading/writing to this database. This is why securing your Kubernetes cluster starts with the etcd deployment and ensuring that things like encryption and access-control are firmly in place.