Renewing Kubernetes certificates

Renewing Kubernetes certificates


Kubernetes uses many different TLS certificates for various levels of internal and external cluster service communication such as kubelet, apiserver, scheduler to name a few. Usually these certificates are only valid for 12 months.

These certificates are created (and signed by the K8s internal CA) during initial installation. However even though there are some options for automated renewals available, they are not always utilised and these certs can become out of date. Updating certain certificates may require restarts of K8s components, which may not be fully automated.

If any of these certificates is outdated or expired, it will stop parts or all of your cluster from functioning correctly. Obviously this scenario should be avoided - especially in production environments.

This blog entry focuses on manual renewals / re-creation of Kubernetes certificates.


Below is a list of K8s (1.16) internal files (on each master node) which include certificates.


There are also some certificates on each worker node, mainly used by kubelet.


Status check

Kubernetes conveniently offers kubeadm command line options to verify certificate expiration. As you can see below, all certificates are still valid for almost a year in this cluster.

$ kubeadm alpha certs check-expiration
admin.conf                 Apr 05, 2021 19:17 UTC   363d            no      
apiserver                  Apr 05, 2021 19:17 UTC   363d            no      
apiserver-etcd-client      Apr 05, 2021 19:17 UTC   363d            no      
apiserver-kubelet-client   Apr 05, 2021 19:17 UTC   363d            no      
controller-manager.conf    Apr 05, 2021 19:17 UTC   363d            no      
etcd-healthcheck-client    Apr 05, 2021 19:17 UTC   363d            no      
etcd-peer                  Apr 05, 2021 19:17 UTC   363d            no      
etcd-server                Apr 05, 2021 19:17 UTC   363d            no      
front-proxy-client         Apr 05, 2021 22:41 UTC   363d            no      
scheduler.conf             Apr 05, 2021 19:17 UTC   363d            no

Alternatively you can use openssl to verify the expiry time when connecting to the apiserver endpoint (can also be used to verify that the apiserver has been restarted since renewing the certificate):

# echo | openssl s_client -showcerts -connect -servername api 2>/dev/null | openssl x509 -noout -enddate
notAfter=Apr  5 19:17:16 2021 GMT

Manual renewal process

It is best practice to backup the /etc/kubernetes/pki folder on each master before renewing certificates.

All Kubernetes certificates can be re-created via kubeadm. Some certificates are specific to each master node name and some are shared across each service across different master servers (if any).

These commands will update / overwrite the corresponding certificate files as described above:

kubeadm alpha certs renew apiserver-kubelet-client
kubeadm alpha certs renew apiserver
kubeadm alpha certs renew front-proxy-client
kubeadm alpha certs renew apiserver-etcd-client
kubeadm alpha phase kubeconfig user --client-name system:kube-controller-manager

Renewing a certificate also requires the corresponding Kubernetes containers to be restarted. In most cases just deleting the pod (such as kubectl delete pod -n kube-system kube-scheduler-master1) or restarting kubelet will cause the containers / pods to be restarted and to read the new certificates.

The kube-apiserver process / pod uses many different TLS certificates - so this should ideally be restarted if any certificate changes / gets updated.

We have experienced problems with deleting the api-server pods using kubectl delete pod -n kube-system kube-apiserver-master1. The command completes as you would expect, that is, the pod resets the age to 0 seconds and the status temporarily transitions to Pending before returning to Running. However, the docker container does not actually get restarted!

Non-restarting api-server pods can be identified by running docker ps | grep kube-apiserver on the master. If the docker container uptime has not been reset, then the container can be killed via:

docker rm -f `docker ps | grep k8s_kube-apiserver | cut -d" " -f1`

We have experienced issues with this problem in the past, where the Kubernetes API server failed to communicate with the metrics-server (kubectl top and the HPA stopped working). The process was just logging Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid as the error message. The root cause was an expired front-proxy-client certificate (which was renewed recently without explicitly restarting the kube-apiserver containers).

Other certs

It is quite common to also have TLS certificates in config maps, which are used by ingress controllers etc. These also need to get renewed in a timely manner.


Kubernetes is quite complex and there are many moving parts which need to be maintained. Even though Reece has operated production Kubernetes clusters for years, we are still constantly learning and experiencing interesting challenges in this ever-changing space.

More information here: