How to make sure the Kubernetes control plane is healthy
How to make sure the Kubernetes control plane is healthy Why is this important We are running an on premise Kubernetes cluster (currently version 1.11.6) on Red Hat Linux 7.5 (in VMware). Most documentation (especially when it comes to master version upgrades) mentions checking that the control plane is healthy prior to performing any cluster changes. Obviously this is an important step to ensure consistency and repeatability - and also important during day to day management of your cluster, but how exactly do we do this?
Docker Container Size Quota
Docker Container Size Quota Configuration We are running an on premise Kubernetes cluster on Red Hat Linux 7.5 (in VMware). The /var/lib/docker filesystem is a separate partition, formatted with ext4 and we used overlay as storage provider for docker, which was recommended for earlier RHEL 7 releases. What happened One fine day, one of our containers started creating core dumps - about 1 GB per minute worth, resulting in /var/lib/docker (100 GB in size) to fill up in less than 90 minutes.
Elasticsearch unassigned shards
Elasticsearch unassigned shards Elasticsearch shards across a cluster can get into many undesirable states. Some such state hit us with our Jaeger collector stopping our Docker containers and Kubernetes pods from starting. Our Elasticsearch cluster was treated harshly and both data nodes were offline at the same time causing a state that Elasticsearch could not recover from without intervention. The below examples showed the steps taken to recover the cluster.
Testing remote port connectivity - Linux hack
Testing remote port connectivity - BASH On occasion, Telnet is not installed on a system, making it difficult to test if a remote port is connectable or not. If this is the case, the below method using simple Linux devices can help diagnose a remote port connectivity issue. Using bash to test a remote port Linux has a built in handler to test port connectivity in the /dev/tcp/ directory. Using the below is a handy way to test remote port connectivity.