on
Edit Helm3 Manifest Secret In Place
Preface
Introduction
At reecetech we have been using Helm to template and apply Kubernetes resources for a few years. We went through the pain of updating from Helm2 to Helm3, a migration that took far longer than we estimated 😅.
Helm3 is a large improvement for us as it uses Kubernetes secrets inside the namespace the Helm chart is installed. This allows for a very small server side footprint for Helm. With Helm3 the Helm client reads the secret for the chart and understands the manifest of Kubernetes resources. The secret is a double base64 encoded
and gzipped
string of the YAML
Kubernetes resource. this all sounds confusing, but with a simple Linux one liner we can edit the secret in place.
This post is in response to a recent upgrade of Kubernetes (1.22) which removed support for Ingress API networking.k8s.io/v1beta1
. As a platform team we had been blocking builds using this API version, but some assets were not built before we planned to migrate to Kubernetes 1.22. After upgrading Kubernetes, we discovered that Helm was not able to update or delete Kubernetes resources if the Helm3 manifest had this older Ingress version
We were left with the option of asking teams to rebuild their offending assets and deploy before we update Kubernetes or to update the Helm3 manifest in place.
Helm3 Secret Manifest Manipulation
Finding Offending Helm Manifests
Reading out the contents of a Helm3 manifest is simple enough with the helm -n <namespace> get manifest <Helm chart name>
, which will print out the YAML resources associated with the Helm chart. Updating manifests is not possible using the helm
binary.
We have a standardised naming convention for our Helm3 YAML templates which allows for a neat double loop to show all manifests that still have the offending Ingress API version.
Below shows the loop to get all namespaces from the current Kubernetes context, looping over every secret having helm
in the name, using jq
to format the output, base64
to decrypt and gzip
to uncompress the data looking for networking.k8s.io/v1beta1
.
for j in $(kubectl get ns | awk '{print $1}')
do
echo "${j}"
for i in $(kubectl get secrets -n "${j}" | grep helm | awk '{print $1}')
do
kubectl get secrets -n "${j}" "${i}" -o json | jq .data.release -r | base64 -d | base64 -d | gunzip | jq '.chart.templates[] | select(.name == "templates/ingress.yaml") | .data' -r | base64 -d | grep networking.k8s.io/v1beta1 && echo "${i}"
done
done
Updating Offending Helm Manifests
Armed with the list of Helm manifests with deprecated API versions, it is easy to create a Kubernetes patch to update the secret in place. Using kubectl patch secret
allows for the Helm manifest data to be updated in place.
The below example shows setting a shell variable PATCH_DATA
to be the resulting updated Helm3 manifest with a substitution using sed
to replace the deprecated API version with the supported API version. Patching the secret is also shown in the below example as the second step using kubectl
.
PATCH_DATA=$(kubectl get secrets -n dummy-namespace sh.helm.release.v1.dummy-chart.v6 -o json | jq .data.release -r | base64 -d | base64 -d | gunzip | sed 's|networking.k8s.io/v1beta1|networking.k8s.io/v1|' | gzip -c | base64 | base64)
kubectl patch secret -n dummy-namespace sh.helm.release.v1.dummy-chart.v6 --type='json' -p="[{\"op\":\"replace\",\"path\":\"/data/release\",\"value\":\"$PATCH_DATA\"}]"
secret/sh.helm.release.v1.dummy-chart.v6 patched