Written by Guy Jackson
on January 12, 2022

Edit Helm3 Manifest Secret In Place

Preface

Introduction

At reecetech we have been using Helm to template and apply Kubernetes resources for a few years. We went through the pain of updating from Helm2 to Helm3, a migration that took far longer than we estimated 😅.

Helm3 is a large improvement for us as it uses Kubernetes secrets inside the namespace the Helm chart is installed. This allows for a very small server side footprint for Helm. With Helm3 the Helm client reads the secret for the chart and understands the manifest of Kubernetes resources. The secret is a double base64 encoded and gzipped string of the YAML Kubernetes resource. this all sounds confusing, but with a simple Linux one liner we can edit the secret in place.

This post is in response to a recent upgrade of Kubernetes (1.22) which removed support for Ingress API networking.k8s.io/v1beta1. As a platform team we had been blocking builds using this API version, but some assets were not built before we planned to migrate to Kubernetes 1.22. After upgrading Kubernetes, we discovered that Helm was not able to update or delete Kubernetes resources if the Helm3 manifest had this older Ingress version

We were left with the option of asking teams to rebuild their offending assets and deploy before we update Kubernetes or to update the Helm3 manifest in place.

Helm3 Secret Manifest Manipulation

Finding Offending Helm Manifests

Reading out the contents of a Helm3 manifest is simple enough with the helm -n <namespace> get manifest <Helm chart name>, which will print out the YAML resources associated with the Helm chart. Updating manifests is not possible using the helm binary.

We have a standardised naming convention for our Helm3 YAML templates which allows for a neat double loop to show all manifests that still have the offending Ingress API version.

Below shows the loop to get all namespaces from the current Kubernetes context, looping over every secret having helm in the name, using jq to format the output, base64 to decrypt and gzip to uncompress the data looking for networking.k8s.io/v1beta1.

for j in $(kubectl get ns | awk '{print $1}')
do
  echo "${j}"
  for i in $(kubectl get secrets -n "${j}" | grep helm | awk '{print $1}')
  do
    kubectl get secrets -n "${j}" "${i}" -o json | jq .data.release -r | base64 -d | base64 -d | gunzip | jq '.chart.templates[] | select(.name == "templates/ingress.yaml") | .data' -r | base64 -d | grep networking.k8s.io/v1beta1 && echo "${i}"
  done
done

Updating Offending Helm Manifests

Armed with the list of Helm manifests with deprecated API versions, it is easy to create a Kubernetes patch to update the secret in place. Using kubectl patch secret allows for the Helm manifest data to be updated in place.

The below example shows setting a shell variable PATCH_DATA to be the resulting updated Helm3 manifest with a substitution using sed to replace the deprecated API version with the supported API version. Patching the secret is also shown in the below example as the second step using kubectl.

PATCH_DATA=$(kubectl get secrets -n dummy-namespace sh.helm.release.v1.dummy-chart.v6 -o json | jq .data.release -r | base64 -d | base64 -d | gunzip | sed 's|networking.k8s.io/v1beta1|networking.k8s.io/v1|' | gzip -c | base64 | base64)

kubectl patch secret -n dummy-namespace sh.helm.release.v1.dummy-chart.v6 --type='json' -p="[{\"op\":\"replace\",\"path\":\"/data/release\",\"value\":\"$PATCH_DATA\"}]"
secret/sh.helm.release.v1.dummy-chart.v6 patched

← → Top