Federate Kubernetes with AWS IAM using OIDC

Introduction

At reecetech, we want to consume AWS services from the pods running in our Kubernetes clusters. The Kubernetes clusters are predominantly hosted on-prem in VMware, although there are some worker nodes that run on EC2 instances. In order to use AWS services, the pods are required to be authenticated with AWS.

We used OpenID connect to federate the identity pods have in our Kubernetes clusters (RBAC) to AWS IAM. This allows the pods to be able to assume a role in AWS using the AWS STS AssumeRoleWithWebIdentity API call. There are open-source projects (Kube2Iam, KIAM) that provide similar functionality, however, we were motivated to try OpenID connect based federation so that we did not have to maintain an extra piece of software to provide the functionality. We also could have used our Hashicorp Vault infrastructure to broker IAM roles, but again, we were motivated to try and not be running any extra side-car pods.

This will make things easier as most AWS SDK’s will automatically make the AssumeRoleWithWebIdentity API call when they detect the presence of the environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE.

Having this federated functionality available means reecetech can stop using AWS API keys injected into the pods via Kubernetes secrets. No more worrying about API key rotation or managing hundreds of AWS IAM users.

At the time of writing (December 2020), there is a lack of information on the Internet regarding Kubernetes to AWS federation via OpenID connect. We hope that this article can help disseminate knowledge about this type of federation more widely.

How to

Configure Kubernetes for federation

apiserver

Note: We assume you’ve already got, or are able to generate some signing keys.

The Kubernetes apiserver needs to have some extra flags added to the runtime process. This is usually done in the kubeadm-config configuration map. Changes in the configuration map are applied to the apiserver pods on the next time a master node joins the cluster (using kubeadm join).

There are 6 flags that need to be set. At the time of writing, we have tested on Kubernetes 1.18 and 1.19.

api-audiences: kubernetes.svc.default  # doesn't matter what it is
feature-gates: ServiceAccountIssuerDiscovery=true
service-account-api-audiences: cluster.identifier.kubernetes.domain  # need to tell AWS this, make it unique to the cluster
service-account-issuer: https://apiserver.domain/cluster/identifier  # need to tell AWS this
service-account-jwks-uri: https://apiserver.domain/cluster/identifier/openid/v1/jwks
service-account-signing-key-file: /etc/kubernetes/pki/sa.key  # your signing key (this flag can be repeated)

Publishing to S3 (optional)

We do not expose our apiserver pods to the Internet. Instead, to allow AWS to access the URLs given in service-account-issuer and service-account-jwks-uri we created a public S3 bucket, and synchronise the data from those two URLs to the S3 bucket. Those flags in our clusters look something like this:

service-account-issuer: https://oidc-bucket.s3-ap-southeast-2.amazonaws.com/cluster/identifier
service-account-jwks-uri: https://oidc-bucket.s3-ap-southeast-2.amazonaws.com/cluster/identifier/openid/v1/jwks

To keep S3 up-to-date with the data being served from the apiservers, we use our CI system to periodically: 1. cURL the URLs 1. store the result in a git repository 1. synchronise the data to S3

RBAC

The apiserver applies RBAC rules to these new endpoints configured in the flags, so you also need to allow anonymous read-access to the endpoints.

Here’s the cluster role binding to apply:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: oidc-reviewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:service-account-issuer-discovery
subjects:
  - kind: Group
    name: system:unauthenticated

Configure AWS for federation

Configuring AWS for the federation is done via API calls or the GUI. Unfortunately it’s not supported in CloudFormation.

Follow the AWS documentation, and if you’re using the GUI, it should look something like this:

Configuring federation in the AWS GUI console

Configure the AWS IAM role

Within AWS there needs to be an IAM role for the pods to assume. Ideally, each different application would have its own role, following the principle of granting least privileges.

The IAM role has two parts: what the role can do, and, who can assume the role.

Here is a CloudFormation template that would create a read-only role, that can be assumed by our newly federated pods:

---
AWSTemplateFormatVersion: 2010-09-09
Description: An example stack that creates an IAM role assumable by OpenID connect federation
Parameters:
  OidcIdentifier:
    Type: String
    Description: The OpenID connect identifier that AWS is configured with for the federated cluster
    Default: apiserver.domain/cluster/identifier
  PodNamespace:
    Type: String
    Description: The Kubernetes namespace that the pods have their service account in
    Default: default
  PodServiceAccount:
    Type: String
    Description: The Kubernetes service account that the pods will be using
Resources:
  BlogExampleRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: blog-example-iam-role
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/ReadOnlyAccess
      MaxSessionDuration: 3600  # 1 hour
      AssumeRolePolicyDocument: !Sub |
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Principal": {
                "Federated": "arn:aws:iam::${AWS::AccountId}:oidc-provider/${OidcIdentifier}"
              },
              "Condition": {
                "StringEquals": {
                  "${OidcIdentifier}:sub": "system:serviceaccount:${PodNamespace}:${PodServiceAccount}"
                }
              }
            }
          ]
        }

After reading the above example, you should be crying “why, oh why, do you have JSON in YAML!?”. The reason is that we need to have a dynamic key in the StringEquals condition, and that’s not possible without resorting to using a string for the entire AssumeRolePolicyDocument. Credit to gjenkins8 on Github.

Note the granular access granted. Only pods that use the service account given in the StringEquals can assume the role.

Configure the pods

Environment variables

To configure our pods to have the AWS related environment variables available at run-time, we need to add data to the pod’s configuration map.

In the example below, the pod is part of a replica set, that is created by a deployment. We have trimmed the example down to the additional lines required to set the AWS relevant environment variables.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: "blog-example-config-map"
data:
  AWS_DEFAULT_REGION: ap-southeast-2
  AWS_ROLE_ARN: "arn:aws:iam::000000000000:role/blog-example-iam-role"  # matches `RoleName` in the CloudFormation template
  AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/k8s-to-iam/serviceaccount/token  # matches the mount in the pod spec

Service account

Within Kubernetes there needs to be an service account for the pods to use - else they are allocated default. Ideally, each different application would have its own service account. Recall that we recommend each application has its own IAM role - in this case there should be a 1:1 mapping between the IAM role and the Kubernetes service account for an application. This is again following the principle of granting least privileges.

You are not required to bind the service account to any Kubernetes role.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: "blog-example-service-account"

Projected mount

To configure our pods to have the OpenID connect authentication data available at run-time, we need to add a projected mount to the pod specification.

In the example below, the pod is part of a replica set, that is created by a deployment. We have trimmed the example down to the additional lines required to provide the projected mount.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "blog-example"
  labels:
    app: "blog-example"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "blog-example"
  template:
    metadata:
      labels:
        app: "blog-example"
    spec:
      serviceAccountName: "blog-example-service-account"
      containers:
        - name: "blog-example"
          image: centos:7
          command: ["sleep"]
          args: ["86400"]
          envFrom:
            - configMapRef:
                name: "blog-example-config-map"
          volumeMounts:
            - mountPath: "/var/run/secrets/k8s-to-iam/serviceaccount/"
              name: aws-token
      volumes:
        - name: aws-token
          projected:
            sources:
              - serviceAccountToken:
                  path: token
                  expirationSeconds: 43200
                  audience: "cluster.identifier.kubernetes.domain"  # matches `service-account-api-audiences` flag on the apiserver

It is worth noting that in our environment, we noticed an issue when running under version 1.18.6 of Kubernetes where the token was mounted into the container with the owner root and the permissions bitmask of 0600 (read/write owner, no access to anybody else).

As far as we could tell, the volume permissions were set correctly to 0644 (or 420 decimal - thanks JSON). For reasons unknown to us, upon upgrading the cluster to version 1.19.5 as part of unrelated maintenance the token was suddenly mounted with correct permissions.

Reviewing the release notes didn’t point to any GitHub issue that might explain the change in functionality, so it’s possible 1.18.6 will work correctly in other environments, but if you do encounter this particular issue it is worth upgrading.

Testing / Demo

With all this configured correctly, you should now be able to assume a role from a pod in your Kubernetes cluster.

We can see that the JWT is mounted at /var/run/secrets/k8s-to-iam/serviceaccount/token and is world readable:

$ ls -la /var/run/secrets/k8s-to-iam/serviceaccount/token
lrwxrwxrwx 1 root root 12 Jan 29 08:37 /var/run/secrets/k8s-to-iam/serviceaccount/token -> ..data/token

$ ls -la /var/run/secrets/k8s-to-iam/serviceaccount/..data/token
-rw-r--r-- 1 root root 1073 Jan 29 08:37 /var/run/secrets/k8s-to-iam/serviceaccount/..data/token

$ cat /var/run/secrets/k8s-to-iam/serviceaccount/token
eyJhb[..snip..]hWSQ

Decoding this JWT, we can see that the subject is our pod’s service account, and the audience is as expected:

{
  "aud": [
    "cluster.identifier.kubernetes.domain"
  ],
  "exp": 1608288436,
  "iat": 1608245236,
  "iss": "https://oidc-bucket.s3-ap-southeast-2.amazonaws.com/cluster/identifier",
  "kubernetes.io": {
    "namespace": "my-namespace",
    "pod": {
      "name": "blog-example-9fc8c46f8-vj8b4",
      "uid": "c1d2fa28-5da5-47b2-b623-dc6f0833b428"
    },
    "serviceaccount": {
      "name": "blog-example-service-account",
      "uid": "0588771a-cd56-420a-9cec-787a792f7567"
    }
  },
  "nbf": 1608245236,
  "sub": "system:serviceaccount:my-namespace:blog-example-service-account"
}

Finally, we should be able to use the AWS CLI to verify we’re able to assume the role correctly:

# yum install python3       # as root
# pip3 install -U awscli    # must be newer than 1.16.232

# useradd foo && su -p foo  # demo non-root user
$ unset HOME

$ aws --version
aws-cli/1.18.222 Python/3.6.8 Linux/3.10.0-1160.el7.x86_64 botocore/1.19.62

$ aws sts get-caller-identity
{
    "UserId": "AROAXXXXXXXXXXXXXXXXX:botocore-session-1608249397",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:role/blog-example-iam-role/botocore-session-1608249397"
}

Final notes

Key rotation

At the time of writing, we have not tested rotating the signing key.

However, the apiserver flag service-account-signing-key-file can be given multiple times to facilitate key rotation. When given multiple times, then the resulting JWKS (JSON Web Key Sets) contains multiple keys. This allows a new key to be rolled out across multiple apiservers without restarting them all at once.

amazon-eks-pod-identity-webhook

It’s worth mentioning AWS provide a webhook to do the configuring the pod part of this article (environment variables & projected mount). The webhook is provided on Github.

We don’t use the webhook due to the way it creates a certificate authority, and it means running more software. We have achieved OpenID connect federation between Kubernetes and AWS by configuration only - no extra pods.

Conclusion

We find that using OpenID connect federation between (on-prem) Kubernetes and AWS to be a powerful and secure method for making AWS services available to our Kubernetes pods.