Using the Kubernetes Python Client with AWS

As someone who normally just uses kubectl and helm to talk to my Kubernetes clusters, the idea of scripting modifications to my Kubernetes cluster was exciting!! I cracked open the kubernetes-python client and started playing.

TL;DR;

We use the boto3, eks-token, and kubernetes python packages to talk to an EKS cluster without depending on kubeconfig.

Why

For those of us interactively building and maintaining kubernetes resources, helm or kubectl become our bread and butter. They provide very nice CLI interfaces and have all the bells and whistles one could ask for!

Moreover, when it comes to AWS (Amazon Web Services) and their EKS (Elastic Kubernetes Service) clusters, they handle authentication smoothly and easily via a handy CLI command and the kubeconfig file (usually stored at ~/.kube/config:

aws eks update-kubeconfig --alias mycluster

(Tip: Don't forget the --alias flag!)

My assume-role settings and environment variables integrate nicely from my shell: everything is dandy!

However, in this case, I want to build an app / software program that itself accesses, modifies, and maintains resources on my Kubernetes cluster. Of course, one can script a solution around these CLIs, but that is not very portable (requires the CLIs installed) and is prone to failure because kubeconfig is user-defined, user-maintained, and hinges entirely on contexts which are arbitrary text strings.

So I headed for the kubernetes-python API client which simplifies making API requests against a Kubernetes cluster. With any luck, I will finish the day with a Python program that creates my Kubernetes resources.

NOTE: Before we get started, it is worth noting that I am presuming you have a way to authenticate to AWS. In my case, I have already provided environment variables, an instance profile, etc. that gives this program access to the AWS API as the necessary IAM role.

The Story is Auth

As with many of my stories, this story will become one largely consisting of authentication. Unraveling the "magic" of the kubeconfig file and AWS's IAM authentication was not an easy task - as I soon learned, the kubernetes-python API client also depends heavily on kubeconfig by default.

Most of the examples look something like this, and if you want to use kubeconfig, this works very well.

from kubernetes import config

config.load_kube_config()
core_v1 = core_v1_api.CoreV1Api()

(Although there is also a cool example of in-cluster config)

However, for portability, I want to bypass kubeconfig but will be running outside of a cluster. So let's see how this class works.

config.kube_config.Configuration

After a bit of flailing, fighting, and digging, I learned that you can initialize an API connection with the API endpoint and a bearer token. Basically, you initialize a configuration object directly and then use that to initialize an API client.

import kubernetes


def k8s_api_client(endpoint: str, token: str, cafile: str) -> kubernetes.client.CoreV1Api:
    kconfig = kubernetes.config.kube_config.Configuration(
        host=endpoint,
        api_key={'authorization': 'Bearer ' + token}
    )
    kconfig.ssl_ca_cert = cafile
    kclient = kubernetes.client.ApiClient(configuration=kconfig)
    return kubernetes.client.CoreV1Api(api_client=kclient)

There are other useful options in kconfig, like kconfig.proxy and kconfig.verify_ssl. Have a look at the class for more details!

Authenticate to AWS / EKS

Now we need to figure out how to get a bearer token to talk to EKS. I cannot use the magic in kubeconfig, which led me to the AWS CLI's aws eks get-token command. I chose to opt for a pure python solution in the eks-token package. You can evaluate the source for yourself here and more specifically, here.

With this module, we can acquire an EKS token easily:

import eks_token

cluster_name = 'my-eks-cluster'
my_token = eks_token.get_token(cluster_name)

NOTE: It is worth checking whether the boto3 or aws libraries provide access to this functionality directly from Python as things improve!

Now TLS

Unfortunately, the kubernetes-python client does not allow for inlining the TLS CA Certificate. As a result, we have to write it to a temp file (which by inspection is exactly what the kubeconfig approach is doing)

   
import boto3
import tempfile
import base64


def _write_cafile(data: str) -> tempfile.NamedTemporaryFile:
    # protect yourself from automatic deletion
    cafile = tempfile.NamedTemporaryFile(delete=False)
    cadata_b64 = data
    cadata = base64.b64decode(cadata_b64)
    cafile.write(cadata)
    cafile.flush()
    return cafile


bclient = boto3.client('eks')
cluster_data = bclient.describe_cluster(name=cluster_name)['cluster']
my_cafile = _write_cafile(cluster_data['certificateAuthority']['data'])

Put it all together

Armed with a bearer token and transport layer security (TLS), we now have the tools we need to succeed!!

api_client = k8s_api_client(
    endpoint=cluster_data['endpoint'],
    token=my_token['status']['token'],
    cafile=my_cafile.name
)

api_client.list_namespace()

Go wild

The first time list_namespace() returned data, I was ecstatic. Authentication successful! Now the world is your oyster.

All that's left is perusing the API client docs and refreshing AWS creds every so often!

For example, creating a configmap:

my_configmap = kubernetes.client.V1ConfigMap(
    api_version='v1',
    metadata={'name': 'my-configmap'},
    kind='ConfigMap',
    data={'my_file.txt': 'mycontent'}
)

api_client.create_namespaced_config_map(namespace='default', body=my_configmap)

# NOTE: to update a configmap, you need to
# use k8s_client.replace_namespaced_config_map
#
# If it already exists, create will give a 409 conflict

Have fun!!

The source for this article is available here (along with a few extra examples)

Outtakes

Yes, there were some outtakes:

• Standing up mitmproxy in a docker container and setting kconfig.proxy='http://localhost:8080 and kconfig.verify_ssl=False before initializing api_client so I could see the requests being made to the Kubernetes cluster and verify whether authentication was being sent
• Flailing on temp files (today, I learned that python disposes of temp files rather quickly by default)
• Trolling the internet and finding an unfortunate absence of docs on this topic. "Is this obvious to everyone else? Am I doing something wrong?" Classic questions of a lonely explorer.
• Accidentally installing the aws package into my pyenv and breaking my terminal's ability to assume roles when in certain directories.

Programming can be hard! It is so nice to know we are not alone!