Bootstrap Security in Kubernetes Deployments

Learn how to practically boostrap security controls into your kubernetes deployments

Last updated on May 1, 2024 13 min read Security

Kubernetes Security

Kubernetes, is one of the most popular and most used container orchestration tool. Kubernetes Workloads are the actual applications that are executed like a simple nginx server or maybe a cron job. Kubernetes Deployments is the most commonly used workload as it can be easily updated,scaled and managed.

The recently released Kubernetes Hardening Guide is an excellent resource that provides a proper guidance on how to effectively secure Kubernetes. The information presented in the guide clearly shows that securing and hardening kubernetes is not just the job of the Kubernetes administrator but also of the developers who are deploying their workload on the clusters.

In this blog I’ll discuss about how developers deploying Kubernetes workloads like Deployments can bootstrap security by applying some of the guidelines provided by the ‘Kubrnetes Hardening guide’.

This will be a practical hands-on guide where I shall take a simple Dockerfile and then incrementally add the security best practices to create a template Deployment manifest file which can then be reused by developers in a hurry.

Pre-Requisites

Docker is required as we’ll be building from ground up.
A single-node Kubernetes cluster like minikube should be sufficient to follow along with this guide alongwith the kubectl utility. You can use the official minikube documentation to set it up in your environment.

I am using a standalone cluster created by Docker Desktop tied to WSL2 as the backend.

This guide will assume that you have a running cluster which is accessible through the kubectl utility as shown below.

Docker Desktop

Securing Deployments

Securing the kubernetes workloads can effectively be compartimentalised into ‘Buildtime’ and ‘Runtime’ security. In order to run with the examples we’ll make use of this simple Spring Boot HelloWorld application and deploy it in Kubernetes with buildtime and runtime security applied.

Code on Github

So before starting off let’s clone this repository,build the docker container and run the application locally

git clone git@github.com:salecharohit/bootstrapsecurityinkubernetesdeployment.git
cd springbootmaven
docker build . -f Dockerfile.basic -t springbootmaven
docker run --name springboot -d -p 8080:8080 springbootmaven
curl http://localhost:8080

Expected Response:

Hello World From Spring Boot Build Using Maven on Alpine OS!

Build Time Security

Buildtime security focusses more on how the underlying containers can be build with a reduced footprint and are programmed to be executed with least possible privileges.

We’ll discuss both these approaches with a problem solution approach

Attack Surface Reduction

When building applications in a container the primary objective is to have the app run consistently and indepdently regardless of the environment be it a data center,cloud or even onpremise. However, when building these apps there is one unwritten rule that it should be a standalone application without much dependencies.

Let’s take example of our SpringBoot application. The only dependency for our application to run is that it needs a JVM or Java runtime. Anything else baked into the container is practically useless.

As an example, in our SpringBoot container , which is build on Alpine OS , we don’t have any specific need to have the requirement for the apk package manager to be installed.

docker exec -it springboot /bin/sh
apk add curl

APK installed

So let’s try to remove the apk binary and rebuild or docker image.

We’ll make use of the Dockerfile.asr at this time to rebuild our docker container which is shared below

FROM maven:3.8.1-openjdk-17-slim AS MAVEN_BUILD
WORKDIR /build/
COPY pom.xml /build/
COPY src /build/src/
RUN mvn package

FROM openjdk:17-alpine

RUN rm -f /sbin/apk && \
    rm -rf /etc/apk && \
    rm -rf /lib/apk && \
    rm -rf /usr/share/apk && \
    rm -rf rm -rf /var/lib/apk

COPY --from=MAVEN_BUILD /build/target/springbootmaven.jar /springbootmaven.jar
EXPOSE 8080
CMD java -jar /springbootmaven.jar

Lets rebuild and re-rerun

# First let's stop the previously running container
docker stop springboot

# Next let's re-build and re-run
docker build . -f Dockerfile.asr -t springbootmaven
docker run --name springboot -p 8080:8080 springbootmaven
docker run --name springboot -d -p 8080:8080 springbootmaven
curl http://localhost:8080

Now let’s try to run the apk add curl command again

docker exec -it springboot /bin/sh
apk add curl

APK removed

So we successfully got rid of the apk dependency and yet have our application running successfully !

Below are some good scripts that’ve been written specifically for hardenning Alpine OS. Pick and choose depending on your programming language and harden your base alpine image accordingly.

On the Flip-side you can also have a look at the distroless container created by google which is also very highly recommended.

https://github.com/GoogleContainerTools/distroless/tree/main/examples

Switching User Context

One might argue that if an attacker gains an RCE inside the container she might not be able to install packages like curl,wget etc … to establish her persistence.

However, we are still running as “root” user and technically it is still possible to install apk back

Lets re-run our docker container and check the privileges with which it is currently running.

docker exec -it springboot /bin/sh
whoami
ping rohitsalecha.com

Running as root

Hence, it is important that we run our container not as root but as a user with limited privileges.

Dockerfile.lpr shows addition of a few more commands that add a user and group called “boot” and assign it a working directory (Which is its home directory) I’ve also assigned numerical values to the user and group which we’ll discuss in detail in the Pod Security Context Section

FROM maven:3.8.1-openjdk-17-slim AS MAVEN_BUILD
WORKDIR /build/
COPY pom.xml /build/
COPY src /build/src/
RUN mvn package

FROM openjdk:17-alpine

# Removing apk package manager
RUN rm -f /sbin/apk && \
    rm -rf /etc/apk && \
    rm -rf /lib/apk && \
    rm -rf /usr/share/apk && \
    rm -rf rm -rf /var/lib/apk

# Adding a user and group called "boot"
RUN addgroup boot -g 1337 && \ 
    adduser -D -h /home/boot -u 1337 -s /bin/ash boot -G boot

# Changing the context that shall run the below commands with User "boot" instead of root
USER boot
WORKDIR /home/boot

# By default even in a non-root context, Docker copies the file as root. Hence its best practice to chown
# the files being copied as the user. https://stackoverflow.com/a/44766666/1679541
COPY --chown=boot:boot --from=MAVEN_BUILD /build/target/springbootmaven.jar /home/boot/springbootmaven.jar
EXPOSE 8080
CMD java -jar /home/boot/springbootmaven.jar

Lets rebuild and re-rerun

# First let's stop the previously running container
docker stop springboot

# Next let's re-build and re-run
docker build . -f Dockerfile.lpr -t springbootmaven
docker run --name springboot -d -p 8080:8080 springbootmaven
curl http://localhost:8080

Now let’s try to run the whoami command and check whats the privileges with which the container is now running

docker exec -it springboot /bin/sh
whoami
ping rohitsalecha.com

Running as boot

Runtime Security

Now that we’ve got a good level of confidence in the build-time security wherein we’ve learnt to remove the packages and also update the user context to run the container with limited privileges. These security features are applied when we are building the docker container however we also need to focus on the security posture of the container when it is running in the Kubernetes environment which we’ll explore below.

Before we start of with securing our Kubrnetes deployment let’s run our application on our Kubernetes cluster by first pushing our docker container to hub.docker.com. You can use this guide to get started for the same

docker build . -f Dockerfile.lpr -t springbootmaven
docker tag springbootmaven salecharohit/springbootmaven
docker push salecharohit/springbootmaven
docker run -d -p 8080:8080 --name springboot salecharohit/springbootmaven
curl http://localhost:8080

Now that our docker image is ready let’s apply our kubernetes-basic.yaml file that will deploy this application and also a service that will help us connect to it.

# Create Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: boot
---
# Create SpringBoot Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
---
# Create Service for SpringBoot Deployment
apiVersion: v1
kind: Service
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: springbootmaven

Next let’s deploy our Kubernetes manifests using the below commands

kubectl apply -f kubernetes-basic.yaml
kubectl get deploy -n boot
# Run a temporary container that will only curl our bootservice
kubectl run -it testpod --image=radial/busyboxplus:curl --restart=Never --rm -- curl http://springbootmaven.boot.svc.cluster.local:8080

Expected Output:
Hello World From Spring Boot Build Using Maven on Alpine OS!pod "testpod" deleted

Running Basic Deployment

Service Account Tokens

If a Pod needs to communicate with the Kubernetes API-Server it needs Service Account Tokens for authentication.

By default every pod gets assigned a service account token which is mounted on /var/run/secrets/kubernetes.io/serviceaccount/token. Lets view this in practice by deploying our SpringBoot app

kubectl get pods -n boot
kubectl exec -it springbootmaven-7d7c5c8597-mndv9 -n boot -- /bin/sh
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -k -H "Authorization:Bearer $TOKEN" https://kubernetes.docker.internal:6443/version

Kubernetes Service Account Token Default

An RCE vulnerability on your application can leak this access token to the attacker which she can abuse to read-write resources in the same namespace or even have a global read permissions.

The resolution for this issue is two-fold depending upon the situation.

Pods donot need any access to the API-Server
Pods need access to the API-Server

Pods that donot need access to the API-Server

This situation is farily simple to solve by simply adding two lines to the kubernetes manifest file as shown below

      serviceAccountName: ""
      automountServiceAccountToken: false

The complete deployment file kubernetes-nosa.yaml is as follows

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
      serviceAccountName: ""
      automountServiceAccountToken: false

Let’s check if the service account token is now mounted or not

# Ensure our previous deploy is deleted. 
kubectl delete ns boot

# Apply with no service account token
kubectl apply -f kubernetes-nosa.yaml
kubectl get pods -n boot
kubectl exec -it springbootmaven-5568b9874f-8nml8 -n boot -- /bin/sh
cat /var/run/secrets/kubernetes.io/serviceaccount/token

As can be seen from the image the default service account token is no longer mounted.

Kubernetes Service Account Token Default

Pods that need access to the API-Server

In this situation we need to create a ServiceAccount,Role and RoleBinding that maps the ServiceAccount to the Role. The below Kubernetes manifest

Creates a ServiceAccount called bootserviceaccount to a specific namepspace i.e. boot
Creates a Role called bootservicerole with only privileges to view running pods
Creates a RoleBinding called bootservicerolebinding
Mount the ServiceAccount thus created using the following lines in the Deployment

    ---
          spec:
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
      serviceAccountName: bootserviceaccount
    ---

This shall allow to only read pods in the “boot” namespace.

The complete deployment file kubernetes-withsa.yaml is as follows

# Create Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: boot
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bootserviceaccount
  namespace: boot
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bootservicerole
  namespace: boot
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bootservicerolebinding
  namespace: boot
subjects:
  - kind: ServiceAccount
    name: bootserviceaccount
    namespace: boot
roleRef:
  kind: Role
  name: bootservicerole
  apiGroup: rbac.authorization.k8s.io
---
# Create SpringBoot Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
      serviceAccountName: bootserviceaccount
---
# Create Service for SpringBoot Deployment
apiVersion: v1
kind: Service
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: springbootmaven

Let’s apply and check if our application is running fine

# Ensure our previous deploy is deleted. 
kubectl delete ns boot

kubectl apply -f kubernetes-withsa.yaml
kubectl run -it testpod --image=radial/busyboxplus:curl --restart=Never --rm -- curl http://springbootmaven.boot.svc.cluster.local:8080

Pod Security Contexts

Though we’ve configured our base docker image to run with non-root privileges however, there are still few more configurations that need to be added as security best practices. These are

Restricting the capabilities of the container and the pod
Disabling Privilege Escalation
Configuring the container to run with a specific uid/gid created earlier in our Dockerfile.lpr

In the kubernetes manifest files there are two types of “SecurityContexts” defined.

Running at Pod-Level which will be applied to all containers running in this pod

      ---
      securityContext:
        fsGroup: 1337
        runAsNonRoot: true
        runAsUser: 1337
      containers:
      ---

Running at Container-level

      ---
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          runAsUser: 1337
          capabilities:
            drop: ["SETUID", "SETGID"]
      serviceAccountName: ""
      automountServiceAccountToken: false
      ---

The complete deployment file kubernetes-ps.yaml embedded with the PodSecurity contexts is below

# Create Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: boot
---
# Create SpringBoot Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      securityContext:
        fsGroup: 1337
        runAsNonRoot: true
        runAsUser: 1337
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          runAsUser: 1337
          capabilities:
            drop: ["SETUID", "SETGID"]
      serviceAccountName: ""
      automountServiceAccountToken: false
---
# Create Service for SpringBoot Deployment
apiVersion: v1
kind: Service
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: springbootmaven

Let’s apply and test if our application is running

# Ensure our previous apply is deleted
kubectl delete ns boot
kubectl apply -f kubernetes-ps.yaml
kubectl run -it testpod --image=radial/busyboxplus:curl --restart=Never --rm -- curl http://springbootmaven.boot.svc.cluster.local:8080
kubectl get pods -n boot
kubectl exec -it springbootmaven-56c64ff85-mqz2z -n boot -- /bin/sh
whoami
id
ping google.com

Kubernetes Low Privilege

You can drop more capabilities as per your requirements. The complete list of capabilities can be found here

https://github.com/torvalds/linux/blob/master/include/uapi/linux/capability.h

Features like AppArmor,SecComp etc … require additional configurations of the control plane components and hence I’ve limited my discussion to out-of-the-box features that can be easily activated and ensure good level of security assurance.

Immutable File-Systems

Applications running in a containerised environment seldom write data as it practically goes against the logic of having an immutable system. However, at times it maybe needed for caching or temporary swapping/processing of files. Hence, to provide this functionality to the developer we can mount an emptyDir as an ephemeral volume which is lost once the container is killed.

With this in place we can also add another security context attribute called “readOnlyRootFilesystem” and set it as true since the application running inside the container no longer needs to write anywhere on the file-system other than the ‘tmp’ directory.

The above requirements can be configured as shown below.

  ---
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /tmp
          name: tmp
      volumes:
      - emptyDir: {}
        name: tmp
  ---

The complete deployment file kubernetes-rofs.yaml is as follows

# Create Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: boot
---
# Create SpringBoot Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      securityContext:
        fsGroup: 1337
        runAsNonRoot: true
        runAsUser: 1337
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          privileged: false
          runAsUser: 1337
          capabilities:
            drop: ["SETUID", "SETGID"]
        volumeMounts:
        - mountPath: /tmp
          name: tmp
      serviceAccountName: ""
      automountServiceAccountToken: false
      volumes:
      - emptyDir: {}
        name: tmp

---
# Create Service for SpringBoot Deployment
apiVersion: v1
kind: Service
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: springbootmaven

Let’s apply and test if our application is running

# Ensure our previous apply is deleted
kubectl delete ns boot

kubectl apply -f kubernetes-rofs.yaml
kubectl run -it testpod --image=radial/busyboxplus:curl --restart=Never --rm -- curl http://springbootmaven.boot.svc.cluster.local:8080
kubectl get pods -n boot
kubectl exec -it springbootmaven-56c64ff85-mqz2z -n boot -- /bin/sh
pwd
touch test

Read-Only File-System

Conclusion

We’ve learnt what are the different controls we can embed in our containerised application and also looked at how to enable run-time protection mechanisms that can atleast make things difficult for an external attacker to gain foothold into our containerised systems.

The kubernetes-rofs.yaml can serve as a good template for developers to containerise their applications with default security features enabled while deploying in a Kubernetes environment.

Offcourse the Dockerfile needs to be created for the specific applications but for that purpose I’ve collected a few of them here

https://github.com/salecharohit/dockerfilesrepo

References

Pre-Requisites

Securing Deployments

Build Time Security

Attack Surface Reduction

Switching User Context

Runtime Security

Service Account Tokens

Pod Security Contexts

Immutable File-Systems

Conclusion

References

devops kubernetes security

Rohit Salecha

Security Engineering

Rohit Salecha is a technology geek who loves to explore anything that runs and understands binary. As a security engineer he is passionate about learning the length,breadth and depth of technology. Being more on the defensive side he has evangelised secure software development at various organizations for more than a decade. He is ridiculously driven by “everything as code” mantra and strongly believes that security team must strive towards making themselves irrelevant.