How to pass the CKS exam tips and tricks with examples (Part I)

The Certified Kubernetes Security Specialist (CKS) exam is a certification that validates one’s ability to design and implement secure Kubernetes clusters. The exam is challenging and requires significant preparation to pass. The exam Syllabus can be found in the link provided blow. https://training.linuxfoundation.org/certification/certified-kubernetes-security-specialist/ . In this article we will cover the common questions types which you can expect in the CKS exam .

SAVE TIME IN EXAM WITH FEW USEFUL COMMANDS

Once you’ve access to your terminal it worth to spend spend ~1 minute to setup your environment. This will save a lot of time in your exam.

alias k=kubectl                         # will already be pre-configured
export do="--dry-run=client -o yaml"    # k create deploy nginx --image=nginx $do
export now="--force --grace-period 0"   # k delete pod x $now

CKS ( Topics and Resources)

1. Get Context : You have access to multiple clusters through kubectl contexts. Write all context names into a file /opt/contexts .

k config get-contexts -o name > /opt/contexts

# /opt/contexts
gimmic@infra-new
uat
prod
stage

Extract the certificate of user : From the kubeconfig extract the certificate of user uat and write it decoded to /opt/cert. If the user. uat is user 2 we can use the following command .

k config view --raw -ojsonpath="{.users[2].user.client-certificate-data}" | base64 -d > /opt/cert

2. Security with Falco: here we have a use a particular context to use the cluster eg.

Example : kubectl config use-context uat

Falco will is installed with default configuration on nodes eg . cluster1-node1. Connect to node using the command ssh cluster1-node1. First we can investigate Falco config in the /etc/falco directory. This is the default configuration, if we look into falco.yaml .

➜ ssh cluster1-node1
➜ root@cluster1-node1:~# service falco status
➜ root@cluster1-node1:~# cd /etc/falco
➜ root@cluster1-node1:/etc/falco# ls
#  you should see the following files
falco.yaml  falco_rules.local.yaml  falco_rules.yaml  k8s_audit_rules.yaml  rules.available  rules.d

# /etc/falco/falco.yaml
...
# Where security notifications should go.
# Multiple outputs can be enabled.

syslog_output:
  enabled: true
.....

This means that Falco is writing into syslog. We have to do 2 tasks now.

Find a Pod running image nginx which creates unwanted package management processes inside its container.

Find a Pod running image httpd which modifies /etc/passwd. Save the Falco logs for case 1 under /opt/falco.log in format: (time-with-nanoseconds, container-id, container-name,user-name. The commands to find the offending pods look like this:

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep nginx | grep process
➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep httpd | grep passwd

We need the container ID of the pod too. The following command is helpful in getting the container ID.

➜ root@cluster1-node1:~# crictl ps -id  <processid>

Eliminate offending Pods by scaling the replicas to 0 . Use Falco from command line . We can also use Falco directly from command line, but only if the service is disabled:

➜ root@cluster1-node1:~# service falco stop

Create logs in correct format

The task requires us to store logs for “unwanted package management processes” in format time,container-id,container-name,user-name. The output from falco shows entries for "Error Package management process launched" in a default format. Let's find the proper file that contains the rule and change it:

➜ root@cluster1-node1:~# cd /etc/falco/
➜ root@cluster1-node1:/etc/falco# grep -r "Package management process launched"
./falco_rules.yaml:    Package management process launched in container (user=%user.name user_loginuid=%user.loginuid .
➜ root@cluster1-node1:/etc/falco# cp falco_rules.yaml falco_rules.yaml_ori
vim falco_rules.yaml
➜ root@cluster1-node1:/etc/falco# vim falco_rules.yaml

change the file format as expected

(Example) 

# Container is supposed to be immutable. Package management should be done in building the image.
- rule: Launch Package Management Process in Container
  desc: Package management process ran inside container
  condition: >
    spawned_process
    and container
    and user.name != "_apt"
    and package_mgmt_procs
    and not package_mgmt_ancestor_procs
    and not user_known_package_manager_in_container
  output: >
    Package management process launched in container %evt.time,%container.id,%container.name,%user.name
  priority: ERROR

For all available fields we can check https://falco.org/docs/rules/supported-fields, which should be allowed to open during the exam.Next we check the logs in our adjusted format

3. Apiserver Security:

Use to context to use a particulat cluster eg kubectl config use-context workload-uatThe security investigation of the k8s cluster1 (workload-uat) states the following about the apiserver setup that it is accessible through a NodePort Service.

Change the apiserver setup so that it is only accessible through a ClusterIP Service

We can also check the Service and see its of type NodePort.

kubectl get svc

The apiserver runs as a static Pod, so we can edit the manifest. But before we do this we also create a copy in case we mess things up:

➜ root@cluster1-master1:~# cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/1_kube-apiserver.yaml

➜ root@cluster1-master1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml

Remove the line which says “— — kubernetes-service-node-port=31111”. Once the changes are made, give the apiserver some time to start up again. Check the apiserver’s Pod status and the process parameters. We need to delete the Service for the changes to take effect.

➜ root@cluster1-master1:~# kubectl -n kube-system get pod | grep apiserver
➜ root@cluster1-master1:~# ps aux | grep kube-apiserver | grep node-port

4. Pod Security Policies : Suppose we have a Deployment hacker in Namespace hello which mounts /run/containerd as a hostPath volume on the Node where its running. This means that the Pod can access various data about other containers running on the same Node. You’re asked to forbid this behavior by:

1. Enabling Admission Plugin PodSecurityPolicy in the apiserver.
2. Creating a PodSecurityPolicy named psp-mount which allows hostPath volumes only for directory /tmp.
3. Creating a ClusterRole named psp-mount which allows to use the new PSP
4. Creating a RoleBinding named psp-mount in Namespace hello which binds the new ClusterRole to all ServiceAccounts in the Namespace hello

Restart the Pod of Deployment hacker afterwards to verify new creation is prevented.

To solve this we need to run the following commands. If it mounts containerd we will see it in the volume section.

k -n hello get pod | grep hacker

k -n hello  describe pod hacker-69b6db5f5d-ldbdh  # (example)
# if it mounts  containerd we will se it in the volume section

Volumes:
  containerdata:
    Type:          HostPath (bare host directory volume)
    Path:          /run/containerd

for this Enable Admission Plugin for PodSecurityPolicy . Log in to the master node

➜ ssh cluster1-master1
➜ root@cluster1-master1:~# cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/4_kube-apiserver.yaml
➜ root@cluster1-master1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml

#   - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy      # change

Enabling the PSP admission plugin without authorizing any policies would prevent any Pods from being created in the cluster. That’s why there is already an existing PSP default-allow-all which allows everything and all Namespaces except hello use it via a RoleBinding.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp-mount
spec:
  privileged: true
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'
  allowedHostPaths:             # task requirement
    - pathPrefix: "/tmp"

k -f 4_psp.yaml create

Create the pods . So far the PSP has no effect because we gave no RBAC permission for any Pods-ServiceAccounts to use it yet. So we do:

The following command will create a ClusterRole like:

# kubectl -n hello create clusterrole psp-mount --verb=use --resource=podsecuritypolicies --resource-name=psp-mount

# kubectl -n hello create rolebinding psp-mount --clusterrole=psp-mount --group system:serviceaccounts

And for the RoleBinding we use the above command .

Test new PSP. We restart the Deployment and check the status.

➜ k -n hello rollout restart deploy hacker

We see FailedCreate and checking for Events shows more information about why. We will edit the deployment and in volume section the path will be /tmp.

k -n hello edit deploy hacker

PodSecurityPolicies can be really hard to come around at first, but once done they’re a powerful part in the security tool box. Though they’ll be replaced with something else in future K8s releases.

5. CIS Benchmark : You’re ask to evaluate specific settings of cluster against the CIS Benchmark recommendations. The tool used is kube-bench.

First we ssh into the master node run kube-bench against the master components:

ssh cluster1-master1
➜ root@cluster1-master1:~# kube-bench run --targets=master

# EXAMPLES
...
== Summary ==
41 checks PASS
13 checks FAIL
11 checks WARN
0 checks INFO

We see some passes, fails and warnings. Let’s check the required task (1) of the controller manager.

On the master node ensure (correct if necessary) that the CIS recommendations are set for:

1. The --profiling argument of the kube-controller-manager
2. The ownership of directory /var/lib/etcd

On the worker node ensure (correct if necessary) that the CIS recommendations are set for:

1. The permissions of the kubelet configuration /var/lib/kubelet/config.yaml
2. The --client-ca-file argument of the kubelet

➜ root@cluster1-master1:~# kube-bench run --targets=master | grep kube-controller -A 3
➜ root@cluster1-master1:~# vim /etc/kubernetes/manifests/kube-controller-manager.yaml

Edit the file make changes in the file (— — profiling=false # add) . We wait for the Pod to restart, then run kube-bench again to check if the problem was solved.

Next task (2) is to check the ownership of directory /var/lib/etcd, so we first have a look:

➜ root@cluster1-master1:~# ls -lh /var/lib | grep etcd

If the owner-ship looks like user root and group root. change it to etcd:etcd

➜ root@cluster1-master1:~# chown etcd:etcd /var/lib/etcd

Number 3

To continue with number (3), we’ll head to the worker node and ensure that the kubelet configuration file has the minimum necessary permissions as recommended:

➜ ssh cluster1-worker1

➜ root@cluster1-worker1:~# kube-bench run --targets=node

EXAMPLE
...
== Summary ==
15 checks PASS
10 checks FAIL
2 checks WARN
0 checks INFO

Also here some passes, fails and warnings. We check the permission level of the kubelet config file. If we get 777 it is highly permissive access level and not recommended by the kube-bench guidelines:

➜ root@cluster1-worker1:~# chmod 644 /var/lib/kubelet/config.yaml

]Finally for number (4), let’s check whether --client-ca-file argument for the kubelet is set properly according to kube-bench recommendations:

➜ root@cluster1-worker1:~# ps -ef | grep kubelet

➜ croot@cluster1-worker1:~# vim /var/lib/kubelet/config.yaml

If theclientCAFile points to the location of the certificate, which is correct.

# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt

6. Verify the Platform Binaries

There are Kubernetes server binaries located at /opt/binaries. You're provided with the following verified sha512 values for these:

kube-controller-manager (Verified Binary ) 60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60

Delete those binaries that don’t match with the sha512 values above. We can actually compare everything properly before. Let’s look at kube-controller-manager again.

➜ cd /opt/binaries 
ls
kube-controller-manager  kube-proxy  
➜ sha512sum kube-controller-manager > compare
➜ vim compare

Edit to only have the provided hash and the generated one in one line each. compare the two hashes and the delete the binary if they are not alike.

➜ cat compare | uniq

These are the few example and questions which you might expect in the exam. More to come in the next blog.