Win of the day: Some talks had really long question sessions (10+). That would be impossible in the offline conference
Loss of the day: Most presenters were not prepared to do the talk from home and had either bad pre-recorded sound and video or bad internet for the live q&a.
Somehow I got involved again into the security topics
“How This Innocent Image Had a Party in My Cluster”
Speakers: Itay Shakury, Amir Jerbi (Aqua Security)
So you encounter this three-headed dragon and start fighting it:
Head 1: The Vulnerabilities in the image.
You take your trusted static vulnerability tester (trivy, clair, Anchore), tested the OS + components, as well as the languages (PHP, Node, Python etc.).
All is green, the head is slain
Head 2: Misconfigurations
This head breathes weak cyphers, no authentication, hardcoded passwords, no SSL
You tested for that as well (again kube-bench for the rescue) and thus the second head was slain
Head 3: Malware
You summon your trusty bash-script that checks all the files in the image, hashes them and runs VirusTotal on each of them.
You’ve finally slain the dragon. It’s time to rest. As you’re resting on the grass you see a shadow looming over the horizon. And the name of it:
Evasive malware
Evasive malware hides using encryption and multi-level tar-files. It cannot be detected by static analysis tool, because it looks innocent but downloads the real malware from internet when the container is started. Usually it comes into your image by a Supply chain attacks.
Meaning you also need some kind of runtime security check. You should look for suspicious patterns and apply policies to block them. But for that you need some kind of tool that is injected deeply into the Linux kernel and logs everything.
One of such tools is the ebpf . With it you can instrument the linux calls and monitor them. AquaSecurity (the presenters) built a command-line tool on top of it called tracee, which captures the ebpf logs.
For the tracee you can pass --security-alerts flag – then it will automatically analyze the logs and you will get hints for suspicious behaviour (like WARNS when the process is trying to chown and chmod)
Another way how the evasive malware works is with something like elfexec. It basically creates executable in memory and runs it for it. This file will not be detected by static security code check and even by a tool that regularly scans normal files in the image. Tracee can detect that as well.
Of course you don’t want to run all this in your production cluster. You should have a pre-production cluster where you check all these things.
“Prometheus Introduction”
Speakers: Julius Volz (co-founder of Prometheus)
Prometheus is a well-known tool to collect metrics. I will point only to some things that were raised in the Q&A Session:
Metric names
Normally the data format has only metric name + labels e.g
api_http_requests_total{method="POST", handler="/messages"}
The speaker claimed that you can also provide a human-readable metric name so that Prometheus could give it to e.g. Grafana. It’s in early stage, not really integrated yet though.
The PromQL tutorial
Prometheus uses a special language PromQL (not SQL-style). People don’t like to use it, so the question was is there a good book on it.
The recommendation is this 2-part tutorial (by the speaker).
Prometheus Operator
There is Prometheus K8s Operator (developed by IBM) tool to run it on k8s cluster and take over some overhead (but makes it harder to diagnose errors)
Keynote: “Why End Users Are More Than Passive Consumers”
Speaker: Cheryl Hung, Director of Ecosystem, Cloud Native Computing Foundation
Simple idea. CNCF has tons of tools, but which ones are good for practical usage for the end-users?
Solution: CNCF now publishes it’s End User technology Radar
“Running K3s, Lightweight Kubernetes, in Production for the Edge and Beyond”
Speaker: Darren Shepherd (creator of K3s)
k3s is lightweight kubernetes. Single binary size, small memory size (~300mb). Low congitive load. And! Designed for production (unlike kind and minikube).
- Focused to be deployed on Edge devices. Edge – are powerful small devices (minimum 1gb memory), so things like modern sophisticated coffee machines.
- Focused on single node or 2-3 nodes cluster. No support for node autoscaling (that we got used to in cloud)
- Easy to package an app – just upload k8s manifests (yamls) in a special dir. So you can create a distro of your app by having a yaml and k3s together
- Suggests to use sqlite or MySQL instead of etcd (unclear why they chose it this way, they promised to look it up)
- You can use it as a testing environment because it’s kubernetes compliant. So if you run in on k3s it will work in k8s (if you use the standard features)
- Requires root but they work on rootless version