This is unreleased documentation for Runtime Enforcer 0.11-dev.

Configuring Kubewarden Runtime Enforcer for production

This page gives configuration for Runtime Enforcer in a production cluster. It covers the scheduling, the resources, and the reliability of the controller and the agent, and the other chart values that matter at scale.

For the kernel and container runtime requirements, see Compatibility. For CPU and memory sizing by cluster size, see Resource consumption. For how the components work together, see Architecture.

Controller

The controller runs as a Kubernetes Deployment. Every controller replica serves three admission webhooks, and the webhooks use failurePolicy: Fail. When the controller is down, a promotion, a policy deletion, or a change to the runtimeenforcer.kubewarden.io/promote label waits until a controller pod answers.

Run more than one controller replica

The Helm value controller.replicas defaults to 1. The controller already turns on leader election (--leader-elect), so you can run more than one replica without conflict. The leader runs the promotion and the status-sync loop. Every replica serves the admission webhooks, so more replicas keep the webhooks available while one replica restarts.

Set controller.replicas to 2 or more:

controller:
  replicas: 2

Add a PodDisruptionBudget for the controller

A PodDisruptionBudget (PDB) limits how many pods a voluntary disruption can remove at the same time, for example a node drain or a cluster upgrade. The chart ships a PDB template for the controller, off by default.

Set controller.pdb.enabled to true. Set minAvailable or maxUnavailable under controller.pdb, in the same way as a PDB spec:

controller:
  pdb:
    enabled: true
    minAvailable: 1

If controller.replicas is 1 and controller.pdb.minAvailable is 1, the PodDisruptionBudget blocks every voluntary disruption of the controller pod, including kubectl drain. Before you turn on the PodDisruptionBudget, set controller.replicas to 2 or more.

Schedule the controller

Use controller.tolerations, controller.nodeSelector, and controller.affinity to control where the controller pods run. For example, spread controller replicas across different nodes with podAntiAffinity:

controller:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/component: controller
            topologyKey: kubernetes.io/hostname

Set controller resources

controller.resources defaults to a Guaranteed 250m CPU and 256Mi memory, with the same requests and limits. See Sizing recommendation for the values to use at your cluster size.

Agent

The agent runs as a DaemonSet, one pod on every node. Each agent pod loads the eBPF program and enforces the WorkloadPolicy of the pods on its node.

A node without an agent pod gives no protection to the pods that start there. The container runtime does not call an agent that is not connected, so the pod starts without a policy.

This differs from a pod that names a policy the agent has not loaded yet. There the agent is present, and it blocks the start by default. agent.nriFailopen controls that second case only. It is false by default. See A missing policy.

Plan the agent scheduling before you taint nodes or restrict its placement.

Tolerate tainted nodes

The chart sets no default tolerations for the agent. If a node carries a taint, for example a control-plane taint, the agent does not run there, and pods on that node run without enforcement.

Add a toleration so that the agent runs on tainted nodes too:

agent:
  tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"

Restrict node selection

Use agent.nodeSelector or agent.affinity to keep the agent off nodes whose kernel does not meet the minimum requirements. For example, keep the agent off an arm64 node with a kernel older than 6.4. The agent does not run on a node that the selector excludes. Make sure that the selector still matches every node that runs workloads:

agent:
  nodeSelector:
    kubernetes.io/os: linux

Agent rolling updates

The agent DaemonSet sets a fixed update strategy: maxUnavailable: 0 and maxSurge: 100%. You cannot change this value. During an upgrade, the new agent pod loads every policy and binds every running pod again. Only then does its startup probe pass, and only then does the old pod stop. Protection does not lapse during the upgrade. See Architecture for the full upgrade sequence.

Set agent resources

agent.resources defaults to a Guaranteed 250m CPU and 256Mi memory. The memory of each agent grows with the number of WorkloadPolicy objects in the cluster, not with the number of pods on its node. See Sizing recommendation for the values to use at your cluster size.

Agent security context

The agent needs hostPID: true and four Linux capabilities (BPF, PERFMON, SYS_RESOURCE, SYS_PTRACE) to load the eBPF program and read the cgroup tree of the node. It also mounts a hostPath volume for the NRI socket, and it does not set runAsNonRoot. The chart sets privileged: false.

An admission policy that forbids one of these fields rejects the agent pod, and the DaemonSet then has no pod on that node. In protect mode, four of the Kubewarden Admission Controller recommended policies reject the agent:

Policy Rejects the agent because

no-host-namespace-sharing

hostPID is true.

drop-capabilities

Its allowed list is empty, and the agent adds four capabilities.

do-not-share-host-paths

The NRI socket path is not under /tmp.

do-not-run-as-root

runAsNonRoot is not true.

Exclude the Runtime Enforcer namespace from these policies, with the recommendedPolicies.skipAdditionalNamespaces value of the Admission Controller Helm chart:

recommendedPolicies:
  skipAdditionalNamespaces:
    - runtime-enforcer

The namespace cattle-runtime-enforcer-system is already in the default exclusion list. When you install Runtime Enforcer under a different name, add your own namespace.

See Security hardening for the full list of recommended policies.

Both the baseline and the restricted Pod Security Admission levels reject the same fields. Keep the Runtime Enforcer namespace at the privileged level, or use audit or warn mode only. See Pod Security Admission for the three levels.

PriorityClass

The chart creates a PriorityClass named runtime-enforcer-priorityclass, with value 1000000, and applies it to the controller and the agent pods. This makes the Kubernetes scheduler prefer these pods over most application pods, and keeps the kubelet from evicting them first under node pressure.

To reuse an existing PriorityClass, for example system-node-critical, turn off the chart-managed one and point priorityClass.name at the existing class:

priorityClass:
  enabled: false
  name: system-node-critical

If you delete a PriorityClass, running pods that use it keep running. New pods that name the deleted class do not start.

Telemetry

By default, the chart deploys its own OpenTelemetry collector, as a single Deployment replica with no tolerations, nodeSelector, or affinity value. This default is enough for a basic setup, but it is a single point of failure for the violation metrics.

For production, point Runtime Enforcer at an OpenTelemetry collector that your own platform operates and schedules for high availability:

telemetry:
  collectorStrategy: external
  externalCollector:
    protocol: grpc
    endpoint: https://otel-collector.observability.svc.cluster.local:4317

See Use an external collector for the full set of values.

Restrict the learning scope

By default, learning.namespaceSelector matches every namespace, so the agent writes a WorkloadPolicyProposal for every workload in the cluster. In production, restrict the selector to the namespaces that you onboard:

learning:
  namespaceSelector:
    matchLabels:
      env: prod

Once every workload has a WorkloadPolicy, set the selector to {} to turn learning off.

You can also learn policies in another cluster, for example staging, and promote them there. If every production workload already carries the runtimeenforcer.kubewarden.io/policy label, turn learning off across the whole cluster:

learning:
  namespaceSelector: {}

See Learning mode configuration for more examples.

Keep the debugger off

debugger.enabled defaults to false. Keep it off in production. If you need it to diagnose a mismatch between the agent cache and the cluster, turn it on. Use --reuse-values, so that the change does not restart the agent:

helm upgrade --install runtime-enforcer kubewarden/runtime-enforcer \
  --namespace runtime-enforcer \
  --set debugger.enabled=true \
  --reuse-values

See Debugger for what it shows.

Require the agent before a workload starts

Add required-plugins.noderesource.dev to a workload that must never run without the agent. This annotation needs containerd 2.2 or later, or CRI-O 1.34 or later. The container runtime refuses to start the pod when the agent is not present as an NRI plugin on that node:

spec:
  template:
    metadata:
      annotations:
        required-plugins.noderesource.dev: '["runtime-enforcer-agent"]'

See Required plugins for the prerequisites and the error that a missing agent produces.

Example production configuration

This example combines the configuration from this page into one Helm values file:

controller:
  replicas: 2
  pdb:
    enabled: true
    minAvailable: 1
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/component: controller
            topologyKey: kubernetes.io/hostname

agent:
  tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"

telemetry:
  collectorStrategy: external
  externalCollector:
    protocol: grpc
    endpoint: https://otel-collector.observability.svc.cluster.local:4317

learning:
  namespaceSelector:
    matchLabels:
      env: prod

debugger:
  enabled: false

Apply it at install or upgrade time:

helm upgrade --install runtime-enforcer kubewarden/runtime-enforcer \
  --namespace runtime-enforcer \
  --create-namespace \
  -f production-values.yaml \
  --wait