This is unreleased documentation for Runtime Enforcer 0.11-dev.

Emergency disable

During an incident, a Runtime Enforcer policy can block a process that your team needs to run. This page gives two ways to stop the blocking, and how to restore protection afterward without losing track of what changed.

The commands on this page use jq to filter JSON output. Install it from https://jqlang.org/ if you do not already have it.

Do not scale the controller to zero to stop a policy from blocking. The controller takes no part in enforcement, so this does not work, and it stops other things that you need during an incident. See Do not scale the controller to zero.

What enforcement depends on

Runtime Enforcer enforces a policy in two places. Each agent loads the policy into the kernel, on its own node. The spec.mode field of the policy, monitor or protect, decides whether the kernel blocks a violation. The controller takes no part in enforcement. A controller that is down does not change what the kernel blocks.

Stop blocking

Runtime Enforcer gives you two ways to stop a policy from blocking. Both are reversible, and both leave every WorkloadPolicy in place.

Way Use it when

Remove the agent from every node

You want enforcement and monitoring to stop at once, on every node, with one command.

Switch protect policies to monitor mode

A pod in your cluster carries the required-plugins annotation. Such a pod cannot start without the agent, so keep the agent running and change the policies instead.

Remove the agent from every node

Patch the agent DaemonSet with a nodeSelector that matches no node in your cluster. Replace runtime-enforcer and its namespace with your release name, if you installed under another name:

kubectl patch daemonset runtime-enforcer-agent \
  -n runtime-enforcer \
  --type merge \
  -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubewarden.io/emergency-disable":"true"}}}}}'

The DaemonSet controller removes every agent pod at once, because no node carries that label. The eBPF program is not pinned to the kernel, so it is removed with the agent process. Enforcement and monitoring both stop, on every node, at once.

While the agent is gone:

Restore the agent

To restore, remove the key that you added:

kubectl patch daemonset runtime-enforcer-agent \
  -n runtime-enforcer \
  --type merge \
  -p '{"spec":{"template":{"spec":{"nodeSelector": null}}}}'

This command clears the whole nodeSelector field. If agent.nodeSelector already held a value before the incident, patch in that value instead of null, so that you do not lose it.

The new agent pod on each node binds every running pod again before it reports ready. Protection resumes node by node, as each agent passes its startup probe.

Switch protect policies to monitor mode

Use this way when you keep the agent running. A policy in monitor mode reports a violation but does not block it. The agent applies the change without restarting any pod.

Mark each policy that you switch, so that you switch back only those policies, and no others, once the incident is over.

Mark the policies that are in protect mode

emergency.kubewarden.io/restore-mode is a convention for this page. It is not a label that Runtime Enforcer reads, and the agent and the controller ignore it. Use any key that you want, as long as every step below uses the same one.

List every policy whose spec.mode is protect, and add the label to each one:

kubectl get workloadpolicy -A -o json \
  | jq -r '.items[] | select(.spec.mode=="protect") | "\(.metadata.namespace) \(.metadata.name)"' \
  | while read -r ns name; do
      kubectl label workloadpolicy -n "$ns" "$name" emergency.kubewarden.io/restore-mode=protect
    done

Switch the marked policies to monitor mode

Patch every policy that carries the label:

kubectl get workloadpolicy -A -l emergency.kubewarden.io/restore-mode=protect -o json \
  | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' \
  | while read -r ns name; do
      kubectl patch workloadpolicy -n "$ns" "$name" --type merge -p '{"spec":{"mode":"monitor"}}'
    done

This command selects policies by the label, not by mode, so you can run it again without harm.

Make sure every policy is in monitor mode

Check the mode of every policy:

kubectl runtime-enforcer policy show protection -A

Every policy that you marked now shows Monitor. A policy in monitor mode still reports a violation, so you keep a record of what ran during the incident.

Reduce the noise during the incident

This section is optional. A policy in monitor mode still reports every violation, through two channels: the policy status, and OpenTelemetry, when telemetry is on.

Policy status

The controller reads the violation buffer of each agent on a timer. Raise controller.wpStatusUpdateInterval to read it less often:

controller:
  wpStatusUpdateInterval: 12h

A longer interval stops the status from being a live view: it does not change until the next read. When the controller reads again, each policy shows at most its 100 most recent distinct violations. Each comes with an occurrences count, so nothing floods the status. Once you restore protection, set controller.wpStatusUpdateInterval back to its usual value. See Keep the status sync interval for the default value.

OpenTelemetry events

This interval does not change the OpenTelemetry events. The agent sends each one as it happens, regardless of the controller. Filter them at your collector instead. Match on the label that you added in Mark the policies that are in protect mode, or on the time window of the incident.

Do not turn off telemetry with telemetry.collectorStrategy=none during the incident. That value change restarts every agent.

Restore protection

Once the incident is over, switch the marked policies back to protect, make sure that the switch worked, then remove the label.

Patch every policy that carries the label back to protect:

kubectl get workloadpolicy -A -l emergency.kubewarden.io/restore-mode=protect -o json \
  | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' \
  | while read -r ns name; do
      kubectl patch workloadpolicy -n "$ns" "$name" --type merge -p '{"spec":{"mode":"protect"}}'
    done

Wait for every policy to report Ready:

kubectl runtime-enforcer policy show protection -A

A policy that still shows Transitioning has not reached every agent yet. See A mode change for what the two states mean.

Once every policy is Ready, remove the label:

kubectl get workloadpolicy -A -l emergency.kubewarden.io/restore-mode=protect -o json \
  | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' \
  | while read -r ns name; do
      kubectl label workloadpolicy -n "$ns" "$name" emergency.kubewarden.io/restore-mode-
    done

Remove the label only after every policy is back to protect mode. If the restore step fails partway, the label still names the policies that you have not finished restoring.

Do not

Two actions look like they disable Runtime Enforcer, but do not.

Do not scale the controller to zero

The controller takes no part in enforcement. The agent reads WorkloadPolicy objects directly, so every policy keeps blocking while the controller is down.

A controller that is down also blocks other work. Every admission webhook of Runtime Enforcer fails closed, so a policy deletion, a proposal, or a promotion waits until the controller returns.

If you restart an agent while the controller is down, the restart does not finish. Learning is on by default, and each agent waits for the controller’s admission webhook before it starts. If an agent must restart during the incident, restore the controller first.

Do not delete a WorkloadPolicy

A validating webhook rejects the deletion of a WorkloadPolicy while any pod in its namespace still carries the runtimeenforcer.kubewarden.io/policy. There is no bypass for this check.

Forcing the deletion, for example by removing the webhook configuration first, leaves pods with a label that names a policy that no longer exists. A pod like that does not start on its next restart, unless agent.nriFailopen is true.

See also