|
This is unreleased documentation for Network Enforcer 0.4-dev. |
Configuring Kubewarden Network Enforcer for production
This page gives configuration for Network Enforcer in a production cluster. It covers the controller, the fluent-bit DaemonSet of the Istio provider, the PriorityClass, the telemetry, and the other chart values that matter at scale.
For the provider and Kubernetes requirements, see Compatibility. For how the components work together, see Architecture. For the debug tools, see Troubleshooting.
Controller
The controller runs as a Kubernetes Deployment with leader election. One replica is active at a time. It reads flows from the data plane, writes proposals and policies, and updates the policy status.
Network Enforcer has no admission webhook. When the controller is down, learning stops and the policy status is not updated. The native policy objects that the controller already rendered stay in place, and the data plane keeps enforcing them.
Controller replicas
controller.replicas defaults to 1. Only the leader does the work. An
extra replica is a warm standby. It shortens the failover time, and it
does not add throughput.
With the Istio provider, keep controller.replicas at 1. The controller
is the OTLP server that receives the ztunnel logs from fluent-bit,
and only the leader listens on that port.
With the Calico or the Cilium provider, you can run two replicas:
controller:
replicas: 2Schedule the controller
Use controller.tolerations, controller.nodeSelector, and
controller.affinity to control where the controller pods run. For
example, spread two replicas across different nodes with
podAntiAffinity. This example matters only when controller.replicas is
more than 1:
controller:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/component: controller
topologyKey: kubernetes.io/hostnameSet controller resources
controller.resources defaults to a Guaranteed 100m CPU and 256Mi
memory, with the same requests and limits. The memory of the
controller grows with the number of distinct flows and the number of
proposals in the cluster. Watch the controller pod after the install, and
raise both values when the pod gets close to its limit. Keep requests
equal to limits, so that the pod keeps the Guaranteed class:
controller:
resources:
limits:
cpu: 250m
memory: 512Mi
requests:
cpu: 250m
memory: 512MiKeep the status sync interval
controller.wnpStatusUpdateInterval defaults to 30s. The quick start
lowers it to make the demo responsive. A shorter interval writes the
policy status more often and adds load on the API server. In production,
keep the default. See
wnpStatusUpdateInterval
latency for what the interval changes.
Add a PodDisruptionBudget for the controller
A PodDisruptionBudget (PDB) limits how many pods a voluntary disruption can remove at the same time, for example a node drain or a cluster upgrade. The chart ships a PDB template for the controller, off by default.
Set controller.pdb.enabled to true. Set minAvailable or
maxUnavailable under controller.pdb, in the same way as a PDB spec:
controller:
pdb:
enabled: true
minAvailable: 1|
If |
fluent-bit DaemonSet, Istio provider
With controller.provider.name: istio, the chart deploys fluent-bit as a
DaemonSet. One pod runs on each node. It reads the log files of
ztunnel and sends the records to the controller. A node without a
fluent-bit pod produces no proposals and no monitor-mode violations
for the pods on that node. Protect mode still blocks traffic there,
because ztunnel enforces the policy on its own.
fluent-bit scheduling
controller.provider.istio.fluentBit has its own resources,
tolerations, nodeSelector, and affinity keys. The pod also gets the
shared PriorityClass.
fluent-bit must run on every node where ztunnel runs. The chart sets a
toleration that matches every taint by default, the same default that
ztunnel itself uses. If you set
controller.provider.istio.fluentBit.tolerations to [], fluent-bit
stops tolerating taints. The pod is then missing from each tainted node
that still runs ztunnel. The chart loses the ztunnel logs from that node,
together with the proposals and the monitor-mode violations of its
workloads.
If you restrict fluent-bit with nodeSelector or affinity, make sure
that the selection still matches every node where ztunnel runs.
In the default TLS mode issuer, each fluent-bit pod receives its
certificate from the cert-manager CSI driver. Make sure that the CSI
driver runs on every node. See
Provider TLS for the
other modes.
fluent-bit security context
fluent-bit runs as root (runAsUser: 0) and mounts two read-only
hostPath volumes, /var/log and /var/lib/docker/containers, to read
the container logs of the node. It is not privileged, it does not allow
privilege escalation, and it drops all capabilities.
These two requirements come from the job, not from a gap in the chart. Every log shipper that tails node log files needs the same two requirements.
The container runtime writes the log files under /var/log/pods with
mode 0640 and owner root:root. A pod’s fsGroup does not change the
ownership of a hostPath volume, so it cannot open the files to a
non-root user. The capability CAP_DAC_READ_SEARCH is not an option
either. The baseline and the restricted Pod Security Admission
levels both forbid it. The kernel also does not activate it for a
non-root process. Reading the files needs UID 0.
The DaemonSet mounts /var/log and /var/lib/docker/containers to reach
those files from the host. No Pod Security Admission level below
privileged admits a hostPath volume. So the fluent-bit pod needs the
privileged level, and the two
recommended
policies that this page excludes next.
PriorityClass
The chart creates a PriorityClass named network-enforcer-priorityclass,
with value 1000000, and applies it to the controller, the fluent-bit,
and the collector pods. This makes the Kubernetes scheduler prefer these
pods over most application pods, and keeps the kubelet from evicting them
first under node pressure.
To reuse an existing PriorityClass, for example system-cluster-critical,
turn off the chart-managed one and point priorityClass.name at the
existing class:
priorityClass:
enabled: false
name: system-cluster-critical|
If you delete a PriorityClass, running pods that use it keep running. New pods that name the deleted class do not start. |
Admission policies and Pod Security Admission
What an admission policy accepts depends on the provider.
With the Calico or the Cilium provider, the chart deploys the controller
and the collector only. Both pods run as non-root, drop all capabilities,
and use no host namespace and no hostPath volume. They meet the
restricted Pod Security Admission level, and none of the Kubewarden Admission Controller
recommended policies rejects them. The Network Enforcer namespace can
carry pod-security.kubernetes.io/enforce: restricted.
With the Istio provider, the fluent-bit pod fails both the baseline and
the restricted level. In protect mode, two of the Admission Controller
recommended policies reject it:
| Policy | Rejects fluent-bit because |
|---|---|
|
|
|
|
Exclude the Network Enforcer namespace from these policies, with the
recommendedPolicies.skipAdditionalNamespaces value of the
Admission Controller Helm chart. The default exclusion list of that chart
does not include a Network Enforcer namespace, so add yours in every case:
recommendedPolicies:
skipAdditionalNamespaces:
- network-enforcerKeep the namespace at the privileged Pod Security Admission level, or
use audit or warn mode only.
See Security hardening for the full list of recommended policies, and Pod Security Admission for the three levels.
Telemetry
By default, the chart deploys its own OpenTelemetry collector, as
a single Deployment replica. telemetry.defaultCollector has
tolerations, nodeSelector, and affinity values, but the replica
count stays fixed at one. The collector keeps its counters in memory, so
a restart resets them. This default is enough for a basic setup, but it
is a single point of failure for the violation metrics.
For production, point Network Enforcer at an OpenTelemetry collector that your own platform operates and schedules for high availability:
telemetry:
collectorStrategy: external
externalCollector:
protocol: grpc
endpoint: otel-collector.observability.svc.cluster.local:4317
otelCollectorCertificateSecret: otel-collector-caThe Secret in otelCollectorCertificateSecret holds the CA certificate
of your collector in the key ca.crt. Without it, the controller sends
the events without TLS verification. The Secret must be in the
Network Enforcer namespace.
|
This collector receives the violation events. It is not the OTLP endpoint that receives the ztunnel logs on the Istio provider. The two hops use different ports and different certificates. See OpenTelemetry collector. |
Learning scope
The chart has no value that limits the learning scope. Learning is cluster-wide and starts when the controller starts. The provider decides which flows the controller sees:
-
With the Istio provider, only the namespaces that carry the label
istio.io/dataplane-mode=ambientsend traffic through ztunnel. That label is the learning scope. -
With the Calico or the Cilium provider, every flow that the data plane reports is learned, in every namespace.
Learning stops for a workload and direction when a
WorkloadNetworkPolicy with the same name exists. You can learn the
policies in another cluster, for example staging, promote them there, and
apply the resulting policies to production as code. The controller then
writes a WorkloadNetworkPolicyProposal only for the workloads that
have no policy yet.
See Learning starts when the controller starts for what learning does not record.
Keep the flow dumper off
controller.flowDumper.enabled defaults to false. Keep it off in
production. When it is on, the chart creates a NodePort Service on port
30080 that exposes the raw flow records on every node. Turning it on or
off also changes the controller arguments, so the controller restarts.
See Flow dumper for what it shows and how to read it.
Example production configuration
This example combines the configuration from this page into one Helm values file, for the Istio provider:
controller:
provider:
name: istio
replicas: 1
resources:
limits:
cpu: 250m
memory: 512Mi
requests:
cpu: 250m
memory: 512Mi
wnpStatusUpdateInterval: 30s
flowDumper:
enabled: false
telemetry:
collectorStrategy: external
externalCollector:
protocol: grpc
endpoint: otel-collector.observability.svc.cluster.local:4317
otelCollectorCertificateSecret: otel-collector-caWith the Calico or the Cilium provider, set controller.provider.name to
calico or cilium. You can then raise controller.replicas to 2 and
turn on the PodDisruptionBudget:
controller:
provider:
name: calico
replicas: 2
pdb:
enabled: true
minAvailable: 1Apply the file at install or upgrade time:
helm upgrade --install network-enforcer kubewarden/network-enforcer \
--namespace network-enforcer \
--create-namespace \
-f production-values.yaml \
--wait