Skip to content

Step 5: Kubernetes with Envoy Gateway

Duration: ~30 minutes Goal: Run Envoy on Kubernetes using the Gateway API and understand how endpoints update dynamically when pods scale.

What you will learn

  • Why static Envoy configs don't scale on Kubernetes
  • How xDS (the Envoy discovery API) enables dynamic configuration
  • How to install and use Envoy Gateway on Kubernetes
  • The Kubernetes Gateway API resources: GatewayClass, Gateway, GRPCRoute
  • How BackendTrafficPolicy maps to the cluster settings from Step 4
  • How to inspect the generated xDS config for debugging

Concepts

Why not static config on Kubernetes?

In the previous steps you hardcoded backend addresses in envoy.yaml. In Kubernetes, backend pod IPs change constantly — they change when:

  • A pod restarts (gets a new IP)
  • A deployment scales up/down
  • A node drains for maintenance
  • A rollout replaces pods

You cannot reload Envoy for every pod change. Instead, Envoy uses the xDS API to receive dynamic updates from a control plane.

The xDS API

xDS is a family of gRPC APIs that Envoy subscribes to. A control plane streams updates to Envoy whenever configuration changes:

API Controls Static equivalent
LDS Listeners static_resources.listeners
RDS Routes route_config inside HCM
CDS Clusters static_resources.clusters
EDS Endpoints (pod IPs) load_assignment inside a cluster
SDS Secrets (TLS certs) Inline tls_context

For Kubernetes, you never write xDS directly. A control plane — like Envoy Gateway — watches Kubernetes resources and translates them to xDS calls automatically.

xDS v3 is the current and only supported API version. All the type.googleapis.com/ URLs you've seen throughout this tutorial are xDS v3 proto types. When you see config generated by Envoy Gateway (e.g. in config_dump), it will use the same proto types as the hand-written configs in Steps 1–4.

Envoy Gateway architecture

You (kubectl apply) → Kubernetes API
                           ↓
                    Envoy Gateway        ← control plane
                    (watches resources)
                           ↓ xDS gRPC stream
                    Envoy proxy          ← data plane (handles real traffic)
                    (managed pod)

Envoy Gateway uses the standard Kubernetes Gateway API, which is now stable (v1) as of Kubernetes 1.28. It replaces the older Ingress resource and supports gRPC natively.

Key Gateway API resources

Resource Role Analogous to
GatewayClass Which controller handles gateways IngressClass
Gateway An Envoy instance (listener config) Listener in envoy.yaml
GRPCRoute Route gRPC traffic to backends Route + virtual host
BackendTrafficPolicy Cluster settings (health, circuit breaking) Cluster in envoy.yaml

appProtocol: kubernetes.io/h2c

This annotation on a Kubernetes Service port tells Envoy Gateway to use HTTP/2 cleartext to that backend — the equivalent of http2_protocol_options: {} from Step 3. Without it, Envoy falls back to HTTP/1.1 and gRPC breaks.

h2c stands for HTTP/2 Cleartext (unencrypted HTTP/2). The standard values for appProtocol that Envoy Gateway recognizes are:

appProtocol Upstream protocol
(omitted) HTTP/1.1
kubernetes.io/h2c HTTP/2 cleartext
kubernetes.io/ws WebSocket over HTTP/1.1
kubernetes.io/wss WebSocket over TLS

The GRPCRoute matching semantics

A GRPCRoute rule's matches section works similarly to Envoy's route match field:

rules:
  - matches:
      - method:
          type: Exact         # or Prefix (default)
          service: helloworld.Greeter
          method: SayHello    # optional: omit to match all methods
    backendRefs:
      - name: grpc-primary-svc
        port: 50051
  • type: Exact — match only this exact service name and (if given) method name.
  • type: Prefix — match any service name starting with the given prefix.
  • Omitting matches entirely routes all gRPC traffic to the backend refs.

Under the hood, Envoy Gateway translates this to an Envoy route with a grpc: {} matcher and a path prefix matching /<service>/<method>.

Setup

1. Create a local Kubernetes cluster

kind create cluster --name envoy-tutorial

Verify:

kubectl get nodes

2. Install Envoy Gateway

helm install eg \
  oci://docker.io/envoyproxy/gateway-helm \
  --version v1.1.0 \
  -n envoy-gateway-system \
  --create-namespace

kubectl wait --timeout=5m \
  -n envoy-gateway-system \
  deployment/envoy-gateway \
  --for=condition=Available

3. Install the Gateway API CRDs (if not bundled)

# Check if GatewayClass CRD exists
kubectl get crd gatewayclasses.gateway.networking.k8s.io 2>/dev/null \
  || kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml

Kubernetes manifests

Create a directory k8s/:

mkdir k8s

k8s/backend.yaml

Deploys 2 primary replicas and 1 backup replica:

# --- Primary deployment ---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grpc-primary
spec:
  replicas: 2
  selector:
    matchLabels:
      app: grpc-backend
      tier: primary
  template:
    metadata:
      labels:
        app: grpc-backend
        tier: primary
    spec:
      containers:
        - name: server
          image: your-registry/grpc-server:latest  # built from Step 3/4
          ports:
            - containerPort: 50051
          readinessProbe:
            grpc:
              port: 50051
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            grpc:
              port: 50051
            initialDelaySeconds: 15
            periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: grpc-primary-svc
spec:
  selector:
    app: grpc-backend
    tier: primary
  ports:
    - port: 50051
      targetPort: 50051
      appProtocol: kubernetes.io/h2c  # (1)
---
# --- Backup deployment ---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grpc-backup
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grpc-backend
      tier: backup
  template:
    metadata:
      labels:
        app: grpc-backend
        tier: backup
    spec:
      containers:
        - name: server
          image: your-registry/grpc-server:latest
          ports:
            - containerPort: 50051
          readinessProbe:
            grpc:
              port: 50051
---
apiVersion: v1
kind: Service
metadata:
  name: grpc-backup-svc
spec:
  selector:
    app: grpc-backend
    tier: backup
  ports:
    - port: 50051
      targetPort: 50051
      appProtocol: kubernetes.io/h2c
  1. appProtocol: kubernetes.io/h2c is how you tell Envoy Gateway "connect to this service with HTTP/2 cleartext." This is the Kubernetes equivalent of http2_protocol_options: {} from Step 3. Without it you will get gRPC errors.

k8s/gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: grpc-gateway
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: grpc
      protocol: HTTP
      port: 80

k8s/grpcroute.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: greeter-route
spec:
  parentRefs:
    - name: grpc-gateway
  rules:
    - matches:
        - method:
            service: helloworld.Greeter  # (1)
      backendRefs:
        - name: grpc-primary-svc
          port: 50051
          weight: 100
  1. You can match on the gRPC service name and method name. service: helloworld.Greeter matches all methods of that service. Omit matches entirely to match all gRPC traffic.

k8s/traffic-policy.yaml

This is the Envoy Gateway equivalent of the cluster settings from Step 4:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: grpc-backend-policy
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: GRPCRoute
    name: greeter-route
  circuitBreaker:  # (1)
    maxConnections: 1024
    maxPendingRequests: 1024
    maxParallelRequests: 1024
    maxParallelRetries: 3
  healthCheck:
    passive:  # (2)
      consecutiveGatewayErrors: 5
      interval: "10s"
      baseEjectionTime: "30s"
      maxEjectionPercent: 50
  retry:  # (3)
    numRetries: 3
    retryOn:
      triggers:
        - ConnectFailure
        - Reset
        - Retriable4xx
        - Unavailable
  1. Circuit breaker settings — same concepts as Step 4, just expressed in the Gateway API resource format.
  2. Passive health check = outlier detection. There is no active health check in BackendTrafficPolicy today; Envoy Gateway uses the pod readiness probe for that.
  3. Retry policy targeting gRPC-specific failure modes.

Priority-based failover in Kubernetes

The Kubernetes Gateway API GRPCRoute does not natively support priority-based failover (priority 0 → priority 1). Envoy Gateway is adding this capability through the BackendRef weight mechanism and custom resources. For now, the cleanest approach is:

  • Use weight: 100 for primaries and weight: 0 for backup in the GRPCRoute
  • Use a separate monitoring system (e.g. a health controller) to flip weights when primaries are all down
  • Or use Envoy Gateway's EnvoyExtensionPolicy to inject raw Envoy config (advanced)

The priority group approach from Step 4 works perfectly in self-managed Envoy deployments (outside the Gateway API).

Deploy and test

Apply all manifests

kubectl apply -f k8s/

Wait for the Gateway to get an address

kubectl get gateway grpc-gateway -w

You should see an ADDRESS appear after Envoy Gateway provisions the load balancer. On kind this will be a NodePort or you can use port-forward.

Port-forward for local testing

# Find the Envoy service name
kubectl get svc | grep envoy

# Port-forward (adjust service name as needed)
kubectl port-forward svc/envoy-envoy-gateway-grpc-gateway 10000:80 &

Test the proxy

grpcurl -plaintext localhost:10000 list

grpcurl -plaintext \
  -d '{"name": "Kubernetes"}' \
  localhost:10000 helloworld.Greeter/SayHello

Exercises

1. See EDS in action — scale and observe

# Scale primaries to 4 replicas
kubectl scale deployment grpc-primary --replicas=4

# Immediately send requests — new pods get traffic as they become ready
for i in $(seq 1 20); do
  grpcurl -plaintext -d '{"name": "test"}' \
    localhost:10000 helloworld.Greeter/SayHello 2>/dev/null \
  | grep message
done

Notice you start seeing new hostnames in responses as each pod passes its readiness probe. No Envoy restart or config reload happened — EDS updated Envoy's endpoint table automatically.

2. Simulate primary failure

kubectl scale deployment grpc-primary --replicas=0

Kubernetes immediately marks the pod endpoints as not ready, which EDS propagates to Envoy. Within seconds, requests fail (since we have weight 0 for backup in this config — see the note above about priority failover in Gateway API).

3. Inspect the generated xDS config

This is the most powerful debugging technique for Gateway API + Envoy:

# Find the Envoy pod managed by Envoy Gateway
ENVOY_POD=$(kubectl get pods \
  -l gateway.envoyproxy.io/owning-gateway-name=grpc-gateway \
  -o name | head -1)

echo "Envoy pod: $ENVOY_POD"

# Dump the full xDS config (Envoy's admin API, port 19000 in Envoy Gateway)
kubectl exec $ENVOY_POD -- \
  wget -qO- http://localhost:19000/config_dump \
  | python3 -m json.tool > /tmp/envoy-config.json

# Inspect clusters
python3 -c "
import json, sys
cfg = json.load(open('/tmp/envoy-config.json'))
for entry in cfg['configs']:
    if entry.get('@type','').endswith('ClustersConfigDump'):
        for c in entry.get('dynamic_active_clusters', []):
            print(json.dumps(c['cluster']['name']))
"

4. Compare the generated config to what you wrote manually

Look for your cluster in the config dump and compare its lb_policy, health_checks, and circuit_breakers to what you wrote in Steps 3 and 4. You should recognize all the fields.

python3 -c "
import json
cfg = json.load(open('/tmp/envoy-config.json'))
for entry in cfg['configs']:
    if entry.get('@type','').endswith('ClustersConfigDump'):
        for c in entry.get('dynamic_active_clusters', []):
            print(json.dumps(c['cluster'], indent=2))
" | less

5. Watch live endpoint updates with EDS

# Stream the EDS config
kubectl exec $ENVOY_POD -- \
  wget -qO- 'http://localhost:19000/config_dump?resource=type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment' \
  | python3 -m json.tool | grep -E "(address|port_value|health_status)"

Scale the deployment and run this command again — you will see the endpoint list update.

Production checklist

Before going to production, ensure:

  • TLS — add protocol: HTTPS to the Gateway listener and configure a certificate via SecretObjectReference
  • mTLS to backends — use BackendTLSPolicy to enforce TLS between Envoy and your gRPC services
  • Resource limits — set CPU/memory requests and limits on the Envoy Gateway deployment
  • Horizontal scaling — configure EnvoyProxy with replicas: 3 for the Envoy data plane
  • Rate limiting — add a BackendTrafficPolicy with rateLimit for per-client or global limits
  • Distributed tracing — configure the EnvoyProxy resource with an OpenTelemetry trace endpoint
  • Admin API security — Envoy Gateway binds the admin API to localhost only; verify this in your deployment
  • PodDisruptionBudget — ensure Envoy pods are not all drained simultaneously during node maintenance

What you learned

  • Why Kubernetes requires dynamic configuration (EDS) instead of static configs
  • The xDS API family and what each component controls
  • Envoy Gateway architecture: control plane translates Gateway API → xDS → Envoy
  • GatewayClass, Gateway, GRPCRoute, BackendTrafficPolicy resources
  • appProtocol: kubernetes.io/h2c as the k8s equivalent of http2_protocol_options
  • How to inspect generated xDS config for debugging
  • EDS: endpoints update automatically when pods scale, no reload needed

Summary: full config translation table

Step 3/4 Envoy YAML Step 5 Kubernetes resource
Listener Gateway
VirtualHost + Route GRPCRoute
cluster.lb_policy Envoy Gateway default (LEAST_REQUEST for gRPC)
http2_protocol_options appProtocol: kubernetes.io/h2c
health_checks.grpc_health_check Pod readinessProbe.grpc
outlier_detection BackendTrafficPolicy.healthCheck.passive
circuit_breakers BackendTrafficPolicy.circuitBreaker
retry_policy BackendTrafficPolicy.retry
priority: 0 / priority: 1 Weight-based split (full priority not yet in Gateway API)