Step 3: Envoy as a gRPC Proxy

Duration: ~30 minutes Goal: Run a real gRPC service, proxy it through Envoy, and understand what's different about gRPC.

---

What you will learn

• Why gRPC requires HTTP/2 on the upstream connection
• How to configure http2_protocol_options correctly
• The grpc_health_check protocol vs HTTP health checks
• The grpc_stats filter for gRPC-specific metrics
• How to use grpcurl to test gRPC through Envoy

---

Concepts

gRPC is HTTP/2

gRPC uses HTTP/2 as its transport. HTTP/2 differs from HTTP/1.1 in ways that matter for Envoy:

Feature	HTTP/1.1	HTTP/2
Multiplexing	One request per connection	Many requests per connection
Streaming	Not supported	Bidirectional streaming
Headers	Text, per-request	HPACK-compressed, binary
Connections	Many short-lived	Few long-lived

The critical Envoy config requirement: You must tell Envoy to speak HTTP/2 to your gRPC backends. Without this, Envoy uses HTTP/1.1 and gRPC breaks entirely. The setting is typed_extension_protocol_options with http2_protocol_options: {} on the cluster.

Request-level load balancing

This is the key advantage of using Envoy for gRPC over a plain TCP load balancer (like kube-proxy with ClusterIP):

• TCP load balancer: picks a backend when the connection opens, then all requests on that connection go to the same backend. With HTTP/2 multiplexing, a single long-lived connection can carry thousands of requests — all going to one backend.
• Envoy (HTTP/2-aware): load-balances at the request level. Each gRPC call can go to a different backend, even over the same connection from the client.

The codec_type field

codec_type on the HTTP Connection Manager controls what protocol Envoy accepts from downstream clients (the side connecting to Envoy):

Value	Behavior
AUTO	Envoy negotiates: HTTP/2 if the client requests it (via ALPN or h2c upgrade), otherwise HTTP/1.1
HTTP1	Only accept HTTP/1.1. Refuse HTTP/2 connections.
HTTP2	Only accept HTTP/2. Refuse HTTP/1.1 connections.

AUTO is the right choice when you want to accept both grpcurl (which uses h2c) and HTTP/1.1 clients on the same port. Note that codec_type only affects the downstream (client → Envoy) side; the upstream (Envoy → backend) protocol is configured separately on the cluster via typed_extension_protocol_options.

Upstream protocol: typed_extension_protocol_options

This is the most critical gRPC config setting. It tells Envoy which protocol to use when connecting to backends:

typed_extension_protocol_options:
  envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
    "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
    explicit_http_config:
      http2_protocol_options: {}  # use HTTP/2 to all backends in this cluster

Why explicit_http_config? The HttpProtocolOptions message has three modes for selecting the upstream protocol: - explicit_http_config — you choose: http_protocol_options (HTTP/1.1), http2_protocol_options (HTTP/2), or http3_protocol_options (HTTP/3) - use_downstream_protocol_config — mirror whatever protocol the client used - auto_config — HTTP/2 if available, fallback to HTTP/1.1

For gRPC backends you must use explicit_http_config with http2_protocol_options: {}. The empty {} uses defaults (no special settings needed for most gRPC use cases).

gRPC health checking protocol

gRPC defines a standard health checking protocol: a service called grpc.health.v1.Health with a Check method. Envoy supports this natively with grpc_health_check: {}. Your backend just needs to implement the protocol (the google.golang.org/grpc/health package does this for Go).

Using grpc_health_check instead of http_health_check means: - Health probes speak the same protocol as real traffic (HTTP/2) - The backend controls its own health status per service name - It works through TLS correctly

---

Setup

mkdir -p envoy-tutorial/step3 && cd envoy-tutorial/step3

Backend: server.go

A minimal gRPC server implementing the standard helloworld.Greeter service and the health protocol:

package main

import (
    "context"
    "log"
    "net"
    "os"

    "google.golang.org/grpc"
    "google.golang.org/grpc/health"
    "google.golang.org/grpc/health/grpc_health_v1"
    "google.golang.org/grpc/reflection"

    pb "google.golang.org/grpc/examples/helloworld/helloworld"
)

type server struct{ pb.UnimplementedGreeterServer }

func (s *server) SayHello(_ context.Context, req *pb.HelloRequest) (*pb.HelloReply, error) {
    hostname, _ := os.Hostname()
    return &pb.HelloReply{
        Message: "Hello " + req.Name + " from " + hostname,
    }, nil
}

func main() {
    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }

    s := grpc.NewServer()
    pb.RegisterGreeterServer(s, &server{})

    // Register health service — required for grpc_health_check
    healthSrv := health.NewServer()
    grpc_health_v1.RegisterHealthServer(s, healthSrv)
    healthSrv.SetServingStatus("", grpc_health_v1.HealthCheckResponse_SERVING)

    // Register reflection — required for grpcurl to list services
    reflection.Register(s)

    log.Printf("listening on :50051 (hostname=%s)", func() string {
        h, _ := os.Hostname()
        return h
    }())
    s.Serve(lis)
}

go.mod

module grpc-server

go 1.22

require google.golang.org/grpc v1.64.0

Run go mod tidy inside the container (handled by the Dockerfile below).

Dockerfile

FROM golang:1.22-alpine AS build
WORKDIR /app
COPY go.mod server.go ./
RUN go mod download && go build -o server .

FROM alpine:latest
COPY --from=build /app/server /server
ENTRYPOINT ["/server"]

docker-compose.yaml

services:
  backend1:
    build: .

  backend2:
    build: .

  envoy:
    image: envoyproxy/envoy:v1.31-latest
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml
    ports:
      - "10000:10000"
      - "9901:9901"
    command: envoy -c /etc/envoy/envoy.yaml
    depends_on:
      - backend1
      - backend2

Envoy config: envoy.yaml

static_resources:
  listeners:
    - name: grpc_listener
      address:
        socket_address: { address: 0.0.0.0, port_value: 10000 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: grpc_proxy
                codec_type: AUTO  # (1)
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                route_config:
                  name: grpc_route
                  virtual_hosts:
                    - name: grpc_services
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                            grpc: {}  # (2)
                          route:
                            cluster: grpc_cluster
                            timeout: 0s  # (3)
                http_filters:
                  - name: envoy.filters.http.grpc_stats  # (4)
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig
                      emit_filter_state: true
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: grpc_cluster
      connect_timeout: 5s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      # --- CRITICAL: tell Envoy to use HTTP/2 to backends ---
      typed_extension_protocol_options:  # (5)
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: {}
      load_assignment:
        cluster_name: grpc_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address: { address: backend1, port_value: 50051 }
              - endpoint:
                  address:
                    socket_address: { address: backend2, port_value: 50051 }
      health_checks:
        - timeout: 2s
          interval: 5s
          unhealthy_threshold: 2
          healthy_threshold: 1
          grpc_health_check: {}  # (6)

admin:
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

1. codec_type: AUTO lets Envoy accept both HTTP/1.1 and HTTP/2 from clients. Clients connecting with grpcurl or any gRPC library will negotiate HTTP/2 via h2c (cleartext). This setting only affects the downstream (client → Envoy) side.
2. grpc: {} is a route matcher that only matches gRPC requests (those with content-type: application/grpc). Non-gRPC requests to the same listener won't match this route — they would fall through to the next route rule or return 404 if no other rule exists.
3. timeout: 0s disables the per-request timeout. The default timeout is 15 seconds, which would abort long-running or streaming RPCs mid-call. Set to 0s when Envoy should not impose a deadline. Use per_try_timeout on a retry policy if you want a per-attempt limit without a global deadline.
4. grpc_stats filter collects per-method request/response message counts. It must be placed before the router filter because the router filter is terminal — once it forwards the request, no earlier filter runs again.
5. typed_extension_protocol_options is the v3 API way to configure upstream protocol. The key must be the full extension name envoy.extensions.upstreams.http.v3.HttpProtocolOptions — this is how Envoy looks up the extension. The older http2_protocol_options field directly on the cluster is deprecated and should not be used.
6. grpc_health_check: {} sends a grpc.health.v1.Health/Check RPC to probe the backend. This is the correct way to health-check gRPC services. The empty {} checks the overall server health (service name ""). To check a specific service, use grpc_health_check: { service_name: "helloworld.Greeter" }.

---

Run it

docker compose up --build

---

Exercises

1. List gRPC services through Envoy

grpcurl -plaintext localhost:10000 list

You should see:

grpc.health.v1.Health
helloworld.Greeter

This works because the backend registers gRPC reflection, which Envoy proxies transparently.

2. Call the gRPC service

grpcurl -plaintext \
  -d '{"name": "World"}' \
  localhost:10000 helloworld.Greeter/SayHello

Expected output:

{
  "message": "Hello World from <container-id>"
}

3. See request-level load balancing

for i in $(seq 1 8); do
  grpcurl -plaintext -d '{"name": "test"}' \
    localhost:10000 helloworld.Greeter/SayHello 2>/dev/null \
  | grep message
done

You should see the container IDs (hostnames) alternating between the two backends — even though each grpcurl invocation may reuse an existing HTTP/2 connection.

4. Check gRPC-specific stats

The grpc_stats filter exposes metrics per method:

curl -s http://localhost:9901/stats | grep grpc

Look for: - grpc.helloworld.Greeter.SayHello.request_message_count - grpc.helloworld.Greeter.SayHello.response_message_count - grpc.helloworld.Greeter.SayHello.success

5. Break it: remove http2_protocol_options

Comment out or delete the typed_extension_protocol_options block from the cluster and restart Envoy:

docker compose restart envoy

Now try the gRPC call:

grpcurl -plaintext -d '{"name": "World"}' \
  localhost:10000 helloworld.Greeter/SayHello

You will get an error like Failed to dial target host: ... or a malformed response. Envoy is now speaking HTTP/1.1 to your gRPC backend, which speaks only HTTP/2. This is the most common gRPC + Envoy misconfiguration.

Restore the config and restart Envoy to fix it.

6. Inspect the access log

Each gRPC call produces an access log line from Envoy. Look at the format — you'll see the gRPC method path (e.g. /helloworld.Greeter/SayHello) as the request path, and the HTTP status code (200 for success, even if the gRPC status is an error).

HTTP 200 ≠ gRPC success

gRPC encodes its status in a trailer header called grpc-status, not in the HTTP status code. grpc-status: 0 means OK. A failed gRPC call may have HTTP status 200 but grpc-status: 14 (UNAVAILABLE). The grpc_stats filter correctly tracks gRPC-level success/failure.

---

What you learned

• http2_protocol_options: {} is mandatory for gRPC upstream clusters
• grpc: {} route matcher to target only gRPC traffic
• grpc_health_check for proper backend health probing
• grpc_stats for gRPC-specific metrics
• timeout: 0s to avoid proxy-level timeouts on streaming RPCs
• Why HTTP 200 ≠ gRPC success

Next step

In Step 4 you will add primary/backup failover, outlier detection, circuit breaking, and retry policies.