Evaluate based on service count and team maturity. Service mesh overhead makes sense when you have 20+ services with complex cross-service communication that would otherwise require duplicating mTLS, retries, circuit breaking, and observability logic in each service. If you have 5 services, the overhead exceeds the benefit. If you have 50 services with a dedicated platform team, the mesh pays for itself by centralizing network policy. Start by measuring current operational burden: how many engineers own network logic, how many custom retry implementations exist, and what does your mTLS coverage look like. If the answers show duplication, evaluate Istio or Linkerd in a non-production cluster first.

First, verify the traffic flow: check that the DestinationRule and VirtualService are correctly configured for the target service. Verify mTLS is actually working by checking Envoy logs for connection failures (blocked by auth policy). Use istioctl authz check <pod> to see the effective authorization policy. Check for AuthorizationPolicy rules that might be blocking traffic — explicitly deny rules take precedence. Check service account labels are correct (authorization policies bind to service accounts). Use istioctl proxy-config cluster <pod> to see what clusters Envoy knows about. If everything looks correct, check for exhausted circuit breakers in the DestinationRule.

Istio uses per-pod Envoy sidecar proxies that intercept all inbound and outbound traffic. mTLS is enforced at the proxy layer — the application is unaware of mTLS. This means Istio can inspect, modify, and route traffic but adds a sidecar per pod. Linkerd uses a "micro-proxy" that is more lightweight than Envoy, also intercepting traffic at the proxy layer. Both provide automatic mTLS. Istio's advantage is flexibility and extensibility (more plugins, finer-grained traffic management). Linkerd's advantage is simplicity, lower resource overhead, and a more opinionated default configuration that works out of the box.

The sidecar proxy adds latency to every service call because it intercepts, inspects, and forwards traffic. With Linkerd, the overhead is typically 1-3% on p99 latency due to its lightweight Rust-based proxy. With Istio/Envoy, overhead can be 2-5ms on p99 depending on configuration and the number of applied policies. mTLS adds additional crypto overhead — plan for this in capacity planning. The key insight is that mesh latency is consistent and predictable, which makes it easier to account for than the unpredictable failures that occur without proper retries, circuit breakers, and timeouts.

A service mesh enforces this via AuthorizationPolicy in Istio or traffic policy in Linkerd. Create a default deny-all policy at the namespace level, then explicitly allow only the specific call from service A's service account to service B. This works at the network layer regardless of Kubernetes network policy — it operates below the application. The mesh also logs all rejected attempts, giving you audit trails for compliance. Without a service mesh, you would need Kubernetes NetworkPolicy plus application-level checks, which is harder to maintain consistently.

Kubernetes uses a mutating admission webhook to inject sidecars automatically. When a pod is created, the webhook intercepts the API request and modifies the pod spec to include the proxy container. The istio-proxy container has an init container that sets up iptables rules to redirect traffic before the main application starts. This order matters because the application needs to bind to its ports, but all traffic gets routed through the proxy. If the application starts first and binds ports before the proxy can redirect traffic, some requests might slip through unproxied. The readiness probe on the sidecar ensures it has fetched its configuration (via xDS) before the pod receives production traffic.

Standard TLS authenticates the server to the client — you know you are talking to the right server. mTLS adds mutual authentication: both sides prove their identity with certificates. In a service mesh, every workload has its own certificate issued by the mesh CA. When service A calls service B, each presents a certificate and verifies the other. This prevents unauthorized services from impersonating legitimate ones, even if they are on the same network. Without mTLS, a compromised service could spoof requests to other services. With mTLS, the mesh enforces identity at the network layer regardless of where the compromise occurs.

The control plane uses xDS APIs (Envoy's management protocol) to push configuration to proxies. Istiod serves as the xDS server; proxies connect via a persistent gRPC stream and receive updates whenever routing rules, policies, or certificates change. This is entirely out-of-band — the application makes normal network calls and the proxy intercepts and applies policies transparently. The key insight is that the application never references the proxy directly; it connects to localhost or a known service name, and the proxy handles the rest. No code changes, no redeployments when you update traffic policies.

503 errors in a mesh usually mean the proxy cannot reach the destination or the circuit breaker is open. Check the Envoy access logs on the proxy handling the failing requests — they will show whether the error is an upstream connection failure, timeout, or local rate limiting. Verify the DestinationRule's circuit breaker settings: consecutive gateway errors or ejection parameters might be prematurely removing healthy hosts. Check for resource exhaustion on the destination pods: OOM kills on sidecars cause exactly this pattern. Also verify the xDS sync status — if a proxy is using stale configuration, it may be routing to a destination that no longer exists or using an outdated subset definition.

xDS is Envoy's management protocol—a family of discovery APIs that the control plane uses to push configuration to proxies. The core APIs are LDS (Listener Discovery Service) for inbound traffic configuration, RDS (Route Discovery Service) for HTTP routing rules, CDS (Cluster Discovery Service) for upstream service clusters, and EDS (Endpoint Discovery Service) for load balancing endpoints. There's also ECDS for extension configurations and SDS for secrets. Proxies maintain a gRPC stream to the control plane and receive incremental updates whenever configuration changes. This means you can update a routing rule and have it propagated to all proxies within seconds—no pod restarts needed. Istiod serves as the xDS server for Istio; Linkerd uses a similar approach with its own control plane. The key advantage is that configuration changes are atomic and eventually consistent across the entire mesh.

Start by checking Envoy's resource usage: kubectl top pod -n istio-system to see which proxies are using the most memory. Envoy's memory scales with the number of routes, clusters, and endpoints it manages—larger configs mean more memory. Check the Envoy config dump: istioctl proxy-config all <pod> to see how many routes and clusters are configured. If a service has hundreds of upstream dependencies, Envoy holds config for all of them. Solutions include: tuning GC_MAX_OBJECTS_PER_GC in the proxy, limiting the number of referenced services in DestinationRules, using Namespace-scoped AuthorizationPolicies instead of mesh-wide policies, and configuring maxRequestsPerConnection to reduce keepalive overhead. Also verify the proxy's memory limits are set appropriately in the Pod spec—the default may be too low for heavily connected services.

Istio uses a built-in CA (Certificate Authority) that issues workload certificates via the SDS (Secret Discovery Service). Each pod has a secret volume mounted that contains a cert and private key—Istiod issues these and they're rotated automatically before expiration. The default workload certificate TTL is 24 hours; the proxy checks for rotation well before expiry. When a certificate rotates, Envoy picks up the new cert on the next SDS push—no connection interruption because the rotation happens out-of-band. The Istiod CA signs certs using a self-signed root CA that lives in the istio-system namespace. For production, you can plug in an external CA like Vault or AWS Private CA by configuring the Istio CA to use an intermediate cert. The key to zero-downtime rotation is that the old and new certs coexist during the rotation window—the proxy always has a valid cert to present.

Start by installing the service mesh in PERMISSIVE mTLS mode—this allows both mTLS and plain-text traffic so services can migrate incrementally without breaking existing communication. Enable sidecar injection namespace by namespace, beginning with the least critical services. Test thoroughly in each namespace before proceeding. Key pitfalls: (1) forgetting to switch from PERMISSIVE to STRICT once migration is complete—leaving a security gap, (2) services that hardcode IP addresses instead of using DNS—Envoy's traffic routing relies on service names, (3) applications that manage their own TLS—double encryption adds overhead without benefit, (4) missing readiness probes—new sidecars need warm-up time to fetch xDS config before receiving production traffic, (5) not accounting for mesh overhead in capacity planning. After all services are injected, audit all AuthorizationPolicies to ensure they reflect intended communication patterns, then lock down mTLS to STRICT mode.

Consistent hashing routes requests to upstream hosts based on a hash of a request attribute—such as the destination IP, a request header, or the source identity. The key property is that when an upstream host is added or removed, only a minimal number of requests get remapped to different hosts. This is critical for session affinity: if a user's requests must go to the same backend (e.g., for cached state), consistent hashing prevents session breaks when the upstream pool changes. Istio's Envoy supports consistent hashing via the ring_hash load balancer type. Use consistent hashing when you have stateful upstream services where client-to-backend affinity matters, or when you want to spread load evenly across heterogeneous backends (different capacities). Round robin is simpler and works well for stateless services where any backend can handle any request. Overusing consistent hashing can cause hot spots if a popular key maps to the same backend repeatedly.

VirtualService defines routing rules—how to route traffic to a destination. It controls what happens to traffic after it arrives at the proxy: which subset (version) to send it to, what percentage of traffic goes where, what headers to match, what timeouts and retries to apply. VirtualService is about directing traffic flow. DestinationRule defines the destination itself—the subsets (labeled groups of pods), load balancing policy, circuit breaker settings, and TLS configuration. It describes the actual endpoints and their characteristics. Think of VirtualService as the " routing layer" and DestinationRule as the "traffic policy layer." You need both: VirtualService says "route 10% of traffic to v2", and DestinationRule defines what "v2" means (which pods have the v2 label) and how to load balance across those pods. A common pattern is one VirtualService per service for routing, with one DestinationRule per service for policies—though you can have multiple VirtualServices for the same destination for different routing scenarios.

Locality-aware load balancing routes traffic to the nearest available upstream endpoint based on the proximity of the source and destination. Envoy tracks the locality (region, zone, sub-zone) of each upstream endpoint and prioritizes routing within the same locality. Only when endpoints in the local locality are exhausted does traffic spill over to other localities. This minimizes latency and cross-zone data transfer costs—critical when running multi-region or multi-AZ deployments. In Istio, you enable this by configuring localityLbSetting in the mesh config. Without it, Envoy distributes traffic statistically evenly across all endpoints regardless of geography, which can add unnecessary latency (e.g., a request from us-east-1 going to us-west-2 adds ~80ms). The trade-off is that you need enough capacity in each locality to handle its local load—if one AZ fails, locality-aware routing can overload the surviving AZ if you don't have proper failover configured.

A service mesh provides identity and encryption at the network layer, but it doesn't make your application immune to attacks. The mesh's threat model assumes the control plane is trusted—if an attacker compromises the control plane, they can reconfigure all proxies. Secure the control plane with strict RBAC, mutual TLS on the istiodgRPC API, and network policies preventing unauthorized access to istio-system pods. Mesh policies operate at L4/L7—they can't enforce application-level constraints like "user X can only access resource Y"—you still need application-level authorization. The mesh also doesn't protect against compromised application containers: if a workload is exploited, the attacker can use the mesh identity (the mounted certificate) to move laterally. Defense in depth requires combining mesh security with Kubernetes NetworkPolicy, runtime security (e.g., Falco), proper RBAC on Kubernetes API, and regular policy audits. Also ensure the mesh's telemetry doesn't inadvertently expose sensitive data in traces or logs—scrub headers like Authorization tokens.

Start by defining your subsets in a DestinationRule: label your v1 and v2 pods (e.g., version: v1 and version: v2), then create a DestinationRule that maps those labels to subsets. Next, create a VirtualService that initially routes 100% of traffic to v1. To begin the canary, update the VirtualService to route 5% to v2 and 95% to v1. Monitor your key metrics: error rate, p99 latency, and custom business metrics for both versions. If v2 looks healthy, incrementally shift traffic—10%, 25%, 50%, 100%. At each step, watch for error rate spikes or latency degradation. If anything looks wrong, roll back by reducing the v2 weight. Service mesh makes this trivial—no pod restarts, no deployment changes, just a VirtualService weight update. For more sophisticated canaries (header-based routing for internal testing, traffic mirroring for shadow testing), use the VirtualService's match conditions. Validate the rollout with Prometheus queries on the per-subset metrics and distributed traces in Jaeger or Zipkin to confirm both versions are receiving the expected traffic patterns.

Start with the observability stack: Prometheus metrics from Envoy give you request duration histograms at each hop. Check the p50, p95, and p99 latencies per service to identify which service shows degradation. Then use distributed tracing (Jaeger or Zipkin) to follow a request end-to-end and pinpoint where latency spikes occur. istioctl proxy-config stats <pod> shows Envoy metrics including upstream latency per cluster. For deep debugging, Envoy access logs contain per-request timing information (time to first byte, total duration). Check whether latency is at the proxy level (Envoy processing) or upstream (the service itself). If it is proxy-level, look for resource contention: CPU throttling, memory pressure, or connection pool exhaustion. If it is upstream, profile the service or check for database query regressions. istioctl authz check <pod> can reveal whether authorization policy evaluation is adding latency.

Service meshes primarily manage intra-cluster traffic, but they can handle egress to external services through egress gateways or direct proxy configuration. For controlled external traffic, Istio's EgressGateway routes outbound traffic through a dedicated proxy so policies and observability apply to external calls. You can configure ServiceEntries to register external services with the mesh, treating them like internal services for routing and policy purposes. This lets you apply mTLS to external calls using mesh-issued certificates, though the external service must support mTLS or you fall back to plain TLS. For services that do not support mTLS, you can use mesh-issued sidecar certificates wrapped in a protocol the external service understands. The trade-off is that the mesh cannot enforce authorization policies on the external service side—it can only control identity for the mesh-side caller.