Response Flags in Access Logs

UH (NoHealthyUpstream) indicates that there are no healthy service instances at the backend.

Typical Symptom

All service instances at the backend are unavailable. For example, the number of service instances is set to 0.

Example Log

Client log

Suggestion

Check the pod configuration of the target service and ensure that all service instances are running properly.

DC (DownstreamConnectionTermination) indicates that the downstream connection is terminated.

If the client disconnects from the server before receiving a response due to various reasons, DC will be recorded in the access logs. In most cases, the size of the response body is 0.

Typical Symptom

When accessing the target service, the client disconnects from the server before receiving a response or receives no response from the server within the timeout set on the client.

Example Log

Client log

Suggestion

Generally, no operation is required. In most cases, time-consuming responses often result in client disconnections. You can optimize the target service to solve such an issue.

UF (UpstreamConnectionFailure) indicates an upstream connection error.

Typical Symptom

The service port of the target service is unreachable. For example, if the client accesses the target service over an invalid port, 503UF will be recorded in the outbound logs of the client agent.

If the pod port of the target service is unreachable, 503UF will be recorded in the inbound log of the server.

Example Logs

Outbound log generated for an invalid service port

Inbound log generated for an invalid pod port

Suggestion

Specify a valid service port, so that the client accesses the target service over this port.

URX (UpstreamRetryLimitExceeded) indicates that the requests are rejected because the HTTP retry limit or the maximum number of TCP connection attempts is reached.

Typical Symptom

If the number of failures to invoke the server reaches the retry limit configured for the target service in the service mesh, or the failure persists after two retries (default value in Istio), URX will be recorded in the outbound logs on the client.

Example Log

Outbound log on the client

Suggestion

Improve the service success rate, so that services can be properly accessed after limited retries.

UPE (UpstreamProtocolError) indicates an upstream protocol error.

Typical Symptom

If an application protocol defined for each service port in the service mesh is inconsistent with the actual protocol, the client will receive protocol error code 502 during service access. 502 UPE will be recorded in the inbound logs on the server.

Example Logs

Outbound log on the client

Inbound log on the server

Suggestion

Define a correct application protocol for each service port. Ensure that the protocol is read from the port name of the service in Istio, or use the AppProtocol field to specify a correct application protocol for each service port.

DPE (DownstreamProtocolError) indicates a downstream protocol error.

Typical Symptom

The downstream client uses an incorrect protocol to access the target service.

Example Logs

Outbound log on the client

Inbound log on the server

Suggestion

Ensure that the client uses a correct protocol to access the target service.

NC (NoClusterFound) indicates that the cluster where the service instance in the service mesh resides was not found.

Typical Symptom

The backend defined in the traffic distribution policy is not defined in the service mesh. For example, as defined in the traffic distribution policy, traffic is routed to the service subset identified by V2, but this service subset is not defined in DestinationRule.

Example Log

Client log

Suggestion

Ensure that the service subset referenced in VirtualService is correctly defined in DestinationRule.

NR (NoRouteFound) indicates that there is no matched route for the requests. Generally, status code 404 is returned.

Typical Symptom

The requests do not meet the route matching conditions defined in VirtualService. As a result, no route is matched for the requests.

Example Log

Outbound log on the client

Suggestion

Ensure that client requests meet the traffic characteristics defined in the route, so that the route defined for the server is matched for the requests.

DI (DelayInjected) indicates that a delay fault is injected into the request.

Typical Symptom

If delay fault injection is configured in VirtualService, the request processing will be delayed for a period specified via fault injection, and DI will be recorded in the access logs.

Example Logs

Outbound log on the client

Inbound log on the server

Suggestion

No operation is required. This is a manual operation recorded in the access log.

FI (FaultInjected) indicates the request was aborted with a response code specified via fault injection.

Typical Symptom

A fault with a specific status code is injected to the target service through VirtualService.

Example Log

Outbound log on the client

Suggestion

No operation is required. This is a manual operation recorded in the access log.

UT (UpstreamRequestTimeout) indicates that an upstream request times out. Generally, HTTP status code 504 is returned.

Typical Symptom

A timeout is configured for the target service in VirtualService. When the specified time period expires, the client agent automatically times out and cancels the request.

Example Log

Outbound log on the client

Suggestion

No operation is required. The timeout is configured for quick access failure of the target service. If the service access times out, the response flag will be returned.

RL (RateLimited) indicates that the traffic limit is triggered. Generally, HTTP status code 429 is returned.

Typical Symptom

A local or global traffic limit policy is configured in the service mesh. If the number of requests in a unit time reaches the specific threshold, the traffic limit is triggered.

Example Logs

Traffic limit policies apply to the scenario where the server receives the traffic

Outbound log on the client

Inbound log on the server

Traffic limit policies apply to the scenario where the client sends the traffic

Suggestion

No operation is required. Traffic limit policies are configured to protect the target service.