Proposal: tell servers why a client cancelled a call (negotiated RST_STREAM reason codes) #13176

Summary

When a client cancels an RPC, the server only learns that the call was cancelled, never why. Today every implementation I checked sends RST_STREAM(CANCEL) regardless of the cause, and the server surfaces CANCELLED.

I'd like to propose a small, opt-in HTTP/2 extension:

1. A new HTTP/2 setting that a server advertises to say "I understand gRPC cancellation reason codes".
2. A small block of new RST_STREAM error codes, one per cancellation reason (deadline expired, hedge/retry attempt superseded, application cancel, ...).
3. Clients send a reason code only to peers that advertised the setting, and CANCEL to everyone else.

Old peers and proxies never see a new code, so this is backwards compatible by construction. Proxies can adopt it independently, one connection at a time.

Is your feature request related to a problem?

Server-side signals increasingly drive load balancing: ORCA load reports, server-side error rates, and SLO metrics. For these, the reason for a cancellation matters:

Because these are indistinguishable, a server (or a load balancer consuming its reports) has to either:

• count CANCELLED as an error, which penalizes healthy hosts for every lost hedge race, or
• ignore CANCELLED, which hides the most important slow-host signal: calls the client gave up on.

The same ambiguity exists on the client side for outlier detection (#12834). There, the client at least knows the reason locally. The server has no way to find out.

Why the server can't infer it today

• The server's own deadline fires late. grpc-timeout is relative, so the server's deadline starts when it receives the headers, which is after the client's deadline started. The client's RST_STREAM usually arrives before the server's own timer fires, so the server records CANCELLED, not DEADLINE_EXCEEDED.
• grpc-previous-rpc-attempts (gRFC A6) isn't enough. It tells the server up front that an attempt is a retry or hedge, but the original attempt (which carries no header) can also lose a hedge race. And the reason for a cancellation is only known at cancel time.
• Nothing else can carry the reason. After RST_STREAM, nothing more can be sent on the stream, and gRPC clients don't send trailers.

Current behaviour (grpc-java)

• The client always resets with CANCEL: NettyClientHandler L851, L961.
• The server reports any received RST_STREAM as CANCELLED with the description "RST_STREAM received for code N": NettyServerHandler L606-L620.

Describe the solution you'd like

1. Setting: SETTINGS_GRPC_CANCEL_REASONS (identifier TBD)

• Values: 0 = disabled (initial value, as RFC 9113 §5.5 requires for negotiation settings); 1 = "I accept gRPC cancellation reason codes".
• Who sends it: a server sets it to 1 on connections where it can interpret the codes. Clients don't need to advertise anything in the client-to-server direction.

2. RST_STREAM error codes (values TBD)

The set should stay small and extensible. A receiver that advertised the setting must treat any code in the block that it doesn't recognize exactly like CANCEL.

3. Client behaviour

• If the peer's most recent SETTINGS has the setting set to 1, reset with the reason code.
• Otherwise (including before the peer's first SETTINGS is processed), reset with CANCEL, exactly as today.

4. Server behaviour

• Cancellation semantics are unchanged: the call is cancelled exactly as for CANCEL (context cancelled, listener onCancel, etc.).
• Only the reported reason changes. Suggested mapping, with the API exposure left to each implementation:
  • deadline: DEADLINE_EXCEEDED, matching what the server's own deadline timer reports today;
  • all others: CANCELLED, with a machine-readable reason visible to stream tracers / interceptors.
• gRPC-based proxies: the server should set the reason as the cancellation cause of the call's context. A gRPC application that proxies calls (cancels its outgoing call when the incoming one is cancelled) can then pass the reason on through its own client. grpc-java already turns an expired context deadline into DEADLINE_EXCEEDED on outgoing calls.

5. Registration

• RFC 9113 §5.5 allows extensions to define new settings and error codes.
• The IANA "HTTP/2 Settings" and "HTTP/2 Error Code" registries use Expert Review.
• RFC 9113 §11 folded the former experimental setting range into the normal registry policy, and the error-code registry never had a private range.
• So gRPC would register one setting identifier and a small block of error codes.

Backwards compatibility

• Old servers never advertise the setting, so they never receive a new code.
• Old clients ignore the unknown setting (RFC 9113 §6.5.2: "MUST ignore") and keep sending CANCEL.
• Why not send the codes unconditionally? RFC 9113 §7 lets a receiver treat unknown codes as INTERNAL_ERROR, and the gRPC HTTP/2 mapping turns that into INTERNAL. grpc-java servers happen to report any code as CANCELLED, but other implementations and proxies shouldn't have to be relied on. Negotiating removes the risk.
• Proxies: HTTP/2 settings apply to a single connection (RFC 9113 §6.5: "SETTINGS frames always apply to a connection, never a single stream"; §5.5: extensions "are effective only within the scope of a single HTTP/2 connection"). A proxy that doesn't know the setting ignores it and doesn't advertise it, so clients behind it keep sending CANCEL. There's no regression and no flag day.
• Direct connections benefit immediately (proxyless/xDS gRPC, client-side load balancing): this is where most service-to-service traffic flows in our deployment.

How proxies can support this over time

Proxies terminate HTTP/2 separately on each side, so "support" means doing the same negotiation on both sides:

1. Client side: advertise the setting to clients when the proxy can relay reasons.
2. Server side: remember, per upstream connection, whether that server advertised the setting.
3. On a client reset: carry the reason code across the proxy. Send it to the server if that connection advertised the setting, otherwise downgrade to CANCEL.

The proxy needs no per-client state. The decision is made per upstream connection at reset time, and pooled upstream connections shared across clients work fine.

The hard part is step 3: carrying a reset reason from one side of the proxy to the other. HAProxy shows this is achievable, and that it's being built for exactly this gRPC use case.

Precedent: HAProxy forwarding gRPC cancellations (2019 → 2024)

• 2019-07 (HAProxy 1.9): haproxy/haproxy#172 reported that gRPC client deadlines and cancellations didn't reach backends, because HAProxy didn't forward RST_STREAM. The maintainer traced it to the design issue in haproxy/haproxy#142 ("shutdown vs abort"): the stream layer could express a clean shutdown but had no way to carry an abort and its reason across sides.
• 2019–2023: incremental rework. The 2.1 end-of-input flag; a partial improvement backported to 2.2; removal of the stream_interface layer so a mux can store a close status shared with the stream; better error propagation in 2.9. In 2022 the maintainer said the plan requires encoding "the reason for the RST" in the internal layer, so it can be mapped between protocols.
• 2024-04 (HAProxy 3.0):
  • 96f8b7ad08 added a generic abort reason (32-bit code + source) passed between the two sides. The commit message says forwarding RST_STREAM codes "is mandatory for gRPC applications". It also says "we can imagine to alter this reason at the applicative level to enrich it".
  • 20b156ee15 forwards an H2 client's CANCEL to H2 servers, fixing Push Java libraries to Maven central #172. The QUIC mux records STOP_SENDING codes the same way.
• Today: the code is carried end to end, then filtered by an allow-list: h2s_is_forwardable_abort ("For now, only CANCEL from the client is forwardable to the server"). It's used when the server-side stream is shut down (L5977, L6066).

Supporting reason codes in HAProxy would now be small: advertise the setting on client connections, record it on server connections (the H2 mux already parses each peer's SETTINGS), and extend the allow-list to forward the new codes when the server advertised them.

Other proxies today

Each proxy can adopt this on its own schedule, and every hop that hasn't yet falls back to CANCEL. Deployments that go through proxies gain the feature as their proxies upgrade, and nothing breaks before then.

Alternatives considered

• Request header marking hedged attempts: works through proxies today, but only covers hedging. It's set before the outcome is known, and attempt 0 can't be tagged. It's still a useful complement.
• Server-side heuristic (treat CANCELLED near the server's deadline as a deadline miss): depends on latency and the extra time from the relative timeout. It can't tell a late hedge cancellation from a deadline.
• Custom extension frame carrying a reason before RST_STREAM: allows a richer payload, but proxies drop unknown frame types, and it adds ordering questions. Error codes reuse the frame every proxy already handles.
• Sending new codes without negotiation: allowed by RFC 9113 §5.5, but risks INTERNAL on old peers (see Backwards compatibility).

Open questions

• Is the gRPC team open to registering a setting and error codes with IANA, or would you prefer another mechanism?
• Which reasons should be in the initial set? Should the server→client direction (e.g. "server overloaded") be in scope later?
• How should each language expose the reason to stream tracers and interceptors?