Skip to content

Wait for demand after the server closes - #2394

Open
mkurz wants to merge 1 commit into
AsyncHttpClient:mainfrom
mkurz:fix/decompression-half-closure
Open

mkurz wants to merge 1 commit into
AsyncHttpClient:mainfrom
mkurz:fix/decompression-half-closure

Conversation

@mkurz

@mkurz mkurz commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Problem

Since #2386, when the connection closes while input is held back, a body that the close cut off is dropped, but the rest of a body that has ended is still decompressed and handed on at once, bounded only by maxDecompressedResponseSize. Netty tears the pipeline down with the close, so nothing can wait for demand. On native transports this happens whenever the server closes the connection right after the body, for example with Connection: close or a body delimited by the connection, because they read the close even while reads are suspended.

A handler that suspends after every part and does not resume, 16 MiB of zeros as gzip, and a server that closes the connection after sending it:

Transport Body cut in half Complete body, then close Body delimited by the close
NIO, main and this PR 1 part; the close is not read 1 part 1 part
epoll, io_uring on main 1 part, then RemotelyClosedException 257 parts (16 MiB) at once, completed 257 parts (16 MiB) at once, completed
epoll, io_uring with this PR 1 part, then RemotelyClosedException 1 part; then one part per resume, completed, connection closed 1 part; then one part per resume, completed, connection closed

The same over TLS (258 parts).

Change

  • While input is held back, Http1ContentDecompressor turns ALLOW_HALF_CLOSURE on for the channel, if it was off. The server's FIN then reaches the pipeline as a ChannelInputShutdownEvent instead of a close. ByteToMessageDecoder, and so HttpObjectDecoder, already handles that event like a close and hands on the end of a body delimited by the connection.
  • On that event, while the response is paused, the end of the body decides, as it does on a close since Drop held input of a body cut off by the close #2386:
    • a body that has ended successfully, including one delimited by the connection, still waits for demand. The response no longer keeps the connection alive (NettyResponseFuture.setKeepAlive(false)), so the connection is closed once the response is done instead of being offered to the pool.
    • a body that the close cut off, or whose last chunk could not be parsed, does not wait: the channel is closed at once, so channelInactive drops the input held back as before. A failed last chunk still fails the request with the parser's error; a cut-off body is retried or fails with RemotelyClosedException.
  • Half-closure is turned off again once nothing is held back, at the latest before the end of the body is handed on, and when the handler is removed. So a connection is not offered for reuse with it on.
  • If the application turned half-closure on itself, it is left on, and a close by the server while input is held back is handled the same way. On main, such a connection was offered to the pool after the server had shut its input down.
  • The Javadoc of ResponseBodyControl.suspend() and of the class say so.
  • Revapi: overriding userEventTriggered drops Netty's internal @ChannelHandlerMask.Skip marker. The same scoped java.annotation.removed ignore as for Http2Handler::userEventTriggered.

NIO reads only when asked to, so in the tested cases it saw the close only once the input held back had been used up and half-closure was off again. TLS can still ask for a read while reads are suspended, for example to complete a record; the tests have not seen that read the close. Half-closure is turned on there too while input is held back, and the NIO tests check that nothing else changes.

Not covered:

  • A close on the client's side (for example AsyncHttpClient.close()), a reset or any other close still reaches channelInactive, where the rest of a body that has ended can still be decompressed and handed on at once, as before.
  • kqueue: it reads the close like epoll and supports half-closure, but I could not run it.
  • A body without a content coding is unchanged: native transports deliver everything that arrived before the close.
  • HTTP/2 is separate.

AI disclosure

Claude Code on behalf of Matthias Kurz. The commit includes Co-Authored-By per AGENTS.md.

Test plan

  • Fail on main, pass with this PR:
    • Http1ContentDecompressorSuspensionTest (EmbeddedChannel with a NioSocketChannel's config, the event fired by hand):

      • aCloseByTheServerOnlyShutsTheInputDownWhileInputIsHeldBack (gzip, deflate, br, zstd, snappy): half-closure on while input is held back; after the event nothing more is handed on, the channel stays open, keep-alive is off; the rest arrives one part per resume, and half-closure is off at the end.
      • aCloseByTheServerFailsABodyThatItCutsOffAtOnce and aCloseByTheServerFailsABodyWhoseLastChunkFailedAtOnce (five codings each): the channel is closed at once, nothing held back is handed on, a failed end is handed on with the parser's error, every input buffer is released.
      • halfClosureIsOffWhenTheEndOfTheBodyIsHandedOn (five codings), removalTurnsHalfClosureOff, discardingABodyHeldBackTurnsHalfClosureOff, halfClosureThatWasOnAlreadyIsLeftOn (ended and cut off).
    • Http1DecompressionSuspensionTest (Linux):

      • aSuspendedBodyThatHasEndedWaitsForDemandAfterTheServerCloses (epoll, io_uring; http, https; a Content-Length body then a close, and a body delimited by the close): the input is shut down, 1 part, the response is not done; then at most one part per resume, the whole body, no offer to the pool, the connection closed.
      • aSuspendedBodyThatTheServerCutsOffOrBreaksFailsAtOnce (epoll, io_uring; http, https; a Content-Length and a chunked body cut in half, and a chunked body whose next chunk size line is zz; 1 MiB limit, no retries): the input is shut down, the request fails with RemotelyClosedException or the parser's NumberFormatException rather than the size limit, 1 part, the connection closed.
      • theRequestTimeoutAfterTheServerClosesFailsTheResponse (epoll, io_uring; http, https): a 2 s request timeout after the input was shut down fails with TimeoutException, 1 part, the connection closed.
      • cancellationAfterTheServerClosesClosesTheConnection (epoll, io_uring; http, https).
      • aConnectionWithHalfClosureOnAlreadyIsNotReusedAfterTheServerCloses (epoll, io_uring): on main the connection is offered to the pool.
      • aConnectionIsReusedWithHalfClosureOffAfterASuspendedBody (NIO, epoll, io_uring; keep-alive, no close): half-closure on while input is held back, off when the connection is offered to the pool, and the next request reuses the connection.
      • onNioASuspendedBodyThatHasEndedWaitsForDemandAfterTheServerCloses (http, https; both closing bodies): after the server has closed the connection, 1 part, the response is not done, half-closure on; then the whole body on demand.

      The last two fail on main only on the check that half-closure is on; their other checks pass there too.

  • Pass on main and with this PR: Drop held input of a body cut off by the close #2386's aSuspendedBodyThatTheConnectionCutsOffIsRetried and aSuspendedBodyWhoseLastChunkCannotBeParsedFailsWithTheParsersError (epoll), which now go through the input shutdown and the close it triggers. /zeros-bad-chunk now writes its raw chunk size line through HttpServerCodec's context, so that it also works over TLS.
  • Each change breaks the tests it should:
    • waiting for demand after any end, failed ones included: 10 tests fail, among them Drop held input of a body cut off by the close #2386's parser-error test, which then hangs until the request timeout;
    • not turning half-closure off before the end is handed on: the five-coding unit test and the pool test fail;
    • not turning it off on removal: the removal test fails;
    • main's decompressor: 61 tests fail.
  • ./mvnw -B -ntp -Dgpg.skip=true clean verify on JDK 11: 2,159 tests, 0 failures, 0 errors, 11 skipped; Revapi passed. No test-skipping flags were used.
  • The Http1ContentDecompressor*Test, Http2ContentDecompressorTest, Http1Decompression*Test and ResponseBodyControlTest classes compiled on JDK 11 and run on JDK 17, 21 and 25 (Maven Surefire's -Djvm): 203 tests each, 0 failures.
  • Http1DecompressionSuspensionTest five more times in a row: 51 tests each, 0 failures.

A suspended response that held compressed input back still had all of
a body that had ended decompressed and handed on at once when the
server closed the connection, on native transports such as epoll,
which read the close even while reads are suspended. Only the
channel's pipeline held that input, and it is torn down with the
close.

While input is held back, Http1ContentDecompressor now turns on
half-closure for the channel, so that an orderly close by the server
only shuts the input down. The end of the body then decides, as it
does on a close:

- the rest of a body that has ended, including one delimited by the
  connection, still waits for demand, and the response no longer
  keeps the connection alive, so it is closed once the response is
  done;
- a body that the close cut off, or whose last chunk could not be
  parsed, does not wait: the channel is closed at once, which drops
  what is held back of it, and the request is retried or fails as
  before.

Half-closure is turned off again once nothing is held back, at the
latest before the end of the body is handed on, so a connection is
not offered for reuse with it on. If it was on already, it is left
on, and a close by the server while input is held back is handled the
same way.

NIO reads only when asked to, so in the tested cases it saw the close
only once the input held back had been used up and half-closure was
off again. TLS can still ask for a read while reads are suspended, for
example to complete a record.

Claude Code on behalf of Matthias Kurz

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant