Release 6.5.9 - #3719

Responses with neither Content-Length nor Transfer-Encoding (as sent by
HTTP/1.0 servers) are read until the connection closes. This path used
IOStream.read_until_close() and never consulted max_body_size, so an
over-large body was accepted instead of being reported as an error.

Worse, read_until_close() accumulates the entire body in the stream's
read buffer. On overflow IOStream._read_to_buffer() closes the stream
before raising StreamBufferFullError, and close() resolves the pending
read-until-close future *successfully* with whatever had been buffered.
When the overflow happened on the event-driven read path the exception
was swallowed by _handle_read(), so the caller of AsyncHTTPClient.fetch
received a silently truncated body with no indication of an error.

Read the body incrementally in chunk_size pieces instead, raising
HTTPInputError("Body too long") as soon as max_body_size is exceeded.
This surfaces the condition to fetch() as an error, stops reading
before an over-large body has been buffered, and makes the body size
independent of the stream's read buffer size (matching the behavior of
the Content-Length and chunked paths).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

BaseIOStream.close() resolved a pending read_until_close future with
whatever was in the read buffer, even when the stream was being closed
because of an error. The caller could not distinguish a complete result
from one truncated by a connection error: ECONNRESET partway through a
read_until_close looked exactly like a clean EOF.

Only resolve the future successfully when self.error is unset. When an
error is present, leave the future pending so that _signal_closed()
fails it with StreamClosedError(real_error=self.error), matching how
ordinary pending reads are already handled.

_read_to_buffer() also closed the stream without recording the error
when the read buffer overflowed, so self.error was left unset and the
condition was lost entirely on the event-driven read path (_handle_read
swallows the exception, and its own close(exc_info=e) is a no-op once
the stream is already closed). Pass the StreamBufferFullError to close()
so it reaches the pending read.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

Informational (1xx) responses are handled by calling _read_message
recursively to read the next message on the connection. Nothing bounded
the number of these, so a server that sent a long stream of "100
Continue" responses drove the client into unbounded recursion: at a few
thousand responses the client raised RecursionError, which reached the
caller of AsyncHTTPClient.fetch as a generic stream-closed error after
logging a stack trace.

Thread a counter through the recursive calls and reject the message with
HTTPInputError once _MAX_1XX_RESPONSES (10) have been seen. There is no
legitimate use for more than a handful of informational responses before
the real one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

The 1xx branch of _read_message recursed to read the real response but
did not return afterward, so the outer frame fell through and ran the
rest of the function a second time. With a single "100 Continue" the
delegate was finished twice and a second body was read; with three, the
delegate was finished four times and _read_body ran four times.

A 1xx response has no Content-Length and is not chunked, so each of
those extra _read_body calls landed in _read_body_until_close. That was
survivable only when the server closed the connection promptly. Against
a server that keeps the connection open, the outer frame is left
blocked on a read that will never be satisfied and read_response never
completes.

Return the recursive call's result directly. The recursive call owns the
delegate from that point, so need_delegate_close is cleared first to
keep the finally block from calling on_connection_close() on a delegate
that has already been finished.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

parse_body_arguments limits the number of fields in a urlencoded body
(added in 6.5.8), but HTTPServerRequest.__init__ parsed the query string
with no such limit, so the same superlinear parsing cost was reachable
through the request line instead of the body.

Measured on the unpatched code, a query string with 400k fields took
3.2s of CPU to parse. In the default configuration the request line is
bounded by max_header_size (64KB), which caps this at roughly 12,700
arguments and ~47ms per request, but that is still an order of magnitude
past the limit the body path enforces, and applications are free to
raise max_header_size.

Pass the same max_arguments limit here. Over-limit query strings now
raise HTTPInputError, which the server already turns into a 400.

The limit is read from the global _DEFAULT_PARSE_BODY_CONFIG because
HTTPServerRequest has no access to any per-connection configuration.
This is the same limitation the body path already has in practice:
_parse_body calls parse_body_arguments without a config, so the body
limit is likewise only adjustable through set_parse_body_config.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

StaticFileHandler.validate_absolute_path checked the requested path
against the configured root using os.path.abspath, which does not
resolve symlinks. A symlink inside the static directory could therefore
point anywhere on the filesystem and its contents would be served: both
a symlinked file and a symlinked directory were confirmed to expose
files outside the root.

Resolve the path with os.path.realpath and require the result to remain
inside a new allowed_symlink_directory, which defaults to the served
path. Deployments that use such symlinks deliberately can widen it to a
directory containing all of the intended targets. Both this check and
the existing one on the path as written raise the same error, so a
caller cannot tell which way a path left the served directory.

The two checks are not merged, and the resolution stays after the check
on the unresolved path. That check must operate on the path as written:
when allowed_symlink_directory is wider than the root, comparing the
resolved path instead would let a ../ traversal out of the root while
rejecting the deliberate symlinks the setting exists to permit. Doing
the cheap string comparison first also avoids walking the filesystem
for a path that is already out of bounds.

Because default_filename is appended after the path has been resolved,
the joined path is resolved again: the directory containing an index
file may be legitimate while the index file itself is a symlink
pointing elsewhere. Deferring to a single resolution after that block
would be cheaper, but the isdir() check and the redirect it can trigger
would then run on a path that had already escaped, and the difference
between a 301 and a 403 would reveal whether an out-of-bounds directory
exists. A test pins this. The second resolution only happens for
directory-index requests; a request for a file resolves once.

realpath() is applied to the allowed directory as well as to the
requested path, so that serving a directory that is itself reached
through a symlink continues to work instead of rejecting every request.
strict=True is not used: it is unavailable on Python 3.9, and a path
that does not exist resolves to itself and is then reported as a 404.

This check reflects the filesystem at one instant, and a path swapped
for a symlink afterwards would not be caught; closing that race fully
would mean holding descriptors across the whole operation. Short of
that, the validated path is now stat'ed once and the result reused for
the Content-Length and Last-Modified headers, replacing the separate
os.path.exists and os.path.isfile calls. That cuts the number of times
a single request resolves the path from ten to six, and ensures the
headers describe the file that was validated. The residual race is
documented rather than papered over.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

…ng-memory-5fbgex-branch6.5' into fixes-659

pypy does not provide the tracemalloc module, so importing it at the top
of httpclient_test made the whole module fail to import there. Import it
conditionally (following the pattern used for pycurl and other optional
modules) and skip the three memory-measuring tests when it is missing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011PBzMWughm8fUFZDTrd8QF

This needs an explicitly hard-coded http client for the test
configuration that uses the `--httpclient` global override.