Repository navigation
DynamoDB warmup connections only warms up single connection #6964
Description
Activity
Confirmed on master:
initialize()awaits a singledescribe_limits(), so only one connection is opened, and sequential calls would reuse it too.Proposed fix: issue the warmup calls concurrently with
asyncio.gather(..., return_exceptions=True), so the connector has to open one connection per in-flight call. The number of calls would bemax_pool_connections, which is what the docs promise ("establishes the TCP connection pool"). A failed call only logs a warning, as now. The unit test would assert the call count and that calls overlap.If warming all 50 by default is too much, I can add a
warmup_connections_countoption instead (default:max_pool_connections). Happy to open a PR for whichever you prefer.Hey @CyberRik , thanks for chipping in. As mentioned in the issue, the describe_limits endpoint might get throttled, so making
max_pool_connectionsconcurrent calls to it might fail easily. I did not identify another endpoint that is not throttled and independent of concrete tables.Ah right, I missed that, thanks. Looking at it again it's actually a bit worse than the warmup just failing. Since the default
retry_modeisadaptive, the throttling errors would also make the client lower its own request rate, and that's the same client the real reads use. So a throttled warmup could slow down the first lookups after startup, and the retries would delayinitialize()too.What do you think about using
describe_endpoints()for the warmup instead? It doesn't depend on any table, there's no once-a-minute note on it like there is forDescribeLimits, and DynamoDB doesn't even check authorization for it on the public endpoints (docs). Behind a VPC endpoint it does needdynamodb:DescribeEndpoints, but even if that's denied the connection still gets opened, and an AccessDenied isn't retried or treated as throttling.One caveat: I don't have an AWS setup where I could check how it behaves under a burst, so if you're able to try it on your side that would help a lot. If it looks good to you I'll update the PR.
Expected Behavior
The new configuration parameter
warmup_connectionsin DynamoDB online store establishes connections for the entire connection pool as mentioned in the documentation.Added in #6711
Current Behavior
A single call to the
describe_limitsendpoint is done, which only establishes one connection.If multiple feature views are queried at the same time multiple connections from the pool will be used and need to be established.
It probably helps though for additional connections because DNS and credentials might be cached.
Steps to reproduce
Specifications
Possible Solution
Either call the endpoint asynchronously for the number of maximum connections in the pool or a configured minimum pool size. The describe_limits endpoint can be throttled though when requested more often than once per minute which makes it difficult to be used for these cases. https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DescribeLimits.html
Alternative would be to change the documentation and mention that a single connection is established which might be good enough for use cases where single feature views are requested.